INSIGHTS | April 2, 2026

Virtual Assistant: Defeating Liveness Detection with the Help of Virtual Devices

By Elvin Gentiles

Introduction

The rise of fraud and identity theft poses a growing concern for both individuals and organizations. As AI and deepfake technologies advance at an unprecedented pace, the need for a robust form of identity verification has become increasingly important. Traditional identity verification technology has become vulnerable to sophisticated attacks, such as spoofing, where fraudsters mimic someone’s identity. To combat the growing threat, identity providers integrated liveness detection to ensure the person undergoing verification is real, live, and physically present. However, as liveness detection evolves, fraudsters have adapted to bypass this protection.

In this article, we will explore how liveness detection, which is designed to distinguish real users from spoofed representations, can be undermined through the use of virtual devices. We will cover the tools and techniques, the setup process, and a demonstration of the bypasses. We will also discuss how criminals take advantage of the biometric data (facial images, voice recordings, and ID photos) individuals and organizations often unknowingly post and leave online. Finally, we will conclude with practical recommendations to help identity providers combat such attacks and improve their systems.

Understanding Remote Identity Proofing

Remote Identity Proofing (RIDP) is the process of verifying someone’s identity through digital channels without requiring their physical presence. This helps combat fraud by confirming the end users are real and who they claim to be. In sectors like banking, insurance, cryptocurrencies, and payments, RIDP has emerged as a key component of digital trust. RIDP also plays a big role in regulatory programs like Know Your Customer (KYC), Customer Due Diligence (CDD), and Anti‑Money Laundering (AML).

Common forms of RIDP include:

Document validation, which relies upon the authenticity of government‑issued IDs such as passports or driver’s licenses.
Biometric verification, such as facial recognition, fingerprint scanning, and voice authentication, which focuses on someone’s unique physical characteristics.
Knowledge-based approaches, wherein multiple signals (e.g., passwords, PINs, OTPs) are collected.

Verification Methods & Their Attacks

This article focuses on the following forms of identity verification:

Facial Recognition
Document Validation
Voice Authentication

Facial Recognition

Facial recognition is the process where a user’s facial features are captured, extracted, and compared against a database of faces to confirm identity. In practice, this makes the camera pipeline and its integrity (how the face data is acquired and delivered to the verification component) a critical security boundary.

With this type of verification, fraudsters seek to impersonate a real person through Facial Spoofing, wherein a valid‑looking fake representation of the target is presented to the camera. This is typically carried out via:

2D Spoofing – presenting a printed photo or image displayed on a digital device (e.g., monitor or phone) to the camera.
Video Replay – presenting a pre‑recorded video to the camera rather than a live capture.
3D/Silicon Masks – creating a 3D reconstruction of a face or a mask to simulate a face in a physically present form.

Document Validation

Another method is verifying the authenticity and validity of an identity document such as a passport, national ID, or driver’s license. This method, called Document Validation, typically includes checking visual and structural cues and determining whether the presented document is legitimate versus altered or fabricated.

In an attempt to bypass document validation, fraudsters typically impersonate someone by presenting stolen, forged, or falsified documents. This is called Document Spoofing, and common methods include:

Printed Copy – using a printed reproduction of an authentic or forged document.
Screen Replay – presenting a legitimate (stolen) or forged document via a digital screen.
Portrait Overlay/Substitution – placing someone else’s photo over the original photo in a legitimate document.

Voice Authentication

Voice authentication identifies a person based on the unique biological characteristics of their voice, including traits such as dynamics, pitch, intensity, and articulation. As with face and document capture, the trustworthiness of the capture channel (microphone input and the path from device to verification system) is foundational.

Speech Spoofing focuses on impersonating a target user by producing a voice output that matches the target’s “voice print/pattern” and speech behavior. Common methods are:

Voice Imitation/Conversion – making the fraudster’s voice sound similar to the target’s voice.
Speech Synthesis/Cloning – using AI and text‑to‑speech (TTS) technologies to generate a replica of the target’s voice.
Voice Replay – using a pre‑recorded voice of the target

Liveness Detection as a Defensive Control

Liveness detection aims to prevent spoofing attacks by ensuring the user is live and physically or actively present at the time of the verification process rather than a recording or synthetic source. Liveness detection is implemented either passively or actively. With passive liveness detection, the verification system analyzes biometric signals without requiring explicit user interaction for a faster and seamless experience. As for active liveness detection, the system prompts users to perform actions such as head movements or speaking phrases to ensure users are actively/physically present during the verification process; however, it adds friction and inconvenience to some users.

The following demonstrates how liveness detection works for each verification system discussed above:

Facial Liveness

RIDP systems typically check for facial liveness using a combination of active and passive methods. Active liveness uses randomized challenge–response prompts, such as blinking or head movements, to confirm real‑time interaction. On the other hand, passive liveness runs in the background, analyzing depth cues, skin texture, light reflections, and subtle physiological signals that are difficult for photos, videos, or deepfakes to replicate.

Credit: PresentID

Document Liveness

Instead of checking a document’s format or text alone, document liveness analyzes how a document interacts with light and motion. Real IDs exhibit natural reflections on holograms and laminates, realistic shadows, and consistent texture patterns. Screen replays and printed copies expose artifacts such as Moiré patterns (image distortion due to the overlapping grid pattern between the display and the camera capturing it), color limitations, or unnatural edges, which modern AI models are trained to detect within a single image or a short video capture.

Credit: ID R&D

Voice Liveness

Passive voice liveness analyzes spectral and temporal characteristics of speech, identifying artifacts common in text‑to‑speech, voice cloning, and replay attacks. Whereas active liveness adds challenge‑response prompts, requiring users to speak unpredictable phrases (instead of predefined ones) in real time, making replay and synthetic attacks far more difficult.

Credit: ID R&D

Injection Attacks Against Liveness Detection

As liveness detection matured and spoofing/replay attacks became easier to detect, threat actors shifted toward more sophisticated techniques. This led to the rise of injection attacks, which target the verification pipeline instead of the sensor. Instead of fooling the sensors by presenting falsified media to a camera or microphone, injection attacks feed prerecorded, manipulated, or synthetic biometric data directly into the application or API. This makes the system believe the input is legitimate, since the injected data is a bit-for-bit replica of the source media and is of high quality.

The following diagram illustrates the difference between a replay and an injection attack:

Typically, Injection Attack Methods (IAM) fall into two categories: attacks performed via modified or falsified devices and attacks performed via substitution of captured media.

Injection via Modified or Falsified Devices

In this type of attack, the attacker manipulates the capture environment itself so that falsified media is treated as legitimate. One approach involves the use of virtual devices, where software‑based cameras or microphones emulate physical hardware. These virtual components can present attacker‑controlled images, videos, or audio streams directly to the verification system while appearing as standard capture devices.

Another approach uses hardware modules inserted between the sensor and the RIDP system. In this method, the hardware intercepts and replaces genuine sensor output with attacker‑supplied data before it reaches the verification logic.

A third variant relies on device emulators, where the entire verification environment is simulated. Emulated devices can present falsified data while mimicking expected device characteristics, allowing attackers to test and refine injection techniques at scale.

Injection via Substitution of Captured Media

Rather than altering the device itself, some threat actors may instead target the verification path that handles the captured media. One technique involves function hooking, where application- or system‑level functions responsible for camera or microphone input are intercepted and modified. This allows attackers to replace genuine capture data with prerecorded or synthetic media at runtime.

Another method relies on traffic interception, commonly described as a man‑in‑the‑middle (MiTM) scenario. In this case, media streams are intercepted and altered as they are transmitted between the client and backend services, enabling substitution without modifying the physical capture process.

Using Virtual Devices to Circumvent Liveness Detection

The final area to address is the use of virtual devices as a cheap and simple means to defeat liveness detection implemented with different verification methods.

Virtual devices are software-based devices that mimic the capabilities of physical devices and allow users to select different source media (images, videos, audio, scenes, etc.) and feed them to other applications. Because virtual devices are inexpensive (mostly open-source), easy to configure, and widely used for legitimate purposes such as streaming and conferencing, they present a scalable and practical attack vector.

Attack Demonstration

Virtual Device Against Facial Liveness

Scenario: Instead of using the phone’s native camera for a live capture, a fraudster injects a pre-recorded video using a virtual camera.

Virtual Device Against Document Liveness

Scenario: Instead of using a camera to capture the ID, the fraudster uses a virtual camera to submit it to the RIDP system.

Virtual Device Against Voice Liveness

Scenario: Instead of speaking through a microphone, the fraudster submits an audio record to the RIDP system using a virtual audio cable.

Threats: Widespread Biometric Data Exposure

A major factor behind the growing success of identity theft and imposter scams is the widespread availability of biometric data. Identity impersonation is no longer a high‑effort operation. Today, attackers can obtain biometric artifacts of their victims remotely, quickly, and at scale. Identity theft and impersonation no longer depend on physical proximity to the victim; attackers can now acquire usable biometric samples remotely and at scale.

Social media platforms have effectively become open biometric repositories. Profile pictures, short‑form videos, and live streams expose high‑quality facial imagery and voice samples across a wide range of lighting conditions, angles, and expressions. For attackers, this content is often more than sufficient to fuel replay attacks, deepfake generation, or voice cloning. Vlogs and user‑generated video content further raise the bar by offering extended, natural recordings of both face and voice. These longer samples reveal behavioral nuances, such as speech rhythm, pauses, expressions, and movement patterns, that can be leveraged to evade passive liveness detection and behavioral analysis.

The most severe risk comes from database and KYC leaks, which may expose high‑resolution IDs, enrollment selfies, and voice samples.

This level of biometric exposure fundamentally reshapes the RIDP threat model. Biometric data can no longer be treated as a protected secret. Instead, trust must shift toward verifiable capture integrity, proving that data was generated by a real sensor on a real device at a specific moment. Without strong capture‑time guarantees and device‑level assurance, even advanced liveness detection remains vulnerable to high‑fidelity impersonation attacks.

As biometric data becomes easier to obtain, the threat shifts from “Can attackers get the data?” to “Can systems reliably prove how and when the data was captured?” Addressing this shift is essential to defending modern RIDP deployments against scalable impersonation attacks.

Recommendations

To mitigate these risks, organizations should adopt a layered defense strategy, such as:

Identifying virtual devices through metadata, hardware identifiers, and supported capabilities.
Detecting emulated environments using hardware characteristics, sensor availability, performance profiles, and interaction patterns.
Ensuring authenticity and fidelity through remote image attestation, cryptographic signing, timestamping, and challenge‑response mechanisms.
Analyzing input data and session metadata for anomalies, mismatches, or missing sensor signals that indicate virtual sources.
Switching to mobile‑only RIDP by taking advantage of mobile security features like Trusted Execution Environments and sensor telemetry.
Combining multimodal verification and human review for high‑risk use cases, supported by logging and analysis of failed attempts.

Conclusion

Remote Identity Proofing is essential to modern digital ecosystems, but it operates in a constantly evolving adversarial environment. As replay attacks lose effectiveness, injection attacks pose a more serious challenge to liveness detection. RIDP systems should assume that the capture device itself may be untrusted and design controls accordingly.

Ultimately, securing remote identity verification remains a cat‑and‑mouse game, and defending them against threats requires more than a single control; it demands continuous improvement, layered defenses, and an in-depth understanding of both attacker capabilities and system limitations.

INSIGHTS | March 20, 2026

The Evolution of AI-Powered Security Consultants

By Lucas Apa

In my fourteen years of security assessments with IOActive, our shared mission has always been defined by a single commitment: stay ahead. Stay ahead of the threats clients face today, and stay ahead of the techniques that will define how we find those threats tomorrow.

That responsibility has driven every meaningful evolution in how our consultants work.

When fuzzing was still a research curiosity, the consultants who built their own frameworks and integrated it into live engagements found entire vulnerability classes that manual reviews missed. When static analysis tools were primitive and noisy, the teams that wrote custom rule sets and built triage pipelines around them separated signal from noise faster than anyone relying on defaults. In each case, the competitive edge went to those consultants who built the infrastructure to use new capabilities correctly before the rest of the industry caught up.

I’ve watched that pattern repeat with every major shift in the offensive landscape. However, none of them compare to what’s happening now with the capabilities of the large language models (LLMs) that fuel AI.

What follows are the decisions shaping how security assessments are built today: from the architectural constraints that dictate where AI can and can’t be used, to the strategies that determine whether it actually finds what matters.

The Model as Both Tool and Risk

The question is already coming up in kickoff calls. On recent engagements, clients have asked directly whether our AI assessment tooling sends their information to external infrastructure. One CISO put it clearly: if it leaves the engagement perimeter, it’s a breach of our NDA.

That posture is what’s driving consultants to build local LLM infrastructures. The reason is simple: the AI tools gaining traction today require sending client artifacts, such as source code, firmware, configuration files and protocol captures, to infrastructure you don’t control.

Hold that thought for a second – there’s an asymmetry there worth mentioning: the codebase and architecture we assess have almost certainly already traveled through cloud AI infrastructure during development. Copilot suggestions, ChatGPT-assisted debugging, Cursor completions, and online forum questions that include function signatures are all common today. Developers share code with online AI constantly, and at scale. That’s the reality of how software is written and architecture designed today.

That asymmetry cuts both ways, but while developers share code to build software, security assessments produce something far more sensitive: a complete map of how that software breaks. Vulnerability findings, exploitation chains, and architectural weaknesses – these are not debugging prompts. Losing control of that output is an entirely different category of risk.

The Model Is the Least Interesting Part

The open-weight code model landscape changes on a quarterly basis (or faster). But for an effective security testing infrastructure: the model should be the most swappable component.

What matters when selecting a model for a new engagement cycle is straightforward:

A context window sufficient for the artifact sizes we work with (code, Nmap scan results, etc.)
Demonstrated performance on the analysis tasks relevant to the engagement surface
Evaluation against the specific requirements the targets demand

Public benchmarks do already exist for evaluating LLMs on vulnerability detection. Most of them use a curated set of functions with known vulnerabilities, where the model must identify the bug, locate the exact source, and explain the exploitation path, resulting in a binary pass/fail. Usually, a model that finds the bug but misidentifies the root cause scores the same as one that misses it entirely.

In addition, the largest model is rarely the best model for a specific security analysis task. A general-purpose frontier model with hundreds of billions of parameters will score well on generic coding benchmarks, but a focused model a fraction of that size, fine-tuned on vulnerability data from domains we actually assess, will outperform it on the task that matters. Domain specialization beats raw scale when the domain is narrow enough and the training data is good enough.

Models rotate on someone else’s schedule, and are shipped, improved, and deprecated by labs we don’t control. That’s fine, as long as the model is the part that’s designed to be swapped.

The Input Changes. The Scaffolding Doesn’t.

If the model is swappable, the scaffolding is the multiplier force across engagement types. What changes between an application security assessment and a firmware audit is the ingestion format, not the pipeline architecture. Source code, decompiled binaries, ICS configurations, protocol captures, Active Directory enumeration output – each is a different artifact, but they all flow through the same stages: ingest, prioritize, analyze, triage. The scaffolding adapts at the ingestion layer. Everything downstream stays the same.

The critical architectural decision is where inference begins and where deterministic processing ends. Not every stage in the pipeline benefits from a language model. Parsing an AST to extract function signatures is deterministic. Matching a known CVE pattern against a dependency manifest is a lookup. Normalizing binary sections for analysis is a format transformation. These operations are faster, cheaper, and more reliable as traditional code. Sending them through an LLM adds latency and unpredictability for zero gain.

The model earns its place at the points where human-like reasoning is required: tracing a data path across service boundaries to determine reachability, assessing whether a configuration combination that appears safe creates an exploitable condition in context, or deciding if a code pattern that resembles a known vulnerability class is actually exploitable given the surrounding control flow. Those are judgment calls. Everything else is plumbing, and plumbing should be deterministic.

The scaffolding bridges both sides. It runs the deterministic stages as code, assembles the structured context the model needs to reason effectively, provides tool access so the model can query additional information during analysis, and routes the model’s output back into the deterministic pipeline for triage and reporting. The result is a system where the LLM operates within a well-defined scope, reasoning over prepared context, rather than being asked to do everything, including the work that doesn’t require intelligence.

A recent firmware engagement made this concrete. The deterministic layer extracted and normalized the binary, identified function boundaries, and reconstructed the call graph. Standard reversing infrastructure. The model received the decompiled output with full cross-reference context and flagged a conditional path in the bootloader’s recovery mode that bypassed signature verification before loading an update image. The logic was split across multiple functions, no single function looked wrong in isolation. An analyst would have reached it eventually, but the overnight batch surfaced it before anyone had started manually reversing that code path. That’s the division of labor working as designed: deterministic tooling prepares the context, the model reasons over it, and the analyst’s first morning starts with the right target.

Teaching the Model What We Know

A base model is a generalist. It knows code. It doesn’t know how our team hunts, what we’ve found before, or what the current target looks like from the inside. Closing that gap requires two complementary strategies: baking knowledge into the model’s weights, and feeding it context at inference time.

Fine-Tuning: What Vulnerable Code Looks Like

Base instruction-tuned models are not optimized for security analysis. The gap between a capable general-purpose coder model and one that has internalized what vulnerable code actually looks like across thousands of real examples is measurable in engagement quality.

There are many approaches to closing that gap: full fine-tuning, reinforcement learning from human feedback on security-specific tasks, synthetic data generation, distillation from larger models, and others. LoRA-based fine-tuning on curated real-world CVE data offers the fastest path to measurable improvement with the lowest infrastructure overhead. The critical distinction is what we train on: not just vulnerable code paired with its fix, but the analytical context around how the vulnerability is identified and exploited: root cause, affected data flow, and exploitation conditions. That context-rich approach produces measurably better results than training on code pairs alone.

The Engagement Wiki: What We Know That Training Data Doesn’t

Fine-tuning teaches the model what vulnerable code looks like. What it doesn’t teach is how our team hunts for it, what we’ve found manually, or what the current target looks like from the inside. IOActive has spent years building and refining internal methodologies across every engagement surface we assess. That accumulated knowledge is exactly what a retrieval layer can operationalize.

A structured internal wiki that feeds directly into the LLM’s context window via retrieval-augmented generation can operate at two layers:

Methodology and technique library:

This includes attack trees, enumeration checklists, and exploitation chains organized by engagement surface. When the model analyzes a Modbus configuration, it retrieves our team’s documented approach for ICS protocol assessment, not a generic prompt. When it reviews Android IPC handlers, it pulls the methodology our mobile team has refined over a decade of engagements. The library grows with every assessment cycle.

Per-engagement context:

At the start of every assessment, we populate a target-specific context layer: technology stack, architecture constraints, scope boundaries, threat model assumptions. The model doesn’t analyze in a vacuum; it reasons with the same situational awareness a human analyst builds during the first days of an engagement.

Reporting Is Also a Technical Challenge

Finding a vulnerability is half the job. The other half is communicating it in a way that produces a response proportional to its severity, and that audience is rarely homogeneous.

A typical engagement report lands in front of at least three different readers with fundamentally different needs. The development team wants precise reproduction steps and code-level context. The CISO wants risk framing, business impact, and remediation priority. The board or executive sponsor wants to understand exposure in terms they can act on without a security background. Until recently, producing documentation that served all three required either multiple document versions, expensive in consultant time, or a dedicated graphics and communications team that most consultancies can’t staff on every engagement.

Before LLM infrastructure, the practical ceiling on report comprehensiveness was analyst time. With a local LLM pipeline producing the first draft of each finding’s documentation, that ceiling lifts. Attack path diagrams that would have required a dedicated graphics pass are generated programmatically. The report becomes a more complete representation of what the assessment actually found, not a triage-filtered subset of it.

The same infrastructure can generate audience-specific derivatives of each finding. An attack path showing how an externally controllable input reaches a deserialization sink three service hops downstream is significantly more persuasive to an engineering leadership team than a paragraph describing the same chain. The diagrams are not just decorative. They’re structural.

This matters because impact is a function of communication, not just discovery. A finding that doesn’t produce a remediation response is a finding that didn’t land. Consultants who can close that gap, and those who can make a complex technical issue comprehensible to the people who control the budget to fix it, deliver more value in the same assessment. AI doesn’t just make reports faster to produce. It makes them more thorough, more detailed, and more actionable than what was previously feasible within the engagement window.

The Commitment

Security assessments are almost always time-boxed. The consultant makes depth tradeoffs every day: which functions to reverse manually, which attack paths to pursue, which custom tooling to build or skip. With LLM-augmented analysis, those tradeoffs change. Binary analysis that previously consumed a full day of manual effort compresses into hours. Custom tooling that would have been out of scope for a two-week engagement becomes feasible to build and deploy within the assessment window. The constraint hasn’t disappeared, but the frontier of what’s reachable within it has moved significantly, enabling you to go deeper on the same clock.

The AI landscape moves faster than any fine-tuning cycle, but security consulting has always rewarded teams that built the infrastructure to use new capabilities correctly rather than waiting for a packaged solution. The teams that figured out fuzzing before it was mainstream, or built custom static analysis pipelines when off-the-shelf options didn’t exist: those teams found more vulnerabilities, found them faster, and became harder to replace.

There’s a dimension to this that goes beyond tooling. Every security consultant brings a distinct set of methodology and analytical instinct to an engagement. A local AI infrastructure should reflect that. Rather than standardizing every analyst’s workflow through a single cloud tool with fixed behavior, a sovereign setup lets each consultant’s accumulated knowledge and methodological preferences shape how the model operates, amplifying individual expertise rather than flattening it.

That’s the commitment: not to just use the latest technology, but to evaluate it seriously, understand its limits, and operationalize it when it earns its place in the workflow. Local LLMs have earned theirs.

INSIGHTS, RESEARCH | May 30, 2024

The Security Imperative in Artificial Intelligence

By Gunter Ollmann

Artificial Intelligence (AI) is transforming industries and everyday life, driving innovations once relegated to the realm of science fiction into modern reality. As AI technologies grow more integral to complex systems like autonomous vehicles, healthcare diagnostics, and automated financial trading platforms, the imperative for robust security measures increases exponentially.

Securing AI is not only about safeguarding data but also about ensuring the core systems — in particular, the trained models that really put the “intelligence” in AI — function as intended without malicious interference. Historical lessons from earlier technologies offer some guidance and can be used to inform today’s strategies for securing AI systems. Here, we’ll explore the evolution, current state, and future direction of AI security, with a focus on why it’s essential to learn from the past, secure the present, and plan for a resilient future.

AI: The Newest Crown Jewel

Security in the context of AI is paramount precisely because AI systems increasingly handle sensitive data, make important, autonomous decisions, and operate with limited supervision in critical environments where safety and confidentiality are key. As AI technologies burrow further into sectors like healthcare, finance, and national security, the potential for misuse or harmful consequences due to security shortcomings rises to concerning levels. Several factors drive the criticality of AI security:

Data Sensitivity: AI systems process and learn from large volumes of data, including personally identifiable information, proprietary business information, and other sensitive data types. Ensuring the security of enterprise training data as it passes to and through AI models is crucial to maintaining privacy, regulatory compliance, and the integrity of intellectual property.
System Integrity: The integrity of AI systems themselves must be well defended in order to prevent malicious alterations or tampering that could lead to bogus outputs and incorrect decisions. In autonomous vehicles or medical diagnosis systems, for example, instructions issued by compromised AI platforms could have life-threatening consequences.
Operational Reliability: AI is increasingly finding its way into critical infrastructure and essential services. Therefore, ensuring these systems are secure from attacks is vital for maintaining their reliability and functionality in critical operations.
Matters of Trust: For AI to be widely adopted, users and stakeholders must trust that the systems are secure and will function as intended without causing unintended harm. Security breaches or failures can undermine public confidence and hinder the broader adoption of emerging AI technologies over the long haul.
Adversarial Activity: AI systems are uniquely susceptible to certain attacks, whereby slight manipulations in inputs — sometimes called prompt hacking — can deceive an AI system into making incorrect decisions or spewing malicious output. Understanding the capabilities of malicious actors and building robust defenses against such prompt-based attacks is crucial for the secure deployment of AI technologies.

In short, security in AI isn’t just about protecting data. It’s also about ensuring safe, reliable, and ethical use of AI technologies across all applications. These inexorably nested requirements continue to drive research and ongoing development of advanced security measures tailored to the unique challenges posed by AI.

Looking Back: Historical Security Pitfalls

We don’t have to turn the clock back very far to witness new, vigorously hyped technology solutions wreaking havoc on the global cybersecurity risk register. Consider the peer-to-peer recordkeeping database mechanism known as blockchain. When blockchain exploded into the zeitgeist circa 2008 — alongside the equally disruptive concept of cryptocurrency — its introduction brought great excitement thanks to its potential for both decentralization of data management and the promise of enhanced data security. In short order, however, events such as the DAO hack —an exploitation of smart contract vulnerabilities that led to substantial, if temporary, financial losses — demonstrated the risk of adopting new technologies without diligent security vetting.

As a teaching moment, the DAO incident highlights several issues: the complex interplay of software immutability and coding mistakes; and the disastrous consequences of security oversights in decentralized systems. The case study teaches us that with every innovative leap, a thorough understanding of the new security landscape is crucial, especially as we integrate similar technologies into AI-enabled systems.

Historical analysis of other emerging technology failures over the years reveals other common themes, such as overreliance on untested technologies, misjudgment of the security landscape, and underestimation of cyber threats. These pitfalls are exacerbated by hype-cycle-powered rapid adoption that often outstrips current security capacity and capabilities. For AI, these themes underscore the need for a security-first approach in development phases, continuous vulnerability assessments, and the integration of robust security frameworks from the outset.

Current State of AI Security

With AI solutions now pervasive, each use case introduces unique security challenges. Be it predictive analytics in finance, real-time decision-making systems in manufacturing systems, or something else entirely, each application requires a tailored security approach that takes into account the specific data types and operational environments involved. It’s a complex landscape where rapid technological advancements run headlong into evolving security concerns. Key features of this challenging infosec environment include:

Advanced Threats: AI systems face a range of sophisticated threats, including data poisoning, which can skew an AI’s learning and reinforcement processes, leading to flawed outputs; model theft, in which proprietary intellectual property is exposed; and other adversarial actions that can manipulate AI perceptions and decisions in unexpected and harmful ways. These threats are unique to AI and demand specialized security responses that go beyond traditional cybersecurity controls.
Regulatory and Compliance Issues: With statutes such as GDPR in Europe, CCPA in the U.S., and similar data security and privacy mandates worldwide, technology purveyors and end users alike are under increased pressure to prioritize safe data handling and processing. On top of existing privacy rules, the Biden administration in the U.S. issued a comprehensive executive order last October establishing new standards for AI safety and security. In Europe, meanwhile, the EU’s newly adopted Artificial Intelligence Act provides granular guidelines for dealing with AI-related risk. This spate of new rules can often clash with AI-enabled applications that demand more and more access to data without much regard for its origin or sensitivity.
Integration Challenges: As AI becomes more integrated into critical systems across a wide swath of vertical industries, ensuring security coherence across different platforms and blended technologies remains a significant challenge. Rapid adoption and integration expose modern AI systems to traditional threats and legacy network vulnerabilities, compounding the risk landscape.
Explainability: As adoption grows, the matter of AI explainability — or the ability to understand and interpret the decisions made by AI systems — becomes increasingly important. This concept is crucial in building trust, particularly in sensitive fields like healthcare where decisions can have profound impacts on human lives.Consider an AI system used to diagnose disease from medical imaging. If such a system identifies potential tumors in a scan, clinicians and patients must be able to understand the basis of these conclusions to trust in their reliability and accuracy. Without clear explanations, hesitation to accept the AI’s recommendations ensues, leading to delays in treatment or disregard of useful AI-driven insights. Explainability not only enhances trust, it also ensures AI tools can be effectively integrated into clinical workflows, providing clear guidance that healthcare professionals can evaluate alongside their own expertise.

Addressing such risks requires a deep understanding of AI operations and the development of specialized security techniques such as differential privacy, federated learning, and robust adversarial training methods. The good news here: In response to AI’s risk profile, the field of AI security research and development is on a steady growth trajectory. Over the past 18 months the industry has witnessed increased investment aimed at developing new methods to secure AI systems, such as encryption of AI models, robustness testing, and intrusion detection tailored to AI-specific operations.

At the same time, there’s also rising awareness of AI security needs beyond the boundaries of cybersecurity organizations and infosec teams. That’s led to better education and training for application developers and users, for example, on the potential risks and best practices for securing A-powered systems.

Overall, enterprises at large have made substantial progress in identifying and addressing AI-specific risk, but significant challenges remain, requiring ongoing vigilance, innovation, and adaptation in AI defensive strategies.

Data Classification and AI Security

One area getting a fair bit of attention in the context of safeguarding AI-capable environments is effective data classification. The ability to earmark data (public, proprietary, confidential, etc.) is essential for good AI security practice. Data classification ensures that sensitive information is handled appropriately within AI systems. Proper classification aids in compliance with regulations and prevents sensitive data from being used — intentionally or unintentionally — in training datasets that can be targets for attack and compromise.

The inadvertent inclusion of personally identifiable information (PII) in model training data, for example, is a hallmark of poor data management in an AI environment. A breach in such systems not only compromises privacy but exposes organizations to profound legal and reputational damage as well. Organizations in the business of adopting AI to further their business strategies must be ever aware of the need for stringent data management protocols and advanced data anonymization techniques before data enters the AI processing pipeline.

The Future of AI Security: Navigating New Horizons

As AI continues to evolve and tunnel its way further into every facet of human existence, securing these systems from potential threats, both current and future, becomes increasingly critical. Peering into AI’s future, it’s clear that any promising new developments in AI capabilities must be accompanied by robust strategies to safeguard systems and data against the sophisticated threats of tomorrow.

The future of AI security will depend heavily on our ability to anticipate potential security issues and tackle them proactively before they escalate. Here are some ways security practitioners can prevent future AI-related security shortcomings:

Continuous Learning and Adaptation: AI systems can be designed to learn from past attacks and adapt to prevent similar vulnerabilities in the future. This involves using machine learning algorithms that evolve continuously, enhancing their detection capabilities over time.
Enhanced Data Privacy Techniques: As data is the lifeblood of AI, employing advanced and emerging data privacy technologies such as differential privacy and homomorphic encryption will ensure that data can be used for training without exposing sensitive information.
Robust Security Protocols: Establishing rigorous security standards and protocols from the initial phases of AI development will be crucial. This includes implementing secure coding practices, regular security audits, and vulnerability assessments throughout the AI lifecycle.
Cross-Domain Collaboration: Sharing knowledge and strategies across industries and domains can lead to a more robust understanding of AI threats and mitigation strategies, fostering a community approach to AI security.

Looking Further Ahead

Beyond the immediate horizon, the field of AI security is set to witness several meaningful advancements:

Autonomous Security: AI systems capable of self-monitoring and self-defending against potential threats will soon become a reality. These systems will autonomously detect, analyze, and respond to threats in real time, greatly reducing the window for attacks.
Predictive Security Models: Leveraging big data and predictive analytics, AI can forecast potential security threats before they manifest. This proactive approach will allow organizations to implement defensive measures in advance.
AI in Cybersecurity Operations: AI will increasingly become both weapon and shield. AI is already being used to enhance cybersecurity operations, providing the ability to sift through massive amounts of data for threat detection and response at a speed and accuracy unmatchable by humans. The technology and its underlying methodologies will only get better with time. This ability for AI to remove the so-called “human speed bump” in incident detection and response will take on greater importance as the adversaries themselves increasingly leverage AI to generate malicious attacks that are at once faster, deeper, and potentially more damaging than ever before.
Decentralized AI Security Frameworks: With the rise of blockchain technology, decentralized approaches to AI security will likely develop. These frameworks can provide transparent and tamper-proof systems for managing AI operations securely.
Ethical AI Development: As part of securing AI, strong initiatives are gaining momentum to ensure that AI systems are developed with ethical considerations in mind will prevent biases and ensure fairness, thus enhancing security by aligning AI operations with human values.

As with any rapidly evolving technology, the journey toward a secure AI-driven future is complex and fraught with challenges. But with concerted effort and prudent innovation, it’s entirely within our grasp to anticipate and mitigate these risks effectively. As we advance, the integration of sophisticated AI security controls will not only protect against potential threats, it will foster trust and promote broader adoption of this transformative technology. The future of AI security is not just about defense but about creating a resilient, reliable foundation for the growth of AI across all sectors.

Charting a Path Forward in AI Security

Few technologies in the past generation have held the promise for world-altering innovation in the way AI has. Few would quibble with AI’s immense potential to disrupt and benefit human pursuits from healthcare to finance, from manufacturing to national security and beyond. Yes, Artificial Intelligence is revolutionary. But it’s not without cost. AI comes with its own inherent collection of vulnerabilities that require vigilant, innovative defenses tailored to their unique operational contexts.

As we’ve discussed, embracing sophisticated, proactive, ethical, collaborative AI security and privacy measures is the only way to ensure we’re not only safeguarding against potential threats but also fostering trust to promote the broader adoption of what most believe is a brilliantly transformative technology.

The journey towards a secure AI-driven future is indeed complex and fraught with obstacles. However, with concerted effort, continuous innovation, and a commitment to ethical practices, successfully navigating these impediments is well within our grasp. As AI continues to evolve, so too must our strategies for defending it.

EDITORIAL | March 1, 2024

Opinion: AGI Influencing the Secure Code Review Profession

By Gunter Ollmann

It’s tough to be a secure code reviewer. There are already over 700 programming languages according to Wikipedia, and seemingly more languages materializing every year. Expectations are high that rapid developments in Artificial Generative Intelligence (AGI) will bring a new suite of languages and security issues that’ll have an oversized impact on software development. Consequently, secure software development lifecycle (SDL) processes and security code review are having to evolve rapidly.

I’m both excited and nervous about AGI advancements in the world of software development and secure application design. It’s exciting to see how prompt engineering of Large Language Models (LLM) and adoption of AI augmentation in the form of copilots and chatbots are increasing the pace of ideation into new products. I’m nervous about the hallucinations and code quality being generated in response though.

English as a Programming Language

2023 was the breakthrough year for AI, with LLM and AGI permeating every industry, technology, and product. Today, the most in-demand languages currently are Python, C, and C++ but, controversially, the future star programming language may in fact be English; something that’ll take some time to adjust to.

For over a decade we’ve been told that the supply of experienced cybersecurity professionals has trailed the market’s requirements, with a deficit growing year-on-year, and a casual scan across office desks and cubicles will highlight a more significant gender gap across the cybersecurity (and software development) industry. I think AGI and emergence of English as a critical programming language are fundamental to correcting both industry problems.

AGI, particularly those based upon LLM advancements, are increasingly sophisticated language machines – and women may have an advantage over men in maximizing utility and productivity from them.

Multiple studies over the last 30 years have constantly highlighted that women are better communicators than men. “Better” is obviously an explosive and controversial term even amongst the academics who published the studies, but in general women have more expansive vocabularies and stronger interpretative communication skills. Modern neuroscience and studies in children and adolescents identify girls as more garrulous than boys, with greater complexity and sophistication of language, and tend to develop more in the realm of listening with greater focus and concentration as they age. This historically translates into women being better coders than men (once you remove the bias in the system).

As I look to AGI and the expanding world of prompt engineering, I anticipate that women will have an advantage over their male developer counterparts. Strong and well-developed communication skills (and the reasoning and understanding that underlays those polished skills) are prerequisites for maximizing efficiency of AGI-returned results and tuning responses – both now and for the immediate future.

Starter-job Experience

But what about experience? The “experience gap” is often called out as a chasm for newly minted degree-holding graduates and landing a starter-job in cybersecurity.

It’s rare to find an entry-level job in our industry that doesn’t require multiple years of hands-on security experience nowadays as many of those traditional starter roles – network scanning, alert triage, playbook maintenance, patch management – have been automated away, with many more projected to disappear as AI adoption increases.

Most successful new entrants into the cybersecurity profession come from adjacent technical industries making a career jump rather than direct from a college or university. Armed with transferable skills and technical experience, they’re capable of crossing the chasm left in the wake of cyber automation. However, the security knowledge gap between a cybersecurity veteran and a recent transfer remains large and a growing concern for the industry.

I’m excited to think AI augmentation and copilot technologies will have one of the largest impacts on our industry – removing much of the security knowledge gap and reducing the overall impact of the experience gap – like what is happening in other industries, such as the medical field. For example, AI use in patient triage, predictive analytics, and virtual assistants are augmenting generalist regional nurses (two-year qualification) and Bachelor of Science in Nursing (four-year qualification) graduates, and allowing them to perform many of the roles and responsibilities traditionally associated with a completed medical doctor degree (10 to 12 years).

Secure Code Reviews

It’s tough to be a secure code reviewer. There aren’t enough of them. The job requires tremendous amounts of experience and advanced security knowledge, and it’s tiring and hard work.

AGI is going to have a huge impact on their job.

On the positive side, English as a programming language and AI augmentation and copilots is going to help increase both the breadth and depth of the cybersecurity talent pool available to perform this critical job. The tools available to code reviewers to automatically review and assess code security are advancing quickly and, while still in their first generation of AI adoption, are anticipated to mature rapidly and identify vulnerabilities and logic flaws with higher fidelity and trust. I’m sure there’ll be a gap between the best that a tool can achieve versus the best-of-the-best human expert though – especially when that expert is augmented and using similar tools themselves.

Meanwhile, AGI is spearheading prompt engineering of new software applications. A new generation of product developers may have little to no influence over the code that powers the application. Indeed, I’ve previously argued that the role of product manager will change greatly in the coming years as their skills in product design and requirement setting pivot from being directed to engineering teams and into AGI prompts instead.

What does an AGI-generated application look like under the covers? Time will tell. We anticipate it’ll increasingly become more secure – using best security practices and recycling pretested code behind the scenes – and that it’ll constantly learn, optimize, and apply best and better security practices – but we’ll still need those human secure code reviewers for some time to come, specially when it comes to high-impact applications and certification.

A concern though as AGI does more of the application development software developers will have less direct influence over the underlying code is that powering the application and business logic. It would be a nightmare if the AGI produced entirely different code throughout the application each time it received new design criteria and re-optimized – making vulnerability triage, reporting, reconciliation, and tracking near impossible, and code reviews and code certifications (human or tool led) largely nonsensical.

Watch this space!

I think we’re still some years away from having to worry about continuously reimagined code generated by AGI without human software developers tweaking and refining the underlaying application code, but it is tremendously exciting to see the rapid advances in prompt engineering and how LLM’s are being incorporated into both old and new products.

Our industry has consistently struggled to attract and retain women. AGI has the potential to not only level the field and make it easier to join the game, but to also leverage previously poorly-tapped communication skills for the betterment of both application development and security. There’s a lot of work ahead. There’s a lot of research to be done. There’s a lot of opportunities to make code more secure!