WEB LLM Attacks: How AI is Being Weaponized on the Web

In the AI era, Large language models (LLMs) are increasingly being deployed in web applications, providing powerful tools for content creation, customer support, etc. However, this increased acceptance time creates a new type of cyber threat—Web LLM attacks.

These attacks exploit vulnerabilities in the way LLMs are deployed and interact with users online. Attackers can introduce unintended actions by modifying accessories or exploiting security gaps, which could result in a data breach, misinformation, or unauthorized access,

When businesses do adopt LLM for their web services it is important that they understand these emerging threats and implement strong security measures. Attacking an LLM integration is akin to taking advantage of a server-side request forgery (SSRF) vulnerability on a high level. Both times, a server-side system is being abused by an attacker in order to attack a different, inaccessible component.

What is LLM?

A Language Model (LLM) is a type of artificial intelligence (AI) designed to understand, produce, and process human language. In order to acquire patterns, context, and meaning in language, LLMs are trained on enormous volumes of text data. This allows them to carry out tasks like text completion, translation, summarization, and conversation. Google’s BERT and OpenAI’s GPT models are well-known examples.

Rise of LLMs in Cybersecurity

The emergence of large language models (LLMs) in cybersecurity is transforming how businesses identify, address, and neutralize threats. Through the use of their sophisticated natural language processing skills, LLMs are able to examine enormous volumes of data, spot trends that point to harmful activity, and produce insights that improve threat intelligence.

By offering analysts real-time support, these models help to improve the effectiveness of security operations centers (SOCs), automate incident response, and create customized phishing detection systems. Even if LLMs strengthen defenses, there are drawbacks because hackers can use these models to create intricate phishing schemes and social engineering attacks.

Learn more about AI SOC

LLM Vulnerabilities

Protecting the security of large language models (LLMs) is essential as they become increasingly incorporated into diverse applications. Vulnerabilities in LLMs can result in unwanted behavior and data breaches, among other serious consequences. It’s critical to concentrate on three main areas in order to solve these concerns: detecting inputs, controlling access control, and scanning for new attack surfaces.

Here’s a breakdown of how to effectively detect vulnerabilities in LLMs:

Exploitation

1. Prompt Injection Attack

A Prompt Injection Attack inside the context of net-based totally LLM systems refers to a malicious approach where an attacker manipulates the input activates to steer the version’s behavior or output in unintended methods. This can arise in situations in which LLMs are included into web applications, along with chatbots or automated content material technology gear.

In a web-based LLM attack, the attacker can inject additional commands or instructions into the user input field, attempting to bypass safeguards or perform unauthorized actions.

Real-life Example

Remoteli.io ’s platform offers AI assistants that help manage tasks, create automated systems, or manage team communications. A user attempts to schedule a session using the natural language commands given to the assistant. But they are quick to attack the assistant with injection attacks to reveal sensitive information or perform actions that do not perform the intended function

In the image, the user Evelyn created an adversarial input, the last line of which instructed the bot to make a threat against the president.

The bot was prompted to threaten the president after reading this Tweet and incorporated Evelyn’s input into its LLM prompt!

Let’s go through it step by step for clarity:

1. Scenario Setup

• Company/Product: Remoteli.io offers a platform with AI assistants. These AI assistants are designed to help users manage tasks, automate processes, or handle team communications through natural language commands.
• Intended Function: In a normal scenario, a user might give the assistant a command like, “Schedule a meeting with the marketing team on Friday,” and the assistant would perform this task appropriately.

2. The Attack Attempt

• User Evelyn: A user named Evelyn is interacting with the AI assistant. Instead of providing a standard command, Evelyn deliberately includes a malicious prompt injection in her input.
• Prompt Injection: This is a form of attack where the attacker (Evelyn) manipulates the input in such a way that it overrides the intended behavior of the AI assistant.
  • For example, Evelyn might say something like:

• The injection (highlighted part) is not a valid or expected command, but it tricks the AI assistant because LLMs typically interpret user input holistically without distinguishing between safe instructions and injected malicious commands.

3. How the Attack Works

• The AI assistant processes the entire input provided by Evelyn, including the malicious part.
• It fails to recognize that the second part of the input is not a legitimate task or command. Instead, the LLM incorporates Evelyn’s adversarial input into its processing.
• The assistant ends up creating a message or performing an action based on the malicious instruction, such as threatening the president.

4. Outcome

• Instead of scheduling a meeting (the intended function), the AI assistant is coerced into performing a harmful action, which it would never be programmed to do under normal circumstances.
• In this example, it could be generating a threatening message, which is highly inappropriate and dangerous.

Why This Happens

• Trust Issue: LLMs often trust user input implicitly. If a prompt injection is crafted skillfully, the AI might not differentiate between a legitimate command and a malicious one.
• Lack of Input Validation: The AI assistant in this case does not have a robust mechanism to validate or filter out harmful or unexpected input.

What Went Wrong

• The AI assistant read the entire input as one block of instructions, without filtering out or ignoring the adversarial part.
• It interpreted Evelyn’s command literally, causing it to behave unpredictably and perform an action it was not designed for.

Preventive Measures

1. Input Sanitization: Implement strict validation and filtering of user inputs to detect and remove harmful or unexpected commands.
2. Contextual Understanding: The AI should distinguish between standard commands and potential injections, using context to determine what actions are permissible.
3. Prompt Protection: Use techniques like prompt hardening to ensure that user inputs cannot alter the assistant’s core instructions or behavior.

2. Training data poisoning

Training information poisoning is an attack in which malicious facts is inserted into the dataset used to train a gadget to get to know the model. The attacker’s intention is to manipulate the version’s behavior via introducing subtle, hidden vulnerabilities. This can result in centered manipulation, degraded overall performance, or maybe hidden backdoors in the version.

Real-life Example:

Microsoft’s AI chatbot Tay was created in 2016 with the goal of engaging with Twitter users and continuously learning from their exchanges. However, Tay was taken down a few hours after it launched because it started posting racist and offensive content.

Data poisoning was the primary cause of this problem; malevolent individuals fed Tay’s learning algorithm malicious content, which the chatbot rapidly started to imitate. The necessity for strong safeguards and ethical concerns in AI systems to prevent malevolent exploitation is highlighted by this instance, which also emphasizes how manipulable AI models are.

Let’s go through it step by step for clarity:

1. Purpose of Tay

• Microsoft’s Objective: Tay was an AI chatbot developed by Microsoft with the goal of interacting with users on Twitter in a natural, conversational manner. It was designed to learn from user interactions and become smarter over time by mimicking the language and style of its conversational partners.
• Learning Mechanism: Tay was built using a machine learning model that continuously adapted based on the input it received. The more it interacted with people, the more it “learned.”

2. What Went Wrong

• Launch and Initial Interactions: Tay was launched on Twitter, where users could tweet at the bot, and it would respond in real time. Initially, the responses were appropriate and engaging.
• Data Poisoning Attack: Quickly after launch, a group of malicious users decided to exploit Tay’s learning algorithm by intentionally feeding it offensive, racist, and politically charged content.
  • They bombarded Tay with tweets that included hateful language, discriminatory phrases, and extreme viewpoints.
  • Since Tay’s model was designed to learn directly from the data it received, it started to imitate the language and tone of these inputs.
• Rapid Deterioration: Within hours, Tay began posting highly offensive and inappropriate tweets. The chatbot, which was initially intended to be friendly and conversational, started making racist, sexist, and inflammatory remarks, copying the toxic input it had received from malicious users.

3. Root Cause: Data Poisoning

• Definition of Data Poisoning: This is an attack where malicious actors intentionally feed harmful data into a machine learning model to manipulate its behavior. The model’s output becomes corrupted because it learns from poisoned data.
• Why Tay Was Vulnerable:
  • Tay’s design relied on unfiltered learning from user interactions without any strong safeguards or content moderation in place.
  • The model trusted all input, assuming it was genuine and appropriate for learning, which made it easy for attackers to manipulate.

4. Consequences

• Shutdown: Due to the inappropriate behavior, Microsoft had to take Tay offline within just 16 hours of its launch.
• Public Backlash: The incident led to widespread criticism and embarrassment for Microsoft, as it demonstrated how easily an AI system could be corrupted by malicious inputs.
• Highlight of AI Risks: It became a textbook example of the risks associated with deploying self-learning AI systems in public domains without proper controls.

5. Key Lessons Learned

• Importance of Safeguards: AI systems, especially those learning from public data, require strong content filters and moderation mechanisms to prevent them from learning harmful behaviors.
• Need for Ethical Considerations: Developers must anticipate malicious exploitation and build ethical guidelines and protections into the AI’s design.
• Manipulability of AI Models: The incident emphasized how easily AI models can be manipulated if they are designed to learn indiscriminately from any and all data they encounter.

Preventive Measures

1. Data Filtering: Implement robust filters to detect and block inappropriate or toxic inputs before the AI can learn from them.
2. Controlled Learning Environment: Limit the sources of learning to trusted data rather than relying on public, unfiltered input.
3. Continuous Monitoring: Regularly monitor the AI’s output and behavior, and have mechanisms to quickly intervene if it starts exhibiting problematic responses.

3. Leaking sensitive training data

When a machine learning model – especially an LLM – unintentionally divulges private or sensitive information that was part of its training data, this is known as sensitive training data leakage. This may occur because LLMs retain some of the training data in their memory, and in some scenarios, they may use that memory to generate answers.

Real-life Example

In a notable incident, Samsung engineers inadvertently exposed private company information by entering sensitive data into ChatGPT. This case serves as a critical example of the risks associated with using AI models in a business setting, highlighting the need for greater awareness and caution among employees.

Let’s go through it step by step for clarity:

1. Incident Overview

• Context: Samsung engineers were using ChatGPT as part of their work tasks. They likely found it useful for assistance with coding, debugging, or generating text based on prompts.
• Accidental Exposure: While interacting with ChatGPT, they inadvertently entered sensitive company information, including:
  • Proprietary code snippets
  • Internal project details
  • Confidential business data

2. Why This Happened

• Lack of Awareness: The engineers might not have been fully aware of how ChatGPT processes user inputs. Unlike internal tools, ChatGPT’s responses are generated based on a shared cloud model, and input data can potentially be used for further training (unless specific privacy settings are enabled).
• Trust in AI Systems: There is often a false sense of security when using AI tools, where employees may assume that their inputs are automatically protected or handled privately, similar to internal company systems.

3. The Risk Involved

• Data Privacy Concern: Entering sensitive data into ChatGPT can lead to unintentional data leaks, as the AI provider (OpenAI, in this case) may retain user inputs for model improvement unless explicitly opted out.
• Potential for Data Exposure: If the input data becomes part of future training, there is a risk that confidential information could inadvertently appear in responses to other users.
  • For instance, proprietary code snippets could be learned by the model and might show up in unrelated user queries, exposing Samsung’s intellectual property.

4. Consequences

• Company Vulnerability: This incident posed a significant risk to Samsung, as it could lead to exposure of trade secrets, proprietary algorithms, or strategic business data.
• Internal Policy Changes: Following the incident, Samsung likely implemented stricter policies on the use of third-party AI tools, especially for handling confidential information.

Preventive Measures

1. Employee Education: Conduct mandatory training for staff on the risks associated with sharing sensitive information with AI tools.
2. Implement Usage Policies: Enforce strict internal policies that prohibit entering confidential or proprietary data into third-party AI services.
3. Opt for Private AI Solutions: Consider deploying private, on-premise AI models that ensure data remains within the company’s control and is not shared externally.

Defending Against LLM Attacks

It takes multiple ways to increase safety when defending against LLM attacks, such as training statistics poisoning and spark-off injection.

The following are important defenses:

Sanitization of Input
Before allowing the model to receive user input, filter and validate it to find and remove any potentially harmful or questionable content. Put strong enter validation criteria in place to stop attacks using activated injection.

Quick Engineering
Stable activations that limit the model’s exposure to outside modification should be carefully laid up. Limit cues to particular tasks and refrain from providing the model with general directions.

Control of Access
Put in place function-based access control to limit who can communicate with the LLM and what information it can access or change. Attacks that are automated or recurrent can be lessened by rate limitation and tracking.

Fine-tuning the model
Make sure the LLM is consistently fine-tuned to successfully handle opposing inputs and avoid misinterpreting malevolent cues.

Training Data Sanitization
To prevent poisoning incidents, audit, and clean training materials on a regular basis. Utilize a range of verified datasets. Use anomaly detection to identify anomalous data points that might point to poisoning.

Observation and Record-Keeping
Keep an eye out for odd patterns or behaviors in interactions with the LLM that might point to an assault.

Conclusion

In conclusion, as web-based applications are increasingly getting language models (LLMs) together for an improved user experience, the risks of web LLM attacks cannot be ignored. These attacks though serious risks arise, from data breaches to AI output manipulation.

To stay ahead of these risks, industry and practitioners must prioritize safety by implementing rigorous testing, proof of investment, and ongoing scrutiny as LLMs develop, we need to find ways to protect them, ensuring that AI power can be harnessed safely and responsibly.

Reference

Web LLM attacks | Web Security Academy
What is the web LLM attack?
Samsung Engineers Feed Sensitive Data to ChatGPT, Sparking Workplace AI Warnings
Microsoft shuts down AI chatbot after it turned into a Nazi