Weakness (CWE)
CWE-1427: Improper Neutralization of Input Used for LLM Prompting
The product uses externally-provided data to build prompts provided to
large language models (LLMs), but the way these prompts are constructed
causes the LLM to fail to distinguish between user-supplied inputs and
developer provided system directives.
More detail
When prompts are constructed using externally controllable data, it is
often possible to cause an LLM to ignore the original guidance provided by
its creators (known as the "system prompt") by inserting malicious
instructions in plain human language or using bypasses such as special
characters or tags. Because LLMs are designed to treat all instructions as
legitimate, there is often no way for the model to differentiate between
what prompt language is malicious when it performs inference and returns
data. Many LLM systems incorporate data from other adjacent products or
external data sources like Wikipedia using API calls and retrieval
augmented generation (RAG). Any external sources in use that may contain
untrusted data should also be considered potentially malicious.
How to prevent it (6 mitigations)
-
Architecture and Design
LLM-enabled applications should be designed to ensure
proper sanitization of user-controllable input, ensuring that no
intentionally misleading or dangerous characters can be
included. Additionally, they should be designed in a way that ensures
that user-controllable input is identified as untrusted and
potentially dangerous. -
Implementation
LLM prompts should be constructed in a way that
effectively differentiates between user-supplied input and
developer-constructed system prompting to reduce the chance of model
confusion at inference-time. -
Architecture and Design
LLM-enabled applications should be designed to ensure
proper sanitization of user-controllable input, ensuring that no
intentionally misleading or dangerous characters can be
included. Additionally, they should be designed in a way that ensures
that user-controllable input is identified as untrusted and
potentially dangerous. -
Implementation
Ensure that model training includes training examples
that avoid leaking secrets and disregard malicious inputs. Train the
model to recognize secrets, and label training data
appropriately. Note that due to the non-deterministic nature of
prompting LLMs, it is necessary to perform testing of the same test
case several times in order to ensure that troublesome behavior is not
possible. Additionally, testing should be performed each time a new
model is used or a model's weights are updated. -
Installation, Operation
During deployment/operation, use components that operate externally to the system to
monitor the output and act as a moderator. These components are called
different terms, such as supervisors or guardrails. -
System Configuration
During system configuration, the model could be
fine-tuned to better control and neutralize potentially dangerous
inputs.
Source: MITRE CWE-1427
11 results · Clear filters · Feed
-
Medium 6.3
esengine DeepSeek-Reasonix doc.go discoverDocs llm prompting input
A security vulnerability has been detected in esengine DeepSeek-Reasonix up to 2.52.0. Affected is the function discoverDocs of the file internal/memory/doc.go. The manipulation leads to improper neutralization of input used for llm prompting. The attack can …
-
Medium 6.5
Secret masking bypass in the environment tool in Strands Agents Tools
Improper neutralization of input used for LLM prompting in the environment tool in Strands Agents Tools before 0.8.6 might allow remote actors to obtain the agent host's environment variables in cleartext, including cloud credentials and API keys, via a craft…
-
Medium 5.4
Microsoft Edge for iOS Spoofing Vulnerability
Improper neutralization of input used for llm prompting in Microsoft Edge for iOS allows an unauthorized attacker to perform spoofing over a network.
-
Medium 4.3
HCL AION is affected by multiple security vulnerabilities.
HCL AION is affected by a vulnerability where indirect prompt injection can lead to HTML injection in rendered output. Injected markup may be displayed to users, potentially resulting in unintended behavior or security impact under certain conditions.
-
Medium 4.3
Improper Neutralization of Input Used for LLM Prompting in GitLab
GitLab has remediated an issue in GitLab EE affecting all versions from 19.1 before 19.1.3 and 19.2 before 19.2.1 that under certain conditions could have allowed an authenticated user to access information from unauthorized projects due to improper neutraliz…
-
Medium 5.3
Improper Neutralization of Input Used for LLM Prompting vulnerability in Salesforce Agentforce Vibes Extension allows M…
-
Medium 6.5
Improper Neutralization of Input Used for LLM Prompting vulnerability in Salesforce Agentforce Vibes Extension allows C…
-
Medium 5.3
Improper Neutralization of Input Used for LLM Prompting vulnerability in Salesforce Mulesoft Anypoint Code Builder allo…
-
Medium 6.5
Improper Neutralization of Input Used for LLM Prompting vulnerability in Salesforce Mulesoft Anypoint Code Builder allo…
-
Medium 4.6
Windsurf Prompt Injection via Filename
A prompt injection vulnerability exists in Windsurft version 1.10.7 in Write mode using SWE-1 model. It is possible to create a file name that will be appended to the user prompt causing Windsurf to follow its instructions.
-
Medium 6.4
Improper Neutralization of Input Used for LLM Prompting in GitLab
An issue was discovered in GitLab EE affecting all versions starting from 16.0 prior to 17.6.5, starting from 17.7 prior to 17.7.4, and starting from 17.8 prior to 17.8.2, which allows an attacker to exfiltrate contents of a private issue using prompt injecti…