Skip to content
OpenAdvisories

Weakness (CWE)

CWE-1427: Improper Neutralization of Input Used for LLM Prompting

Base Incomplete

The product uses externally-provided data to build prompts provided to
large language models (LLMs), but the way these prompts are constructed
causes the LLM to fail to distinguish between user-supplied inputs and
developer provided system directives.

More detail

When prompts are constructed using externally controllable data, it is
often possible to cause an LLM to ignore the original guidance provided by
its creators (known as the "system prompt") by inserting malicious
instructions in plain human language or using bypasses such as special
characters or tags. Because LLMs are designed to treat all instructions as
legitimate, there is often no way for the model to differentiate between
what prompt language is malicious when it performs inference and returns
data. Many LLM systems incorporate data from other adjacent products or
external data sources like Wikipedia using API calls and retrieval
augmented generation (RAG). Any external sources in use that may contain
untrusted data should also be considered potentially malicious.

How to prevent it (6 mitigations)
  1. Architecture and Design

    LLM-enabled applications should be designed to ensure
    proper sanitization of user-controllable input, ensuring that no
    intentionally misleading or dangerous characters can be
    included. Additionally, they should be designed in a way that ensures
    that user-controllable input is identified as untrusted and
    potentially dangerous.

  2. Implementation

    LLM prompts should be constructed in a way that
    effectively differentiates between user-supplied input and
    developer-constructed system prompting to reduce the chance of model
    confusion at inference-time.

  3. Architecture and Design

    LLM-enabled applications should be designed to ensure
    proper sanitization of user-controllable input, ensuring that no
    intentionally misleading or dangerous characters can be
    included. Additionally, they should be designed in a way that ensures
    that user-controllable input is identified as untrusted and
    potentially dangerous.

  4. Implementation

    Ensure that model training includes training examples
    that avoid leaking secrets and disregard malicious inputs. Train the
    model to recognize secrets, and label training data
    appropriately. Note that due to the non-deterministic nature of
    prompting LLMs, it is necessary to perform testing of the same test
    case several times in order to ensure that troublesome behavior is not
    possible. Additionally, testing should be performed each time a new
    model is used or a model's weights are updated.

  5. Installation, Operation

    During deployment/operation, use components that operate externally to the system to
    monitor the output and act as a moderator. These components are called
    different terms, such as supervisors or guardrails.

  6. System Configuration

    During system configuration, the model could be
    fine-tuned to better control and neutralize potentially dangerous
    inputs.

Source: MITRE CWE-1427

1 result · Clear filters · Feed

  • Low 3.7

    Vikunja veans before 2.7.1 Prompt Injection via Project and Task Metadata

    Vikunja veans before 2.7.1 contains a prompt injection vulnerability that allows project members with write access to inject instructions by setting project titles or task content rendered verbatim inside the coding agent's authoritative instruction block. At…