Weakness (CWE)
CWE-1426: Improper Validation of Generative AI Output
The product invokes a generative AI/ML
component whose behaviors and outputs cannot be directly
controlled, but the product does not validate or
insufficiently validates the outputs to ensure that they
align with the intended security, content, or privacy
policy.
How to prevent it (4 mitigations)
-
Architecture and Design
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
-
Operation
Use "semantic comparators," which are mechanisms that
provide semantic comparison to identify objects that might appear
different but are semantically similar. -
Operation
Use components that operate
externally to the system to monitor the output and
act as a moderator. These components are called
different terms, such as supervisors or
guardrails. -
Build and Compilation
During model training, use an appropriate variety of good
and bad examples to guide preferred outputs.
Source: MITRE CWE-1426
2 results · Clear filters · Feed
-
Low 3.0
Channel member objects leak read status
Mattermost versions 10.11.x <= 10.11.3, 10.5.x <= 10.5.11 fail to enforce access permissions on the Agents plugin which allows other users to determine when users had read channels via channel member objects
-
Low 3.0
Data exfiltration via AI plugin Jira tool
Mattermost versions 10.4.x <= 10.4.2, 10.5.x <= 10.5.0, 9.11.x <= 9.11.9 fail to restrict domains the LLM can request to contact upstream which allows an authenticated user to exfiltrate data from an arbitrary server accessible to the victim via performing a …