Vanna AI中的提示注入漏洞暴露数据库面临RCE攻击

Cybersecurity researchers have disclosed a high-severity security flaw in the Vanna.AI library that could be exploited to achieve remote code execution vulnerability via prompt injection techniques.

网络安全研究人员披露了Vanna.AI库中一个高危安全漏洞，该漏洞可以通过提示注入技术实现远程代码执行漏洞。

The vulnerability, tracked as CVE-2024-5565 (CVSS score: 8.1), relates to a case of prompt injection in the "ask" function that could be exploited to trick the library into executing arbitrary commands, supply chain security firm JFrog said.

此漏洞被跟踪为CVE-2024-5565（CVSS评分：8.1），与"ask"函数中的提示注入案例有关，可以被利用来欺骗库执行任意命令，供应链安全公司JFrog表示。

Vanna is a Python-based machine learning library that allows users to chat with their SQL database to glean insights by "just asking questions" (aka prompts) that are translated into an equivalent SQL query using a large language model (LLM).

Vanna是一个基于Python的机器学习库，允许用户通过"仅仅提出问题"（即提示）与他们的SQL数据库进行交流，从而利用大型语言模型（LLM）将问题翻译成等效的SQL查询以获取洞见。

The rapid rollout of generative artificial intelligence (AI) models in recent years has brought to the fore the risks of exploitation by malicious actors, who can weaponize the tools by providing adversarial inputs that bypass the safety mechanisms built into them.

近年来生成人工智能（AI）模型的快速推出，凸显了恶意行为者利用这些工具的风险，他们可以通过提供绕过内置安全机制的对抗性输入来武装这些工具。

One such prominent class of attacks is prompt injection, which refers to a type of AI jailbreak that can be used to disregard guardrails erected by LLM providers to prevent the production of offensive, harmful, or illegal content, or carry out instructions that violate the intended purpose of the application.

其中一种突出的攻击类别是提示注入，指的是一种可以用来忽视LLM提供商设置的防范措施以防止生成冒犯性、有害或违法内容的AI越狱的攻击类型，或执行违反应用程序预期目的的指令。

Such attacks can be indirect, wherein a system processes data controlled by a third party (e.g., incoming emails or editable documents) to launch a malicious payload that leads to an AI jailbreak.

这类攻击可以是间接的，系统处理由第三方控制的数据（例如传入电子邮件或可编辑文档），以启动导致AI越狱的恶意有效负载。

They can also take the form of what's called a many-shot jailbreak or multi-turn jailbreak (aka Crescendo) in which the operator "starts with harmless dialogue and progressively steers the conversation toward the intended, prohibited objective."

它们还可以采取所谓的多次越狱或多轮越狱（也称为Crescendo）形式，操作员"从无害对话开始，逐渐引导对话朝着预期的、被禁止的目标方向发展。

This approach can be extended further to pull off another novel jailbreak attack known as Skeleton Key.

这种方法还可以进一步扩展以执行另一种称为Skeleton Key的新型越狱攻击。

"This AI jailbreak technique works by using a multi-turn (or multiple step) strategy to cause a model to ignore its guardrails," Mark Russinovich, chief technology officer of Microsoft Azure, said. "Once guardrails are ignored, a model will not be able to determine malicious or unsanctioned requests from any other."

"这种AI越狱技术通过使用多轮（或多步）策略来使模型忽略其防护措施，"微软Azure首席技术官Mark Russinovich表示。"一旦防护措施被忽略，模型将无法区分恶意或未经授权的请求与其他请求。"

Skeleton Key is also different from Crescendo in that once the jailbreak is successful and the system rules are changed, the model can create responses to questions that would otherwise be forbidden regardless of the ethical and safety risks involved.

Skeleton Key与Crescendo不同，一旦越狱成功并更改系统规则，模型就可以对问题创建响应，而无论涉及的道德和安全风险如何，这些问题在其他情况下都是被禁止的。

"When the Skeleton Key jailbreak is successful, a model acknowledges that it has updated its guidelines and will subsequently comply with instructions to produce any content, no matter how much it violates its original responsible AI guidelines," Russinovich said.

"当Skeleton Key越狱成功时，模型会确认已更新其指南，并随后遵守指令以生成任何内容，无论违反其原始负责任AI指南多少，"Russinovich说。

"Unlike other jailbreaks like Crescendo, where models must be asked about tasks indirectly or with encodings, Skeleton Key puts the models in a mode where a user can directly request tasks. Further, the model's output appears to be completely unfiltered and reveals the extent of a model's knowledge or ability to produce the requested content."
与其他越狱工具如 Crescendo 不同，后者需要通过间接提问或编码方式来请求任务，Skeleton Key 让模型进入一种用户可以直接请求任务的模式。此外，模型的输出似乎完全没有经过过滤，揭示了模型知识的广度或生成所请求内容的能力。

The latest findings from JFrog – also independently disclosed by Tong Liu – show how prompt injections could have severe impacts, particularly when they are tied to command execution.

JFrog最新的发现，同时由Tong Liu独立披露，展示了提示注入可能会造成严重影响，特别是当它们与命令执行相关联时。

CVE-2024-5565 takes advantage of the fact that Vanna facilitates text-to-SQL Generation to create SQL queries, which are then executed and graphically presented to the users using the Plotly graphing library.

CVE-2024-5565利用了Vanna促进文本到SQL生成以创建SQL查询的事实，然后使用Plotly图形库执行并以图形方式呈现给用户。

This is accomplished by means of an "ask" function – e.g., vn.ask("What are the top 10 customers by sales?") – which is one of the main API endpoints that enables the generation of SQL queries to be run on the database.

这是通过"ask"函数实现的，例如vn.ask("销售排名前10的客户是谁？")，这是主要API端点之一，可以生成要在数据库上运行的SQL查询。

The aforementioned behavior, coupled with the dynamic generation of the Plotly code, creates a security hole that allows a threat actor to submit a specially crafted prompt embedding a command to be executed on the underlying system.

上述行为，加上Plotly代码的动态生成，创造了一个安全漏洞，允许威胁行为者提交一个特别制作的提示，嵌入一个要在底层系统上执行的命令。

"The Vanna library uses a prompt function to present the user with visualized results, it is possible to alter the prompt using prompt injection and run arbitrary Python code instead of the intended visualization code," JFrog said.

"Vanna库使用提示函数向用户展示可视化结果，可以通过提示注入修改提示并运行任意Python代码而不是预期的可视化代码，"JFrog表示。

"Specifically, allowing external input to the library's 'ask' method with 'visualize' set to True (default behavior) leads to remote code execution."

"具体来说，允许外部输入到库的'ask'方法，并将'visualize'设置为True（默认行为）会导致远程代码执行。"

Following responsible disclosure, Vanna has issued a hardening guide that warns users that the Plotly integration could be used to generate arbitrary Python code and that users exposing this function should do so in a sandboxed environment.

在负责披露后，Vanna发布了一个警告用户的<硬化指南>，警告用户Plotly集成可能被用于生成任意Python代码，并且暴露此功能的用户应该在一个沙箱环境中执行。

"This discovery demonstrates that the risks of widespread use of GenAI/LLMs without proper governance and security can have drastic implications for organizations," Shachar Menashe, senior director of security research at JFrog, said in a statement.

"这一发现表明，未经适当治理和安全性的GenAI/LLM广泛使用可能对组织产生严重影响，"JFrog的安全研究高级主任Shachar Menashe在一份声明中表示。

"The dangers of prompt injection are still not widely well known, but they are easy to execute. Companies should not rely on pre-prompting as an infallible defense mechanism and should employ more robust mechanisms when interfacing LLMs with critical resources such as databases or dynamic code generation."

"提示注入的危险仍未受到广泛关注，但很容易执行。公司不应依赖预提示作为不可靠的防御机制，并在将LLM与关键资源（如数据库或动态代码生成）进行接口时应采用更强大的机制。

参考资料

[1]https://thehackernews.com/2024/06/prompt-injection-flaw-in-vanna-ai.html

原文始发于微信公众号（知机安全）：Vanna AI中的提示注入漏洞暴露数据库面临RCE攻击