⚠ This page is served via a proxy. Original site: https://github.com
This service does not collect credentials or authentication data.
Skip to content

FEAT: Sora target: support remix, image-to-video #1341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
varunj-msft wants to merge 3 commits into
base: main
Choose a base branch
from
Open

FEAT: Sora target: support remix, image-to-video #1341

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0de7a0a
Sora target: support remix, image-to-video
varunj786 Jan 30, 2026
5efe8cc
Update files for Sora target PR
varunj786 Feb 9, 2026
fbcdc47
Update test and notebook
varunj786 Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 112 additions & 1 deletion doc/code/targets/4_openai_video_target.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,24 @@
"source": [
"# 4. OpenAI Video Target\n",
"\n",
"This example shows how to use the video target to create a video from a text prompt.\n",
"`OpenAIVideoTarget` supports three modes:\n",
"- **Text-to-video**: Generate a video from a text prompt.\n",
"- **Remix**: Create a variation of an existing video (using `video_id` from a prior generation).\n",
"- **Image-to-video**: Use an image as the first frame of the generated video.\n",
"\n",
"Note that the video scorer requires `opencv`, which is not a default PyRIT dependency. You need to install it manually or using `pip install pyrit[opencv]`."
]
},
{
"cell_type": "markdown",
"id": "0ebc1dc5",
"metadata": {},
"source": [
"## Text-to-Video\n",
"\n",
"This example shows the simplest mode: generating video from text prompts, with scoring."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -762,6 +775,104 @@
"for result in results:\n",
" await ConsoleAttackResultPrinter().print_result_async(result=result, include_auxiliary_scores=True) # type: ignore"
]
},
{
"cell_type": "markdown",
"id": "e21b0718",
"metadata": {},
"source": [
"## Remix (Video Variation)\n",
"\n",
"Remix creates a variation of an existing video. After any successful generation, the response\n",
"includes a `video_id` in `prompt_metadata`. Pass this back via `prompt_metadata={\"video_id\": \"<id>\"}` to remix."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0a29f796",
"metadata": {},
"outputs": [],
"source": [
"from pyrit.models import Message, MessagePiece\n",
"\n",
"# Use the same target from above, or create a new one\n",
"remix_target = OpenAIVideoTarget()\n",
"\n",
"# Step 1: Generate a video\n",
"text_piece = MessagePiece(\n",
" role=\"user\",\n",
" original_value=\"A bird flying over a lake at sunset\",\n",
")\n",
"result = await remix_target.send_prompt_async(message=Message([text_piece])) # type: ignore\n",
"response = result[0].message_pieces[0]\n",
"print(f\"Generated video: {response.converted_value}\")\n",
"video_id = response.prompt_metadata[\"video_id\"]\n",
"print(f\"Video ID for remix: {video_id}\")\n",
"\n",
"# Step 2: Remix using the video_id\n",
"remix_piece = MessagePiece(\n",
" role=\"user\",\n",
" original_value=\"Make it a watercolor painting style\",\n",
" prompt_metadata={\"video_id\": video_id},\n",
")\n",
"remix_result = await remix_target.send_prompt_async(message=Message([remix_piece])) # type: ignore\n",
"print(f\"Remixed video: {remix_result[0].message_pieces[0].converted_value}\")"
]
},
{
"cell_type": "markdown",
"id": "a7f0708b",
"metadata": {},
"source": [
"## Image-to-Video\n",
"\n",
"Use an image as the first frame of the generated video. The input image dimensions must match\n",
"the video resolution (e.g. 1280x720). Pass both a text piece and an `image_path` piece in the same message."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b417ec67",
"metadata": {},
"outputs": [],
"source": [
"import uuid\n",
"\n",
"# Create a simple test image matching the video resolution (1280x720)\n",
"from PIL import Image\n",
"\n",
"from pyrit.common.path import HOME_PATH\n",
"\n",
"sample_image = HOME_PATH / \"assets\" / \"pyrit_architecture.png\"\n",
"resized = Image.open(sample_image).resize((1280, 720)).convert(\"RGB\")\n",
"\n",
"import tempfile\n",
"\n",
"tmp = tempfile.NamedTemporaryFile(suffix=\".jpg\", delete=False)\n",
"resized.save(tmp, format=\"JPEG\")\n",
"tmp.close()\n",
"image_path = tmp.name\n",
"\n",
"# Send text + image to the video target\n",
"i2v_target = OpenAIVideoTarget()\n",
"conversation_id = str(uuid.uuid4())\n",
"\n",
"text_piece = MessagePiece(\n",
" role=\"user\",\n",
" original_value=\"Animate this image with gentle camera motion\",\n",
" conversation_id=conversation_id,\n",
")\n",
"image_piece = MessagePiece(\n",
" role=\"user\",\n",
" original_value=image_path,\n",
" converted_value_data_type=\"image_path\",\n",
" conversation_id=conversation_id,\n",
")\n",
"result = await i2v_target.send_prompt_async(message=Message([text_piece, image_piece])) # type: ignore\n",
"print(f\"Image-to-video result: {result[0].message_pieces[0].converted_value}\")"
]
}
],
"metadata": {
Expand Down
84 changes: 83 additions & 1 deletion doc/code/targets/4_openai_video_target.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,18 @@
# %% [markdown]
# # 4. OpenAI Video Target
#
# This example shows how to use the video target to create a video from a text prompt.
# `OpenAIVideoTarget` supports three modes:
# - **Text-to-video**: Generate a video from a text prompt.
# - **Remix**: Create a variation of an existing video (using `video_id` from a prior generation).
# - **Image-to-video**: Use an image as the first frame of the generated video.
#
# Note that the video scorer requires `opencv`, which is not a default PyRIT dependency. You need to install it manually or using `pip install pyrit[opencv]`.

# %% [markdown]
# ## Text-to-Video
#
# This example shows the simplest mode: generating video from text prompts, with scoring.

# %%
from pyrit.executor.attack import (
AttackExecutor,
Expand Down Expand Up @@ -65,3 +73,77 @@

for result in results:
await ConsoleAttackResultPrinter().print_result_async(result=result, include_auxiliary_scores=True) # type: ignore

# %% [markdown]
# ## Remix (Video Variation)
#
# Remix creates a variation of an existing video. After any successful generation, the response
# includes a `video_id` in `prompt_metadata`. Pass this back via `prompt_metadata={"video_id": "<id>"}` to remix.

# %%
from pyrit.models import Message, MessagePiece

# Use the same target from above, or create a new one
remix_target = OpenAIVideoTarget()

# Step 1: Generate a video
text_piece = MessagePiece(
role="user",
original_value="A bird flying over a lake at sunset",
)
result = await remix_target.send_prompt_async(message=Message([text_piece])) # type: ignore
response = result[0].message_pieces[0]
print(f"Generated video: {response.converted_value}")
video_id = response.prompt_metadata["video_id"]
print(f"Video ID for remix: {video_id}")

# Step 2: Remix using the video_id
remix_piece = MessagePiece(
Copy link
Contributor

@romanlutz romanlutz Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the image target, we pass text and image together and then the image gets edited based on the text. That is kind of similar to remix. I understand why sending the video would be a bad idea. That said, I would like us to think through how multimodal attacks with this would work (similarly with the image editing functionality). One could alternatively envision passing text + video and the target handles it from there (i.e., ignores the video but just takes the video_id and attaches it to the request). Putting the video_id into metadata is a little bit of a hack so I just want us to be conscious while making this choice.

role="user",
original_value="Make it a watercolor painting style",
prompt_metadata={"video_id": video_id},
)
remix_result = await remix_target.send_prompt_async(message=Message([remix_piece])) # type: ignore
print(f"Remixed video: {remix_result[0].message_pieces[0].converted_value}")

# %% [markdown]
# ## Image-to-Video
#
# Use an image as the first frame of the generated video. The input image dimensions must match
# the video resolution (e.g. 1280x720). Pass both a text piece and an `image_path` piece in the same message.

# %%
import uuid

# Create a simple test image matching the video resolution (1280x720)
from PIL import Image

from pyrit.common.path import HOME_PATH

sample_image = HOME_PATH / "assets" / "pyrit_architecture.png"
resized = Image.open(sample_image).resize((1280, 720)).convert("RGB")

import tempfile

tmp = tempfile.NamedTemporaryFile(suffix=".jpg", delete=False)
resized.save(tmp, format="JPEG")
tmp.close()
image_path = tmp.name

# Send text + image to the video target
i2v_target = OpenAIVideoTarget()
conversation_id = str(uuid.uuid4())

text_piece = MessagePiece(
role="user",
original_value="Animate this image with gentle camera motion",
conversation_id=conversation_id,
)
image_piece = MessagePiece(
role="user",
original_value=image_path,
converted_value_data_type="image_path",
conversation_id=conversation_id,
)
result = await i2v_target.send_prompt_async(message=Message([text_piece, image_piece])) # type: ignore
print(f"Image-to-video result: {result[0].message_pieces[0].converted_value}")
24 changes: 24 additions & 0 deletions pyrit/models/message.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,30 @@ def get_piece(self, n: int = 0) -> MessagePiece:

return self.message_pieces[n]

def get_pieces_by_type(self, *, data_type: PromptDataType) -> list[MessagePiece]:
"""
Return all message pieces matching the given data type.

Args:
data_type: The converted_value_data_type to filter by.

Returns:
A list of matching MessagePiece objects (may be empty).
"""
return [p for p in self.message_pieces if p.converted_value_data_type == data_type]
Copy link
Contributor

@romanlutz romanlutz Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converted value data type...

I think this is fine BUT we may want to have the same for original value data type at some point in which case we'd have to probably change the param name. That is technically breaking so maybe we just want to generalize this to have two input params (original_value_data_type, converted_value_data_type) from the start?

I know it's not something you need here, just trying to think ahead.


def get_piece_by_type(self, *, data_type: PromptDataType) -> Optional[MessagePiece]:
"""
Return the first message piece matching the given data type, or None.

Args:
data_type: The converted_value_data_type to filter by.

Returns:
The first matching MessagePiece, or None if no match is found.
"""
return next((p for p in self.message_pieces if p.converted_value_data_type == data_type), None)

@property
def api_role(self) -> ChatMessageRole:
"""
Expand Down
Loading
Loading