I am completely new to pytorch. Thus, please bear with me that my question may be very naive. I am merely playing this for fun with my personal project. And I checked the questions in FAQ. At the sight of How do I get the accuracy of an unbalanced dataset or segmentation task?, it seems to be my question, but taking a closer look at it, they are different. So here is my question.
I want to test extracting the summary of this YouTube video using pytorch. The code is below
from transformers import AutoProcessor, AutoModelForImageTextToText
import torch
model_path = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
model_path,
torch_dtype=torch.bfloat16,
# _attn_implementation="flash_attention_2"
).to("cpu") # cuda
# model.config.num_frames = 64 # 96 # frames count
for name, param in model.named_parameters():
if param.dtype == torch.float32:
param.data = param.data.to(torch.bfloat16)
video_path = "/tmp/in.mp4" # This is the YouTube video mentioned above
messages = [
{
"role": "user",
"content": [
{"type": "video", "path": video_path, "fps": 1},
{"type": "text", "text": "Describe this video in detail"}
# {"type": "text", "text": "Focus only on describing the key dramatic action or notable event occurring in this video segment. Skip general context or scene-setting details unless they are crucial to understanding the main action."}
]
},
]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to("cpu") # cuda
for key, value in inputs.items():
if torch.is_floating_point(value): # Convert only float tensors
inputs[key] = value.to(torch.bfloat16)
generated_ids = model.generate(**inputs, max_new_tokens=512) # original: 254
generated_texts = processor.batch_decode(
generated_ids,
skip_special_tokens=True,
)
print('OUT: ', generated_texts[0])
At the beginning of summary, the description looks ok. After a few sentences later, it is like broken radio that kept repeating sentences:
The video continues with a close-up of the meat being stirred in the pan, and the text explains the cooking process. The next scene shows a close-up of the meat being stirred in the pan, and the text explains the cooking process. … (the rest sentences is just repeating)
I attempt to increase the frame count model.config.num_frames = 64 # 96, to change the prompt tone to {"type": "text", "text": "Focus only on describing the key dramatic action or notable event occurring in this video segment. Skip general context or scene-setting details unless they are crucial to understanding the main action."}, or to switch max_new_tokens from from 254 to 512, and so on. But the difference is either the summary becomes extremely short (this looks affected by the message prompt tone), or the repeating sentence become lengthier e.g. the same sentence repeats from 3 to 10.
Which parameters to adjust so that I can extract more accurate summary for a video? Thanks.