音频处理问题 #6044
Description
源码中有关audio的处理在哪？我要如何保证模型的输入一定包含音频呢？
我的数据格式如下：
{"query": "What are the two main types of sounds heard in the audio, in the order they appear? ", "response": "Sounds of playing with a soft material followed by an electronic musical jingle", "audios": ["/home/master/audio_16k/7ZrxeJR5XbY_000203.wav"]}
运行的是GRPO训练，代码如下：
python -m torch.distributed.run \
--nproc_per_node 2 \
/home//swift/cli/rlhf.py \
--rlhf_type grpo \
--model /home/Qwen2-Audio-7B-Instruct \
--reward_funcs format \
--max_completion_length 2056 \
--soft_cache_length 256 \
--epsilon 0.2 \
--epsilon_high 0.28 \
--dynamic_sample true \
--overlong_filter true \
--max_resample_times 10 \
--use_vllm true \
--vllm_gpu_memory_utilization 0.8 \
--train_type lora \
--torch_dtype bfloat16 \
--dataset ${DATA_FILE} \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 2 \
--learning_rate 1e-6 \
--eval_steps 10000 \
--save_steps 200 \
--max_steps 2000 \
--save_total_limit 10 \
--logging_steps 5 \
--warmup_ratio 0.05 \
--dataloader_num_workers 1 \
--dataset_num_proc 1 \
--num_generations 4 \
--temperature 1.0 \
--top_p 1.0 \
--deepspeed zero2 \
--log_completions true \
--num_iterations 8 \
--report_to tensorboard \
--beta 0.0 \
{"step": ["1", "1", "1", "1"], "prompt": ["<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nThe detailed description mentions "subtle variations in the texture and intensity" of the chopping sounds, as well as "occasional brief pauses or changes in the rhythm." What does the description suggest as a potential reason for these variations in texture and intensity? <|im_end|>\n<|im_start|>assistant\n",
训练结束后的completions.jsonl文件打印出来的prompt好像没有加上?