环境安装

推理环境安装

# 下载repo
git clone https://github.com/OpenGVLab/InternVL.git

# conde建一个虚拟环境
conda create -n internvl python==3.10.14
conda activate internvl

cd InternVL/
pip install -r requirements.txt
pip install flash-attn --no-build-isolation

如果flash-attn==2.3.6安装失败，可能是flash-attn找不到对应nvcc版本。

如果版本不对，重新安装nvcc

sudo apt-get remove --purge nvidia-cuda-toolkit
# 或者离线安装run文件（更稳定）
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install cuda-toolkit-12-1

安装后续库

pip install flash-attn --no-build-isolation
pip install -U openmim
mim install mmcv-full

微调环境安装

Tutorial/docs/L1/XTuner at camp3 · InternLM/Tutorial

注意的是，可以将xtuner的分支改到最新tag，这样pip安装才会出现可执行的xtuner。

建议微调和推理环境分开。

数据集处理

Chat Data Format — InternVL

如果需要输入坐标信息，可以将坐标信息归一化到[0, 1000]之间的整型，token中有0到1000的数字，这样文本的token数量不会太多

微调

关键设计决策（先说结论）

视觉 LoRA 而非解冻全部视觉参数：你当前 freeze_visual_encoder=True，配合 visual_encoder_lora 是正确的方向 —— 冻结 base 参数、只训练 LoRA adapter，适合少样本场景且显存友好。
target_modules 用子串匹配就够：你列出的模块里，视觉的 attention 是 attn.qkv，mlp 是 mlp.fc1/fc2，用 ["attn.qkv","mlp.fc1","mlp.fc2"] 即可匹配所有层。
少样本时减小 LoRA 容量、再加低 LR 与更多 epoch：把视觉 LoRA 的 r / alpha 调小（例如 r=32, alpha=32），学习率降到 1e-5 ~ 5e-6，并增加 epoch（3~10，视样本数）或用更多 gradient_accumulation_steps。
把 bbox 显式写入训练 prompt：训练样本的文本（context）应包含 bbox 信息（格式化为 boxes: [[x1,y1,x2,y2,label], ...] 或自然语言描述），模型依据该 context 学习把视觉特征和 bbox 对齐。
数据量小 -> 强正则/数据增强/重复采样：对图片做轻微 augment（翻转/色彩 jitter）、使用重复采样或 upsampling 生成足够 batch。

遇到问题

报错信息

09/18 12:55:43 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
09/18 12:55:43 - mmengine - INFO - Checkpoints will be saved to /mnt/d/Study/codes/home_codes/internvl_codes/xtuner/20250917_carton/work_dirs/20250918_internvl3.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/d/Study/codes/home_codes/internvl_codes/xtuner/xtuner/tools/train.py", line 392, in <module>
[rank0]:     main()
[rank0]:   File "/mnt/d/Study/codes/home_codes/internvl_codes/xtuner/xtuner/tools/train.py", line 388, in main
[rank0]:     runner.train()
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/mmengine/runner/_flexible_runner.py", line 1200, in train
[rank0]:     model = self.train_loop.run()  # type: ignore
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/mmengine/runner/loops.py", line 289, in run
[rank0]:     self.run_iter(data_batch)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/mmengine/runner/loops.py", line 313, in run_iter
[rank0]:     outputs = self.runner.model.train_step(
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/mmengine/_strategy/deepspeed.py", line 133, in train_step
[rank0]:     losses = self._run_forward(data, mode='loss')
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/mmengine/_strategy/deepspeed.py", line 176, in _run_forward
[rank0]:     results = self.model(**data, mode=mode)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank0]:     ret_val = func(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2131, in forward
[rank0]:     loss = self.module(*inputs, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1879, in _call_impl
[rank0]:     return inner()
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1827, in inner
[rank0]:     result = forward_call(*args, **kwargs)
[rank0]:   File "/mnt/d/Study/codes/home_codes/internvl_codes/xtuner/xtuner/model/internvl.py", line 229, in forward
[rank0]:     outputs = self._llm_forward(
[rank0]:   File "/mnt/d/Study/codes/home_codes/internvl_codes/xtuner/xtuner/model/internvl.py", line 267, in _llm_forward
[rank0]:     vit_embeds = self.model.extract_feature(pixel_values)
[rank0]:   File "/home/sun/.cache/huggingface/modules/transformers_modules/InternVL3-2B/modeling_internvl_chat.py", line 186, in extract_feature
[rank0]:     vit_embeds = self.vision_model(
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/peft/peft_model.py", line 2721, in forward
[rank0]:     return self.base_model(
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/sun/miniconda3/envs/xtuner/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 197, in forward
[rank0]:     return self.model.forward(*args, **kwargs)
[rank0]: TypeError: InternVisionModel.forward() got an unexpected keyword argument 'input_ids'

明白了，你 _llm_forward 里已经把视觉模型和语言模型都混合在一起了，但问题是 PEFT 封装会把 input_ids、attention_mask 等参数传给整个模型，导致 InternVisionModel.forward() 报错。

核心思路是：视觉模型只接受 pixel_values，语言模型只接受 inputs_embeds 或 input_ids，两者不要混用参数。

你可以这样改 _llm_forward：

def _llm_forward(
    self,
    pixel_values: torch.FloatTensor,
    input_ids: torch.LongTensor = None,
    attention_mask: Optional[torch.Tensor] = None,
    position_ids: Optional[torch.LongTensor] = None,
    image_flags: Optional[torch.LongTensor] = None,
    past_key_values: Optional[List[torch.FloatTensor]] = None,
    labels: Optional[torch.LongTensor] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, CausalLMOutputWithPast]:

    return_dict = return_dict if return_dict is not None else self.model.config.use_return_dict

    # --- 1. 视觉前向，只有 pixel_values ---
    vit_embeds = self.model.extract_feature(pixel_values)
    if image_flags is not None:
        image_flags = image_flags.squeeze(-1)
        vit_embeds = vit_embeds[image_flags == 1]
    vit_batch_size = pixel_values.shape[0]

    # --- 2. 文本嵌入 ---
    input_embeds = self.model.language_model.get_input_embeddings()(input_ids).clone()

    B, N, C = input_embeds.shape
    input_embeds = input_embeds.reshape(B * N, C)

    input_ids_flat = input_ids.reshape(B * N)
    selected = input_ids_flat == self.model.img_context_token_id

    if selected.any():
        vit_embeds = vit_embeds.reshape(-1, C)
        n_token = selected.sum()
        input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds[:n_token]

    input_embeds = input_embeds.reshape(B, N, C)

    # --- 3. 语言模型前向 ---
    outputs = self.model.language_model(
        inputs_embeds=input_embeds,
        attention_mask=attention_mask,
        position_ids=position_ids,
        past_key_values=past_key_values,
        use_cache=use_cache,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    logits = outputs.logits
    loss = None
    if labels is not None:
        shift_logits = logits[..., :-1, :].contiguous()
        shift_labels = labels[..., 1:].contiguous()
        loss_fct = CrossEntropyLoss()
        shift_logits = shift_logits.view(-1, self.model.language_model.config.vocab_size)
        shift_labels = shift_labels.view(-1).to(shift_logits.device)
        loss = loss_fct(shift_logits, shift_labels)

    if not return_dict:
        output = (logits,) + outputs[1:]
        return (loss,) + output if loss is not None else output

    return CausalLMOutputWithPast(
        loss=loss,
        logits=logits,
        past_key_values=outputs.past_key_values,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

✅ 改动重点：

视觉模型单独前向，不传 input_ids、attention_mask 等。
文本模型单独使用 inputs_embeds 前向。
LoRA/PEFT 调用 _llm_forward 时不会把不需要的参数传给视觉模型。

推理

# prompt = """Detect all objects in the image and output only bounding boxes as a list of integers.
# Each box should tightly enclose the object but slightly expand to fully cover it.
# Prioritize larger objects and boxes with higher confidence scores.
# Avoid duplicate or overlapping boxes; each object should appear only once.
# Format each box as [x1, y1, x2, y2] (top-left to bottom-right).
# Coordinates normalized to 0-1000, integers, x1<x2, y1<y2.
# Return [] if no objects are found.
# Do NOT include any Markdown, quotes, or extra text.
# Example: [[225, 315, 531, 633], [352, 836, 546, 995]]"""

对于多边形的推理和微调

LoRA 可以让模型学会 polygon 推理，但需要大量高质量 polygon 标注 + 格式约束，且训练难度高。
更实用的办法 是用 InternVL3 做 bbox 检测，再接 SAM / 语义分割网络 → polygon。
如果你只是想快速落地，建议优先用 检测 + SAM + mask2polygon，而不是直接逼 LLM 输出 polygon。

环境安装#

推理环境安装#

微调环境安装#

数据集处理#

微调#

关键设计决策（先说结论）#

推理#

环境安装

推理环境安装

微调环境安装

数据集处理

微调

关键设计决策（先说结论）

推理