Thank you again for your continued interest.

Our model was trained on the MIMIC-CXR dataset and directly applied to the Two-view CXR dataset for zero-shot report generation. Specifically:

For samples originating from MIMIC-CXR, this is not strictly zero-shot.
For samples derived from IU X-ray (included in Two-view CXR), the model has not seen them during training, so these cases are evaluated in a zero-shot setting.

You can find the full results of this experiment under:
MLRG/generated-radiology-reports/“zero-shot” report generation on the Two-view CXR Dataset

To evaluate the model performance specifically on IU X-ray samples, we extracted the relevant subset (total: 1223 samples) using the dicom_ids.

We modified the following script for this evaluation:
script/MIMIC-CXR/run_cxr_test_mlrg_v1011.sh
by replacing the ann_path with the Two-view CXR annotation file (i.e., hugging face). test_ckpt_path is https://huggingface.co/MK-runner/MLRG/blob/main/mimic-cxr/finetune/best_model.ckpt

The performance metrics on IU X-ray (zero-shot setting) are as follows:

# Sample count: 1223
{
  'F1-Radgraph-partial': 0.2991,
  'chexbert_5_micro_f1': 0.3163,
  'chexbert_5_macro_f1': 0.3338,
  'chexbert_all_micro_p': 0.5645,
  'chexbert_all_micro_r': 0.5923,
  'chexbert_all_micro_f1': 0.5781,
  'chexbert_all_macro_p': 0.3645,
  'chexbert_all_macro_r': 0.2118,
  'chexbert_all_macro_f1': 0.2281,
  'BLEU_1': 0.3979,
  'BLEU_2': 0.2278,
  'BLEU_3': 0.1426,
  'BLEU_4': 0.0934,
  'METEOR': 0.1601,
  'ROUGE_L': 0.3027,
  'CIDEr': 0.1653
}

Hope this helps! A star would be much appreciated.

Please let us know if you need any further clarification.

Best regards,
Kang Liu

Iu-xray #10

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions