medical_report_gen

KO | EN

gitlite — search

#python #hacktoberfest #ai #typescript #llm #android #rust #macos #awesome-list #pytorch #security #ios #mcp #openai

medical_report_gen

(★ 2)

Anatomy-guided Cross-modal Fusion for automated radiology report generation from DICOM CT scans. Built with CT-CLIP, MedSAM, GatorTron, and LLaMA-3 (LoRA).

medical_report_gen Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Medical Report Generation System

Anatomy-guided Cross-modal Fusion for automated medical report generation from DICOM CT scans.

**Architecture**: CT-CLIP + MedSAM + GatorTron + LLaMA-3 (LoRA)

```
Input: DICOM series + Clinical text
    |
[CT-CLIP]         [MedSAM]          [GatorTron]
Visual encoder    Anatomy masks     Text encoder
(FREEZE)          (FREEZE)          (FREEZE)
    |                 |                 |
Region features <-- mask crop     Text chunk features
    |_________________________________|
         Anatomy-guided Cross-Attention
              (TRAINABLE)
                   |
          Adaptive Gate Fusion
                   |
         LLaMA-3-8B + LoRA adapters
              (LoRA TRAINABLE)
                   |
Output: Findings -> Impression -> Recommendation
```

## Prerequisites

- Python >= 3.10
- CUDA-capable GPU (recommended: >= 24GB VRAM)
- [uv](https://docs.astral.sh/uv/) package manager

## Quick Start with uv

### 1. Install uv (if not already installed)

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

### 2. Create environment and install dependencies

```bash
cd medical_report_gen

# Create virtual environment and install core dependencies
uv venv --python 3.10
source .venv/bin/activate

# Install the project in editable mode
uv pip install -e .

# Or install with optional dependency groups:
uv pip install -e ".[dev]"          # + pytest, ruff, jupyter
uv pip install -e ".[serving]"     # + FastAPI, Neo4j, Qdrant
uv pip install -e ".[all]"         # everything
```

### 3. Setup environment variables

```bash
cp .env.example .env
# Edit .env with your API keys:
#   WANDB_API_KEY, HF_TOKEN, NEO4J_URI, etc.
```

### 4. Download pretrained models

```bash
uv run python scripts/download_models.py --all

# Or download individually:
uv run python scripts/download_models.py --model ct_clip
uv run python scripts/download_models.py --model medsam
uv run python scripts/download_models.py --model gatortron
uv run python scripts/download_models.py --model llama3
```

## Development Roadmap

### Stage 1 - Data Pipeline

```bash
# Preprocess raw DICOM data
uv run python scripts/preprocess_dataset.py \
    --input_dir data/raw \
    --output_dir data/processed
```

### Stage 2 - Contrastive Pretraining (Align encoders)

```bash
uv run python scripts/train_stage1.py \
    --config configs/training/stage1_pretrain.yaml
```

### Stage 3 - Fusion + Decoder Training (Core work)

```bash
uv run python scripts/train_stage2.py \
    --config configs/training/stage2_finetune.yaml \
    --gpus 0,1,2,3
```

### Stage 4 - Evaluation

```bash
uv run python scripts/evaluate.py \
    --checkpoint outputs/best_checkpoint \
    --config configs/data/test.yaml
```

### Stage 5 - Ablation Study

```bash
uv run python scripts/run_ablation.py \
    --config configs/training/stage2_finetune.yaml
```

## Running Tests

```bash
uv run pytest

# With coverage
uv run pytest --cov=. --cov-report=term-missing
```

## Serving (Production API)

```bash
# Install serving dependencies
uv pip install -e ".[serving]"

# Start FastAPI server
uv run uvicorn serving.api:app --host 0.0.0.0 --port 8000 --workers 4
```

API endpoints:

| Method | Endpoint            | Description                     |
|--------|---------------------|---------------------------------|
| GET    | `/health`           | Health check                    |
| POST   | `/report/generate`  | Generate report from DICOM      |
| POST   | `/report/debate`    | Physician Q&A on generated report |

## Project Structure

```
medical_report_gen/
├── configs/                    # YAML configuration files
│   ├── base.yaml               # Main config
│   ├── model/                  # Per-component model configs
│   ├── data/                   # Data pipeline configs
│   └── training/               # Training stage configs
├── data/                       # Data loading & preprocessing
│   ├── dicom/                  # DICOM loader, preprocessor, augmentation
│   ├── text/                   # Text tokenizer, chunker, cleaner
│   ├── dataset.py              # Main Dataset class
│   └── collator.py             # Custom batch collation
├── models/                     # Model code
│   ├── encoders/               # Frozen pretrained encoders
│   ├── fusion/                 # Trainable fusion module (core contribution)
│   │   ├── region_extractor.py # Masked average pooling per anatomy region
│   │   ├── cross_attention.py  # Multi-head cross-attention (region x text)
│   │   ├── gate.py             # Adaptive gate per region
│   │   └── fusion_module.py    # Full fusion pipeline
│   ├── decoder/                # LLaMA-3 + LoRA decoder
│   └── pipeline.py             # End-to-end model
├── training/                   # Training loops & losses
├── evaluation/                 # Metrics (BLEU, ROUGE, BERTScore, clinical)
├── inference/                  # Inference pipeline & postprocessing
├── serving/                    # FastAPI production server + RAG
├── scripts/                    # CLI entry points
├── tests/                      # Unit tests
├── utils/                      # Logging, seeding, GPU, visualization
├── pyproject.toml              # Project config (uv/pip compatible)
├── Makefile                    # Shortcut commands
└── .env.example                # Environment variable template
```

## Configuration

All configs are in `configs/` using YAML format and loaded via OmegaConf.

Key config: `configs/base.yaml`

| Component       | Model            | Params | Trainable |
|-----------------|------------------|--------|-----------|
| Visual encoder  | CT-CLIP          | ~400M  | Frozen    |
| Segmentor       | MedSAM           | ~600M  | Frozen    |
| Text encoder    | GatorTron-8.9B   | ~345M  | Frozen    |
| Cross-attention | Custom           | ~30M   | Yes       |
| Gate fusion     | Custom MLP       | ~2M    | Yes       |
| Report decoder  | LLaMA-3-8B+LoRA | ~8M    | LoRA only |

## Using Make (alternative to uv run)

```bash
make setup          # Install dependencies
make download       # Download pretrained models
make preprocess     # Preprocess dataset
make train-stage1   # Stage 1 training
make train          # Stage 2 training
make eval           # Evaluate
make ablation       # Run ablation study
make serve          # Start API server
make test           # Run tests
make clean          # Clean __pycache__
```

medical_report_gen

medical_report_gen Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

medical_report_gen

medical_report_gen Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README