PosterSVG

KO | EN

gitlite — search

#pytorch #php #python #self-hosted #proxy #nextjs #gemini #claude-code #redis #agents

(★ 9)

Structured Semantic Vectorization for Editable Multimedia Design Images

파일 탐색기

ZIP 다운로드

- dataset_info_postersvg.json
- environment_liuzq.yml
- train_4xA100_mixed_svg_seg.yaml
- data_format.md
- OPEN_SOURCE_SCOPE.md
- - input.png
  - merged_pred.svg
  - README.md
  - rendered_pred.png
  - stage1_skeleton.svg
- README.md
- - __init__.py
- - manifest.csv
- - check_package.sh
- README.md
- requirements.txt
- build_mask.py
- build_mixed_sft.py
- build_sft1.py
- build_sft2.py
- enrich_sft2_with_seg_fields.py
- font_match.py
- inferDOM.py
- inpaint.py
- prebuild_font_cache.py
- raster_aux.py
- sft_stage_utils.py
- stage2_sampler.py
- step.py
- svg_cleanup_shared.py
- svg_io.py
- apply_llamafactory_patch.sh
- check_release.sh
- register_dataset.py
LICENSE
OPEN_SOURCE_MANIFEST.txt
README.md
requirements.txt

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Author:** [liuziqi](https://github.com/annriki)  
**Email:** [l15076526281@163.com](mailto:l15076526281@163.com)

## Project Overview

PosterSVG is a multimodal SVG generation framework for reconstructing poster images as structured, editable SVGs with text, vector elements, and image assets.

## Method Pipeline

PosterSVG follows a three-stage main pipeline:

1. **Stage-1 skeleton planning**: predict a global semantic SVG skeleton from the input poster image.
2. **Stage-2 node generation**: reconstruct role-conditioned SVG nodes for text, vector decorations, and image elements.
3. **SEG_MASK-guided SAM2 asset extraction**: use SEG_MASK supervision and SAM2 features to localize image assets and insert them into the final SVG.

## Repository Layout

- `postersvg/`: core PosterSVG scripts for SFT data construction, mask supervision, step decomposition, and single-image inference.
- `llamafactory_patch/`: tested LLaMA-Factory `0.9.5.dev0` source replacements required for `QwenVLWithSAM`, `use_sam_head`, segmentation loss, and `SEG_MASK` processing.
- `configs/`: main training config, dataset registration snippet, and environment references.
- `poster-svg-eval/`: evaluation code for the main table metrics, including validity, visual similarity, text occurrence F1, complexity diagnostics, and aggregate scores.
- `scripts/`: patch installation, dataset registration, and release validation helpers.
- `examples/`: a complete single-image inference case with the input poster, Stage-1 skeleton, extracted asset, final SVG, and rendered preview.
- `docs/`: data schema and open-source scope.

## Installation

The original experiments used Python 3.12.3, PyTorch 2.10.0 with CUDA 12.8, and LLaMA-Factory 0.9.5.dev0. Install a PyTorch build matching your CUDA driver first.

1. Clone LLaMA-Factory and check out a source tree compatible with version `0.9.5.dev0`.
2. From the PosterSVG root, apply the tested source replacements:

```bash
bash scripts/apply_llamafactory_patch.sh /path/to/LLaMA-Factory
pip install -e /path/to/LLaMA-Factory
```

The patch installer creates a timestamped backup inside the target LLaMA-Factory directory. The original source snapshot did not retain Git metadata, so an exact upstream commit is unavailable; see `llamafactory_patch/README.md`.

3. Install the PosterSVG runtime dependencies:

```bash
pip install -r requirements.txt
```

For an exact record of the original experiment environment, use `configs/environment_liuzq.yml` as a reference or create it with Conda.

4. Install the evaluation dependencies:

```bash
pip install -r poster-svg-eval/requirements.txt
```

Headless browser rendering is required for PosterSVG SVG-to-PNG rendering and mask construction. Install Chromium for Playwright if it is not already available:

```bash
pip install playwright
playwright install chromium
```

PosterSVG uses the available font collection when Chromium renders text masks. For posters containing non-Latin text, build a font metadata cache before inference; the font files and generated cache are not included in this repository:

```bash
python postersvg/prebuild_font_cache.py \
  --font_cn_dir /path/to/chinese_fonts \
  --font_com_dir /path/to/common_fonts \
  --out_cache /path/to/font_meta_cache.json
```

## Model and Environment Versions

The following information was queried from the original experiment environment under `/data1/fangsc/liuzq/`.

### Source Snapshot

- LLaMA-Factory path: `/data1/fangsc/liuzq/LLaMA-Factory`
- LLaMA-Factory version: `0.9.5.dev0`
- Git metadata: no `.git` metadata was found in the queried source snapshot, so no commit hash is available.
- PosterSVG training config: `configs/train_4xA100_mixed_svg_seg.yaml`

### Base Models

- Base VLM path: `/data1/fangsc/liuzq/Qwen3-VL-8B-Instruct`
  - Model name: `Qwen3-VL-8B-Instruct`
  - Architecture: `Qwen3VLForConditionalGeneration`
  - Model type: `qwen3_vl`
  - Text layers: `36`
  - Text hidden size: `4096`
  - Vision depth: `27`
  - Dtype in config: `bfloat16`
  - Config `transformers_version`: `4.57.0.dev0`
  - Generation config `transformers_version`: `4.56.0`
  - Local weight shards: `4` safetensor shards

- SAM2 path: `/data1/fangsc/liuzq/sam2-hiera-small`
  - Model name: `facebook/sam2-hiera-small`
  - Architecture: `Sam2VideoModel`
  - Model type: `sam2_video`
  - Image size: `1024`
  - Config dtype: `float32`
  - Config `transformers_version`: `4.56.0.dev0`
  - Local checkpoint files: `model.safetensors`, `sam2_hiera_small.pt`

- Evaluation DINO path: `/data1/fangsc/liuzq/hf_cache/dinov2-base-local`
  - Fallback model name in code: `facebook/dinov2-base`
  - Architecture: `Dinov2Model`
  - Model type: `dinov2`
  - Hidden size: `768`
  - Layers: `12`
  - Image size: `518`
  - Patch size: `14`
  - Config `transformers_version`: `4.31.0.dev0`

### Training Environment

The main training environment was `liuzq`.

- Platform: `Linux-5.15.0-122-generic-x86_64-with-glibc2.31`
- Python: `3.12.3`
- PyTorch: `2.10.0+cu128`
- Torch CUDA: `12.8`
- cuDNN: `91002`
- NVIDIA driver: `550.90.07`
- GPUs: `4 x NVIDIA A100-PCIE-40GB`
- Transformers: `5.0.0`
- Datasets: `3.0.0`
- Accelerate: `1.11.0`
- PEFT: `0.18.1`
- TRL: `0.24.0`
- DeepSpeed: `0.17.1`
- BitsAndBytes: `0.49.1`
- TorchVision: `0.25.0`
- TorchAudio: `2.10.0`
- NumPy: `2.4.1`
- Pillow: `11.3.0`
- lxml: `6.0.2`
- CairoSVG: `2.8.2`
- Playwright: `1.43.0`
- Chromium: used through Playwright for headless SVG-to-PNG rendering
- OpenCV Python: `4.12.0.88`
- Safetensors: `0.7.0`

Reference environment files:

- `configs/environment_liuzq.yml`: main training/inference environment.
- `poster-svg-eval/requirements.txt`: evaluation package dependencies.

### Auxiliary Assets Used in the Original Environment

These assets are not included in the repository, but the original experiment environment used local font resources for SVG text/font handling:

- `/data1/fangsc/liuzq/font_cn`: 9,837 font files, about 14 GB.
- `/data1/fangsc/liuzq/font_com`: 14,975 font files, about 6.8 GB.

## Model Preparation

Prepare the following external assets yourself before training or inference:

- Qwen3-VL-8B-Instruct base model.
- SAM2 Hiera Small model.
- A trained PosterSVG LoRA checkpoint for inference.

These models and checkpoints are not included in this repository. See `Model and Environment Versions` above for the exact local model configurations used in the original experiments.

## Dataset Release

The PosterSVG dataset is currently being prepared for public release.

**Status:** Coming soon  
**Download:** To be announced  
**Dataset license:** To be announced

The source code can be inspected and installed without the dataset. Training requires the released dataset or user-prepared data following `docs/data_format.md`.

## Data Preparation

Until the public dataset is available, this step can be skipped for code inspection and environment setup. Users may also prepare their own data following the schema below and `docs/data_format.md`.

PosterSVG uses LLaMA-Factory ShareGPT-style JSON records. The training data contains three types of records:

1. Stage-1 skeleton records: map the full poster image to a global semantic SVG skeleton.
2. Stage-2 node-level records: map a boxed full-image cue to one target SVG/XML node fragment.
3. SEG_MASK-enriched records: supervise complex image-asset nodes with mask and SAM2 feature paths.

A minimal Stage-1 record has the following structure:

```json
{
  "id": "example_case_0001",
  "images": [
    "/path/to/input.png"
  ],
  "conversations": [
    {
      "from": "human",
      "value": "<image>\nGenerate the empty poster skeleton SVG for the given poster image."
    },
    {
      "from": "gpt",
      "value": "<svg viewBox=\"0 0 1024 436\"><image bbox=\"[0,0,1024,436]\" data-role=\"complex_decor\"/><text bbox=\"[193,61,289,334]\" data-role=\"text\"/></svg>"
    }
  ],
  "canvas_size": [
    1024,
    436
  ],
  "roles": [
    "complex_decor",
    "text"
  ]
}
```

A minimal Stage-2 node-level record has the following structure:

```json
{
  "id": "example_case_0001_text0",
  "images": [
    "/path/to/boxed_full_image.png"
  ],
  "conversations": [
    {
      "from": "human",
      "value": "<image>\nCanvas Size: 1024x436\nTarget BBox: [193, 61, 289, 334]\nRole: text\nOutput only the target SVG/XML node."
    },
    {
      "from": "gpt",
      "value": "<text x=\"193\" y=\"96\" font-size=\"32\" fill=\"#111111\">Poster Title</text>"
    }
  ],
  "case_id": "example_case_0001",
  "canvas_size": [
    1024,
    436
  ],
  "bbox": [
    193,
    61,
    289,
    334
  ],
  "prompt_role": "text"
}
```

For complex foreground image nodes, the Stage-2 output uses `__SEG_MASK__`, and the record is enriched with mask supervision:

```json
{
  "id": "example_case_0002_img1",
  "images": [
    "/path/to/boxed_full_image.png"
  ],
  "conversations": [
    {
      "from": "human",
      "value": "<image>\nCanvas Size: 1024x1024\nTarget BBox: [0, 0, 1018, 1019]\nRole: complex_decor\nOutput only the target SVG/XML node."
    },
    {
      "from": "gpt",
      "value": "<image x=\"0\" y=\"0\" width=\"1018\" height=\"1019\" href=\"__SEG_MASK__\"/>"
    }
  ],
  "case_id": "example_case_0002",
  "canvas_size": [
    1024,
    1024
  ],
  "bbox": [
    0,
    0,
    1018,
    1019
  ],
  "prompt_role": "complex_decor",
  "seg_target": true,
  "seg_mask_path": "/path/to/crop.mask.png",
  "sam_feat_path": "/path/to/crop.sam_feat.pt"
}
```

The complete schema, preprocessing details, and optional bookkeeping fields are documented in `docs/data_format.md`.

After downloading the released mixed SFT JSON, register it without overwriting other LLaMA-Factory dataset entries:

```bash
python scripts/register_dataset.py /path/to/LLaMA-Factory \
  --data-file /path/to/mixed_train.json
```

## Training

Use the main training config currently included in this repository:

```bash
cd /path/to/LLaMA-Factory
FORCE_TORCHRUN=1 NPROC_PER_NODE=4 llamafactory-cli train /path/to/PosterSVG/configs/train_4xA100_mixed_svg_seg.yaml
```

Before running, update the config paths for your local Qwen3-VL, SAM2, dataset, and output directory if needed.

## Inference

Run single-image inference with the trained LoRA checkpoint:

```bash
python postersvg/inferDOM.py \
  --input_image /path/to/input.png \
  --out_dir /path/to/output_dir \
  --model_path /path/to/Qwen3-VL-8B-Instruct \
  --lora_path /path/to/postersvg_lora \
  --sam_model_path /path/to/sam2-hiera-small \
  --font_cn_dir /path/to/chinese_fonts \
  --font_com_dir /path/to/common_fonts \
  --font_meta_cache /path/to/font_meta_cache.json \
  --use_sam_head
```

The open-source inference path is fixed to the main method: boxed-full Stage-2 input, SAM2 mask features, browser-based SVG-to-PNG rendering for masks/PNGs, and local non-generative background inpainting and image asset extraction. The font cache is important for correct text-shaped occlusion masks, especially for Chinese posters.

## Example

The repository includes one complete result generated with the PosterSVG main-method checkpoint. The SVG keeps its extracted image asset as a relative path, so the example can be viewed or edited after cloning the repository.

| Input poster | PosterSVG reconstruction |
| --- | --- |
| ![Case 001 input](https://raw.githubusercontent.com/csust-sonie/PosterSVG/main/examples/case_001/input.png) | ![Case 001 reconstruction](https://raw.githubusercontent.com/csust-sonie/PosterSVG/main/examples/case_001/rendered_pred.png) |

See [`examples/case_001/`](examples/case_001/README.md) for the Stage-1 skeleton, editable merged SVG, extracted asset, and reproduction command.

## Evaluation

`poster-svg-eval/` contains the evaluation package used for the main table metrics.

Install the evaluation dependencies:

```bash
cd poster-svg-eval
pip install -r requirements.txt
```

Prepare a CSV manifest with one sample per row:

```csv
sample_id,gt_svg,pred_svg,gt_image,pred_image
example_001,examples/gt/example_001.svg,examples/pred/example_001.svg,examples/gt/example_001.png,examples/pred/example_001.png
```

Run batch evaluation:

```bash
python -m eval.cli.eval_batch \
  --manifest examples/manifest.csv \
  --out-dir outputs/example
```

The main outputs include `main_summary.json`, `main_results.csv`, full per-sample result files, and per-sample JSON records.

## Open-source Scope

This release contains the PosterSVG main-method source, training configuration, LLaMA-Factory modifications, and evaluation toolkit. It does not contain raw datasets, generated SFT JSON files, model weights, LoRA checkpoints, baseline repositories, or M0-M3 ablation data-generation code. See `docs/OPEN_SOURCE_SCOPE.md` for the file-level scope.

## Notes

Some scripts still contain local default paths from the original experiment environment. They are kept as experiment references and should be replaced with your own paths when running the code.

## Release Status

This repository currently releases the code, configuration files, and evaluation toolkit. Dataset availability is tracked in `Dataset Release` above. Trained LoRA checkpoints are not included in the current release.

Before publishing a release, run:

```bash
bash scripts/check_release.sh
```

## License

PosterSVG is released under the MIT License. See `LICENSE` for details.

PosterSVG

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README