VCF2Dis

KO | EN

gitlite — search

#python #typescript #llm #go #golang #deep-learning #linux #nodejs #java #windows #awesome #cli #pytorch #open-source

(★ 60)

VCF2Dis: an ultra-fast and efficient tool to calculate pairwise genetic distance and construct population phylogeny from VCF files

File Explorer

Download Latest Version (.zip)

- percentageboostrapTree.pl
- VCF2Dis
- VCF2Dis_multi
- VCF2Dis_single
- - Khuman.vcf.gz
  - OUT.png
  - pop.info
  - step1.sh
  - subample.list
  - vistreecode.r
- - single-copy.cds.fa.gz
  - single-copy.cds.phy.gz
  - step1.sh
- - step1.sh
  - step2.sh
- - aa.r
  - ChangFormat.pl
  - fastreeR_test.r
  - PickBi-Allele.pl
  - RunFig1.sh
  - sample.group
  - step1.sh
  - step2.sh
  - subsample203.list
  - vistreecode.r
- cc0_LICENSE
- - biallelic.snp.gz
  - cc0_license
  - download_atGoogleDrive
  - GetSampleVCF.pl
  - step1_download.sh
  - step2_newSite.sh
  - step3_newSample.sh
- - aa.r
  - ngsDist_test.sh
  - RunA.sh
  - RunB.sh
  - RunC.sh
  - RunD.sh
- - aa.r
  - md5sum.txt
  - ngsDist_test.sh
  - RunA.sh
  - RunB.sh
  - RunC.sh
  - RunD.sh
- - md5sum.txt
  - pop.info
  - RunFig1.sh
  - sample.group
  - subsample203.list
- - aa.r
  - md5sum.txt
  - pop.info
  - RunA.sh
  - RunB.sh
- - aa.r
  - Run.sh
- md5sum.txt
- readme.pdf
- readme.txt
- - comm.h
  - DataClass.h
  - kseq.h
- - gzstream.C
  - gzstream.h
- - VCF2Dis_multi.cpp
  - VCF2Dis_single.cpp
- - Readme
- GetFig.h
INSTALL.md
Install.NJ.cn.md
Install.NJ.en.md
LICENSE
make.sh
OtherWay2ConstructTree.md
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VCF2Dis
VCF2Dis: an ultra-fast and efficient tool to calculate pairwise genetic distance and construct population phylogeny from VCF files

The [VCF2Dis article](https://doi.org/10.1093/gigascience/giaf032) has been published in [GigaScience](https://doi.org/10.1093/gigascience/giaf032), please cited this article if possible

PMID: [40184433](https://pubmed.ncbi.nlm.nih.gov/40184433/)   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;    PMCID：[PMC11970368](https://pmc.ncbi.nlm.nih.gov/articles/PMC11970368/)    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;     DOI:[10.1093/gigascience/giaf032](https://doi.org/10.1093/gigascience/giaf032)

###  1) Install and Parameter
------------

* 1) Install

The new version will be updated and maintained in [hewm2008/VCF2Dis](https://github.com/hewm2008/VCF2Dis), please click below Link to download the latest version
[hewm2008/VCF2Dis](https://github.com/hewm2008/VCF2Dis)
 [Download](https://github.com/hewm2008/VCF2Dis/archive/v1.56.tar.gz)

#### Option 1 : Local compilation
Just `sh make.sh` to compile. The executable `VCF2Dis` can be found in the folder of `bin/VCF2Dis`
 For Linux /Unix and macOS
 <pre>
 tar -zxvf VCF2DisXXX.tar.gz # if Link do not work ,Try re-install [zlib]library
 cd VCF2DisXXX; # [zlib] and copy them to the library Dir
 sh make.sh; # VCF2Dis-xx/src/include/zlib
 ./bin/VCF2Dis
 </pre>
 
Performance note: `make.sh` uses `-O3 -march=native` by default. The optimization includes:
- Bit-packing genotype encoding — GT strings (e.g., "0/1") are packed into 16-bit integers, enabling branchless XOR-based comparison in the inner distance loop.
- OpenMP schedule(static) instead of schedule(dynamic) to reduce thread dispatch overhead for better multi-thread performance.
Fallback to `-O2` occurs automatically if `-O3 -march=native` is not supported on the platform.

**Note:** If fail to link,try to re-install the libraries [**_zlib_**](https://zlib.net/)
**Note:**: [R](https://www.r-project.org/) with [ape](https://cran.r-project.org/web/packages/ape/index.html), [dplyr](https://cran.r-project.org/web/packages/dplyr/index.html) and [ggtree](https://www.bioconductor.org/packages/release/bioc/html/ggtree.html) are recommended

#### Option 2: Docker container
You can use Docker to install and run VCF2Dis. Follow the steps below:
1. **Install Docker**: Ensure Docker is installed on your system. If not, you can install it by following the [Docker Official Documentation](https://docs.docker.com/get-docker/).
2. **Pull the Docker Image**: Use the following command to pull the VCF2Dis Docker image from the Alibaba Cloud Container Registry:
    ```sh
    docker pull registry.cn-shenzhen.aliyuncs.com/knight134/vcf2dis:v1.53e    ##  Docker image from the Alibaba Cloud Container Registry
    docker run -it --rm vcf2dis:v1.53e VCF2Dis                                ## After pulling the image, you can run the containe
    ```
#### Option 3: Singularity container   
1. **Install Singularity**: Ensure Singularity is installed on your system. If not, you can install it by following the [Singularity Official Documentation](https://sylabs.io/guides/3.5/user-guide/).
2. **Build the SIF File**: Use the following command to build a Singularity image file (SIF) from the Docker image:
    ```sh
    singularity build vcf2dis_1.53e.sif docker://registry.cn-shenzhen.aliyuncs.com/knight134/vcf2dis:v1.53e   # you can download follows 
    singularity exec  vcf2dis_1.53e.sif  VCF2Dis
    ```
3. **Download the SIF File**:Alternatively, you can download the built SIF file directly from the [vcf2dis_1.53e.sif](https://github.com/hewm2008/VCF2Dis/releases/download/v1.53/vcf2dis_1.53e.sif). Once downloaded, you can run it using Singularity.

* 2) Main parameter description:
```sh
	Usage: VCF2Dis -InPut <in.vcf> -OutPut <p_dis.mat>

-InPut <str> Input one or muti GATK VCF genotype File
		-OutPut <str> OutPut Sample p-Distance matrix

-InList <str> Input GATK muti-chr VCF Path List
		-SubPop <str> SubGroup SampleList of VCF File [ALLsample]
		-Rand <float> Probability (0-1] for each site to join Calculation [1]

\t\t-help                Show more help [hewm2008 v1.56s]

```
For more details, please use -help and see the [example](https://github.com/hewm2008/VCF2Dis/blob/main/example)
```sh
		-InFormat <str> Input File is [VCF/FA/PHY] Format,defaut: [VCF]
		-InSampleGroup <str> InFile of sample Group info,format(sample groupA)
		-TreeMethod <int> Construct Tree Method,1:NJ-tree 2:UPGMA-tree [1]
		-KeepMF Keep the Middle File diff & Use matrix
```

### 2) Example
 Three examples were provided in the directory of [example/Example*](https://github.com/hewm2008/VCF2Dis/blob/main/example)

###  1) an Example of nj-tree with no boostrap
------------

* 1) To Create the p_distance matrix and construct nj-tree newick tree

```
# 1.1) To new all the sample p_distance matrix and newick tree based VCF, run VCF2Dis directly
      ./bin/VCF2Dis	-InPut	in.vcf.gz	-OutPut p_dis.mat
      #  ./bin/VCF2Dis     -InPut  in.fa.gz	-OutPut p_dis.mat -InFormat  FA

# 1.2) To new sub group sample p_distance matrix and and newick tree ; put their sample name into File sample.list
      ./bin/VCF2Dis	-InPut	chr1.vcf.gz chr2.vcf.gz	-OutPut p_dis.mat  -SubPop  sample.list

# 1.3) To new group tree p_distance matrix and and newick tree ; pop.info file Format[sample group] (v1.56)
      ./bin/VCF2Dis	-InPut	in.vcf.gz	-OutPut p_dis.mat -InSampleGroup pop.info

```

* 2) Simple tree visualization (for advanced tree display and annotation please refer to `iTOL`, `Evolview`, `MEGA`)
 you will obtain the `p_dis.nwk` tree file and neighbor-joining tree in pdf format `p_dis.pdf` after VCF2Dis.

**Note:**:if you can't get the p_dis.nwk tree file but had the p_dis.mat, here are the [3 methods to get the tree file](https://github.com/hewm2008/VCF2Dis/blob/main/OtherWay2ConstructTree.md).

###  2) an Example of nj-tree with boostrap

* 1) Running multiple times by using  a method of sampling with replacement.
Users can randomly select  a part of the sites [-Rand] and construct a new nj-tree as above, and Repeat NN times [recommand NN=100]. X=(1,2....NN);

```sh
#!/bin/bash
NN=100
if [ "$#"  -eq  1 ]; then
	NN=$1
fi
for X in $(seq 1 $NN)
do
	./bin/VCF2Dis -InPut in.vcf.gz -OutPut p_dis_${X}.mat -Rand 0.25
    # PHYLIPNEW-3.69.650/bin/fneighbor -datafile p_dis_${X}.mat -outfile tree.out1_${X}.txt -matrixtype s -treetype n -outtreefile tree.out2_${X}.tre
done
```
* 2) Merge all the nj-tree and construct  and display a boostrap nj-tree. (For advanced display tree and annotation please refer to `iTOL`, `Evolview` and `MEGA`)

```sh
#!/bin/bash
NN=100
if [ "$#"  -eq  1 ]; then
  NN=$1
fi

cat  p_*.nwk  >    alltree_merge.tre   #  cat  tree*.tre  > alltree_merge.tre
PHYLIPNEW-3.69.650/bin/fconsense   -intreefile   alltree_merge.tre  -outfile out  -treeprint Y
perl  ./bin/percentageboostrapTree.pl    alltree_merge.treefile    $NN    Final_boostrap.tre  # NN is the input number

```
How to Install PHYLIPNEW please Click on [here](https://github.com/hewm2008/VCF2Dis/blob/main/Install.NJ.en.md) or Click on [here(Chinese)](https://github.com/hewm2008/VCF2Dis/blob/main/Install.NJ.cn.md)

------------
###  3.5) Performance Benchmark
------------
Benchmarked with large SNP dataset (5.4M sites × 57 samples, 1.5 GB gzipped VCF) on an x86-64 Linux server (8 threads):

| Version | Config | Wall time | Speedup vs v1.55 single | Memory |
|---------|--------|-----------|------------------------|--------|
| v1.55 | Single-thread `-O2` | 84.38s | 1.00× (baseline) | 3200 KB |
| v1.55 | Multi-thread `-O2` 8t | 65.41s | 1.29× | 3840 KB |
| v1.56 | Single-thread `-O2` | 71.75s | 1.18× | 3840 KB |
| v1.56 | Single-thread `-O3` | 72.23s | 1.17× | 3840 KB |
| v1.56 | Multi-thread `-O2` 8t | 63.01s | 1.34× | 3840 KB |
| **v1.56** | **Multi-thread `-O3` 8t** | **62.27s** | **1.36×** | 3840 KB |

Benchmarked with small dataset (623K sites × 57 samples, 12 MB gzipped):

| Version | Config | Wall time | Speedup vs v1.55 single | Memory |
|---------|--------|-----------|------------------------|--------|
| v1.55 | Single-thread `-O2` | 6.23s | 1.00× (baseline) | 3840 KB |
| v1.55 | Multi-thread `-O2` 8t | 4.69s | 1.33× | 3840 KB |
| v1.56 | Single-thread `-O2` | 3.95s | 1.58× | 3840 KB |
| **v1.56** | **Multi-thread `-O3` 8t** | **4.06s** | **1.53×** | 3840 KB |

**Key optimizations:**
- Bit-packing: GT strings encoded as `uint16_t`, inner loops use XOR + branchless diff counting (up to 58% single-thread speedup in compute-bound scenarios).
- OpenMP schedule(static): Reduced thread dispatch overhead from O(N·samples) dynamic work items to O(N/Block) static chunks (2× multi-thread speedup vs schedule(dynamic) for small sample counts).

**Best config per use case:**
- Small samples (&lt;100): `VCF2Dis_single` (single-thread, faster than multi due to lower overhead)
- Large samples (&gt;500): `VCF2Dis_multi -Threads 8` (multi-thread, ~4-8× speedup over single-thread)

###  4) Introduction
------------
The formula for calculating p-distance between indivisuals from VCF SNP datasets was listed below:

D_ij=(1/L) * [(sum(d(l)_ij))]

Where L is the length of regions where SNPs can be identified, and given the alleles at position `l` are A/C between sample `i` and sample `j`:
```
 d(l)_ij=0.0 if the genotypes of the two individuals were AA and AA;
 d(l)_ij=0.5 if the genotypes of the two individuals were AA and AC;
 d(l)_ij=0.0 if the genotypes of the two individuals were AC and AC;
 d(l)_ij=1.0 if the genotypes of the two individuals were AA and CC;
 d(l)_ij=0.0 if the genotypes of the two individuals were CC and CC;
```
To further know about the p_distance matrix based the VCF file, please refer to [this website.](https://phylipweb.github.io/phylip/doc/distance.html)
### 5) Results
------------
VCF2Dis have been cited in more than 170 times by [searching against google scholar](https://scholar.google.com.hk/scholar?hl=zh-CN&as_sdt=0%2C5&q=VCF2Dis&btnG=).
Below were some NJ-tree images that I draw in the paper before.

* [50 Rices NBT](https://www.nature.com/articles/nbt.2050/figures/1)
* [31 soybeans NG](https://www.nature.com/articles/ng.715/figures/1)
Display tree by MAGA after test Data VCF2Dis -i ALL.chr*.genotypes.vcf.gz -SubPop subsample203.list -InSampleGroup pop.info

![example1.png](https://github.com/hewm2008/VCF2Dis/blob/main/example/Example1/OUT.png)

### 6) Discussing
------------
- [:email:](https://github.com/hewm2008/VCF2Dis) hewm2008@gmail.com / hewm2008@qq.com / 
- join the QQ Group : 125293663
- other Contributors ： Lian Xu (xulian@ntu.edu.cn); Xun Liao (1911751806@qq.com）

######################swimming in the sky and flying in the sea ########################### ##

VCF2Dis

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

VCF2Dis

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README