tp-berta - GitLite

KO | EN

gitlite — search

#hacktoberfest #react #go #android #machine-learning #docker #nodejs #java #cli #kubernetes #awesome-list #vue #pytorch #ios

tp-berta

(★ 71)

[ICLR 2024 spotlight] Making Pre-trained Language Models Great on Tabular Prediction

파일 탐색기

최종 버전 다운로드 (.zip)

- __init__.py
- autoint.py
- dcn2.py
- mlp.py
- saint.py
- tpberta_modeling.py
- - __init__.py
  - augmentations.py
  - data_openml.py
- __init__.py
- aux.py
- data.py
- data_utils.py
- deep.py
- env.py
- feature_encoder.py
- metrics.py
- optim.py
- util.py
- - 0-pretrain.sh
  - 1-finetune_default.sh
  - 2-tune.sh
  - 3-finetune_tuned.sh
- - pretrain_tpberta.py
- clean_feat_names.py
LICENSE
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/jyansir/tp-berta

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd tp-berta

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# TP-BERTa: A Fundamental LM Adaption Technique to Tabular Data

This repo contains original PyTorch implementation of:

- [Making Pre-trained Language Models Great on Tabular Prediction](https://openreview.net/pdf?id=anzIzGZuLi) (ICLR 2024 Spotlight)

## Key Features

The following key features are proposed in this paper:

- Relative magnitude tokenization (*RMT*): a distributed representation method for continuous values to enhance LM's numerical perception capability.

- Intra-feature attention (*IFA*): a mechanism to pre-fuse feature-wise information for reasonable tabular feature contexts & model acceleration.

- *TP-BERTa*: a resulting LM pre-trained from RoBERTa with the above features for tabular prediction.

## Project Structure

The repo structure and module functions are as follows:

```
├─bin ---- // Implementation of tabular models
│ ├─tpberta_modeling.py ---- // TP-BERTa base class
│ └─xxx.py ----------------- // Other non-LM DNN baselines
├─lib ---- // Utilities
│ ├─aux.py --------------- // Auxiliary Loss: Magnitude-aware Triplet Loss
│ ├─feature_encoder.py --- // Numerical Value Binner (C4.5 discretization)
│ ├─optim.py ------------- // Utilities for optimizer & trainer
│ ├─env.py --------------- // Environment Variables configs
│ ├─data.py -------------- // Dataset & Data Transformation class
│ ├─data_utils.py -------- // Data Config & Multi-task Loader class
│ └─xxx.py --------------- // Other standard utils
├─data --- // csv file path for pre-training & fine-tuning
│ ├─pretrain-bin
│ ├─pretrain-reg
│ ├─finetune-bin
│ ├─finetune-reg
│ └─finetune-mul
├─checkpoints --- // Pre-trained model weights & configs (RoBERTa, TP-BERTa)
├─configs --- // Model & Training configs for non-LM baselines
│ ├─default --- // default configs
│ └─tuned ----- // tuned configs (generated with hyperparameter tuning scripts)
├─scripts --- // Experiment codes
│ ├─examples --- // Example shell scripts for main experiments
│ ├─pretrain --- // Codes for TP-BERTa pre-training
│ ├─finetune --- // Codes for baseline fine-tuning & hyperparameter tuning
│ └─clean_feat_names.py --- // Text clean for table feature names
```

## Dependencies

All necessary dependencies for TP-BERTa are included in `requirement.txt`. To conduct the packaged baselines, uncomment the corresponding lines.

### How to pre-train a TP-BERTa from scratch

In experiment we saved weights and configs of [RoBERTa-base](https://huggingface.co/FacebookAI/roberta-base/tree/main) in the local `checkpoints/roberta-base` folder (network unavailable) and conducted pre-training with `scripts/pretrain/pretrain_tpberta.py`. You can use online HuggingFace APIs by assigning the argument `--base_model_dir` with "FacebookAI/roberta-base".

### Considerations for fine-tuning a TP-BERTa

The TP-BERTa is designed for standard supervised tabular data prediction, it requires fine-tuning on downstream datasets, and a larger training round (in experiment we uniformly used 200 max training epochs with an early stop of 50 epochs, codes [here](https://github.com/jyansir/tp-berta/blob/main/scripts/finetune/default/run_default_config_tpberta.py)) is preferred compared to the small tabular deep models (e.g., FT-Transformer) since **slightly fine-tuning the BERT-sized model is tend to be underfit** in our practice.

## TODO

- [x] Upload pre-trained TP-BERTa checkpoints.
    1. Download TP-BERTa checkpoints pre-trained on [single task type](https://drive.google.com/uc?export=download&id=13_GAK2VcShxm5TgqSvLk2afBTIYcCbEs) or [both task types](https://drive.google.com/uc?export=download&id=1ArjkOAblGPErmxUyVIfpiM0IztnjjYxq).
    2. Move the `*.tar.gz` file to the `checkpoints` folder (create one if not exists)
    3. Unzip the file and run TP-BERTa according to the scripts in `scripts/examples/finetune`.

- [x] Sort and update experiment datasets.
    1. We have acquired permission on distributing the used data subset from [TabPertNet (OpenTabs currently)](https://arxiv.org/abs/2307.04308) datasets.
    2. Download datasets for [pre-training](https://drive.google.com/uc?export=download&id=1Jy45I_vTKn6McMROi5IKjKoSi9QJtx9A) (202 datasets) and [fine-tuning](https://drive.google.com/uc?export=download&id=1JhOJR1kxjyu4w4ZHi8VcxgMh-iYJRDgG) (145 datasets).
    3. Unzip the `*.tar.gz` file to the `data` folder (create one if not exists).

- [ ] Integrate TP-BERTa to HuggingFace🤗 community.

## Citation

If you find this useful for your research, please cite the following paper:

```
@article{yan2024making,
  title={Making Pre-trained Language Models Great on Tabular Prediction},
  author={Yan, Jiahuan and Zheng, Bo and Xu, Hongxia and Zhu, Yiheng and Chen, Danny and Sun, Jimeng and Wu, Jian and Chen, Jintai},
  journal={arXiv preprint arXiv:2403.01841},
  year={2024}
}
```

## Acknowledgments

Our codes are influenced by the following repos:

- [HuggingFace Transformers](https://github.com/huggingface/transformers)
- [RTDL Numerical Embeddings](https://github.com/yandex-research/rtdl-num-embeddings)