scaling_laws_data_filtering

KO | EN

gitlite — search

#python #hacktoberfest #llm #golang #machine-learning #deep-learning #docker #java #windows #awesome #awesome-list #vue #ios #openai

scaling_laws_data_filtering

(★ 65)

No description available.

파일 탐색기

최종 버전 다운로드 (.zip)

- finding_a.py
- finding_k.py
- finding_tau.py
all_paths_128.py
estimate_best_pool.py
grid_search.py
objective.py
plotter.py
process_128_grid.py
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Scaling Laws for Data Filtering

### Registering data buckets

The buckets should be registered in the following file: `all_paths_128.py`
This file contains the following information:
- `path`: The path to the data file that has the evaluation results for a model trained on that dataset.
- `samples_per_epoch_dict`: The number of samples per epoch for the corresponding dataset.
- `match_with_dict`: This tells us if the evaluation is done at a fixed epoch interval, or a fixed sample interval.
- `subsample_every_dict`: In case you want to take the average of every `k` evaluations. This is usually only useful when the evaluation is done at a fixed sample interval.

### Estimating data bucket parameters

This step involves estimating the scaling parameters for each bucket of interest.

### Grid search to find the bucket scaling parameters

Grid search is performed to find the best scaling parameters for each bucket. The grid search is performed using the following file: `grid_search.py`. The objective minimized in the grid search is defined in `objective.py`. We chose grid search because the of instabilities observed in scipy based optimization methods.

### Objective Functions

This file implements scaling laws based on FADU, and also those inspired from work on Scaling Data Constrained Language Models.

- `func_effective_utility`: This is the function that uses the effective utility formulation as proposed in our work. 
- `func_effective_data`: This is the function that uses the formulation of effective data from Scaling Data Constrained Language Models.

```
python process_128_grid.py --a_upper 0.02 --objective effective_utility  --d 0.1
```
Here `a_upper` is used to give an upper limit to the grid search for `a`, and `d` is the irreducibile loss. Refer to `ablations/finding_a.py` if you want to jointly minimize `a` across the pools.
Copy the obtained scaling parameters to the `results/parameter_values.py` file, and give an appropriate key name.

### Finding best bucket combination
```
python estimate_best_pool.py --key given_key_name
```

scaling_laws_data_filtering

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

scaling_laws_data_filtering

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README