llm_concept

KO | EN

gitlite — search

#javascript #python #hacktoberfest #llm #golang #go #rust #deep-learning #macos #java #windows #awesome #ios #open-source

(★ 13)

Code and data for "Revealing emergent human-like conceptual representations from language prediction"

llm_concept Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Revealing emergent human-like conceptual representations from language prediction

## Setup

To set up the project, follow these steps to organize the necessary directories and environment:

### Prerequisite

Before running this project, ensure that you have Conda installed on your machine. Conda is used for managing environments and dependencies.

### 1. Set Up the Python Environment

1. Create a new Conda environment and activate it:

```bash
   conda create -n dolphin python=3.11
   conda activate dolphin
   ```

2. Install the required dependencies:

```bash
   pip install torch transformers accelerate fire omegaconf sentencepiece protobuf nltk wordfreq openai
   pip install numpy scipy scikit-learn pandas seaborn matplotlib nibabel pycortex
   ```

3. Download the NLTK WordNet data:

```bash
   python -c 'import nltk; nltk.download("wordnet")'
   ```

### 2. Unzip Source Code

1. Create the project root directory:

```bash
   mkdir ${project_root}
   ```

2. move the `llm_concepts.zip` file into project root director:

```bash
   mv llm_concepts.zip ${project_root}
   ```

3. Navigate to the project root:

```bash
   cd ${project_root}
   ```

4. unzip the source code to the `${project_root}/dolphin` directory:
   ```bash
   unzip llm_concepts.zip -d dolphin
   ```

### 3. Download Hugging Face Models

To download the necessary Hugging Face models, follow these steps:

1. Navigate to the project root:

```bash
   cd ${project_root}
   ```

2. Inside the project root, create a subdirectory for model storage:

```bash
   mkdir models
   ```

3. Navigate to the `${project_root}/models` directory:

```bash
   cd models
   ```

4. Use the commands in `${project_root}/dolphin/scripts/download_models.sh` to download the large language models (LLMs) into the `${project_root}/models` directory.

## Data

For information on the data required for our experiments, please refer to the `corpus.pdf` file (or `README.md` in the `${project_root}/dolphin/corpus` directory).

The GPT-generated descriptions used in this project are stored in the `${project_root}/dolphin/corpus/things` directory, in the file `things_gpt_description.json`.

## Scripts

The commands to execute different tasks can be found in the `${project_root}/dolphin/scripts` directory.
Below is a summary of each file and its corresponding code directory:

### 1. Deriving conceptual representations from LLMs:

- **Script**: `reverse_dictionary.sh`
- **Description**: Derives conceptual representations from LLMs for following analyses.
- **Code**: Located in the `${project_root}/dolphin/exps/reverse_dictionary` directory.

### 2. Convergence towards a context-independent conceptual structure:

- **Script**:
  - `converge.sh` — Measures alignment between model representations.
  - `commqa.sh` — Evaluates model performance on downstream tasks.
- **Description**: Uncovers a shared, context-independent conceptual structure that predicts model downstream performance.
- **Code**:
  - `{project_root}/dolphin/exps/converge` — Alignment between model representations.
  - `{project_root}/dolphin/exps/commqa` — Model performance on downstream tasks.

### 3. Predicting human concept usage:

- **Script**: `hmn_concept.sh`
- **Description**: Predicts various facets of human concept usage including similarity judgments, categorization and gradient scales along multiple features.
- **Code**:
  - `{project_root}/dolphin/exps/structure` — Similarity judgments and categorization.
  - `{project_root}/dolphin/exps/sem_proj` — Gradient scales along features.

### 4. Mapping to neural activity patterns in the human brain:

- **Script**: `encoding.sh`
- **Description**: Maps LLM-derived conceptual representations to neural activity patterns in the human brain.
- **Code**: Located in the `{project_root}/dolphin/exps/encoding` directory.

### Notes:

- Ensure that the necessary dependencies are installed and the required datasets are placed in the `{project_root}/dolphin/corpus` directory before running the scripts.
- Refer to the respective code directories for more detailed implementation and configurations.

## Notebooks & Results

The notebooks used for result analysis and visualization are available in the `{project_root}/dolphin/notes` directory. The corresponding results can be found in the following directories and files.

### 1. Deriving concepts from definitional descriptions

#### Notebooks

- Probing results for LLaMA3-70B: `probe_results_70b.ipynb` (Figure 2)

- Probing results for LLaMA3-8B: `probe_results_8b.ipynb`

- Probing results for all LLMs on the THINGS database: `probe_results_all_llms.ipynb`

- Probing results for LLMs on WordNet data, GPT-generated descriptions, and shuffled descriptions: `probe_results_general.ipynb`

- Code for plotting Figure S1: `probe_figure.ipynb`

#### Results

- LLaMA3-70B probing results: `{project_root}/dolphin/notes/probing/relation_llama3-70b.csv`
- Results for all 67 LLMs: `{project_root}/dolphin/notes/probing/all_llms_results.csv`
- Results on WordNet data and GPT-generated descriptions: `{project_root}/dolphin/notes/probing/general_results.csv`
- Results on shuffled descriptions: `{project_root}/dolphin/notes/probing/shuffle_results.csv`

### 2. Convergence towards a context-independent conceptual structure

#### Notebooks

- Correlation between exact match accuracy and alignment correlation under different contextual demonstrations (within-model): `converge_corr_demo.ipynb`

- Visualization of different LLMs based on their alignment: `converge_visualize.ipynb`

- Correlation between exact match accuracy and alignment correlation (across different LLMs): `converge_overall.ipynb`

- Correlation between downstream performance and alignment correlation (across different LLMs): `converge_performance.ipynb`

- Code for plotting Figure 3: `converge_figure.ipynb`

- Code for plotting Figure S2: `converge_figure_si.ipynb`

- Code for plotting Figure S3: `converge_figure_perf_si.ipynb`

#### Results

- Alignment between LLM-derived conceptual representations under different contextual demonstrations: `{project_root}/dolphin/output/converge`
- Correlation between exact match accuracy and alignment with reference (conceptual representations formed based on 120 demonstrations) for LLaMA3-70B: `{project_root}/dolphin/notes/converge/converge_demos_*.csv`
- Alignment between different LLMs: `{project_root}/dolphin/notes/converge/converge_llms_spearman.npy`
- Two-dimensional visualization of models based on their alignment: `{project_root}/dolphin/notes/converge/tsne_visualize_spearman.csv`
- Correlation between exact match accuracy and alignment with LLaMA3-70B: `{project_root}/dolphin/notes/converge/factors_spearman.csv`
- Correlation between downstream performance and alignment with LLaMA3-70B: `{project_root}/dolphin/notes/converge/downstream_perf.csv`

### 3. Predicting human concept usage

#### Similarity

- Notebooks

- Results for SimLex-999 and MTurk-771:
    - `rep_simlex.ipynb` (SimLex-999)
    - `rep_mturk.ipynb` (MTurk-771)
  - Results for odd-one-out triplets in the THINGS database: `rep_things_ooo.ipynb`
  - Results for error analysis: `rep_error_ana.ipynb`
  - Code for plotting Figure S4: `rep_sim_figure_si.ipynb`

- Results
  - SimLex-999: `{project_root}/dolphin/notes/structure/simlex_sc_5runs.csv`
  - MTurk-771: `{project_root}/dolphin/notes/structure/mturk_sc_5runs.csv`
  - Error analysis: `{project_root}/dolphin/notes/structure/error_ana/ooo_fail_rate_24demos_5runs.csv`

#### Category

- Notebooks

- Categorization results: `rep_category.ipynb`
  - Visualization of LLM-derived conceptual representations and static word embeddings: `rep_tsne_visualize.ipynb`
  - Code for plotting Figure 4: `rep_similarity_visualize_figure.ipynb`
  - Code for plotting Figure S6: `rep_sim_cat_figure_si.ipynb`

- Results
  - Categorization: `{project_root}/dolphin/notes/structure/category_5runs.csv`
  - Two-dimensional visualization: `{project_root}/dolphin/notes/structure/vis_category.csv`

#### Gradient scales along features

- Notebooks

- Results for LLaMA3-70B: `sem_proj_rep_ana.ipynb` (Figure 5)
  - Comparison between LLM-derived representations and static word embeddings (GloVe): `sem_proj_rep_concept_vs_word.ipynb` (Figure 6)
  - Results after removing concepts with extreme feature values: `sem_proj_rep_ana_rme.ipynb` (Figure S7)
  - Full comparisons between LLM-derived representations and baselines: `sem_proj_rep_concept_vs_baseline.ipynb` (Figures S8–S9)

- Results
  - Results for LLaMA3-70B & baseline embeddings: `{project_root}/dolphin/output/sem_proj`

### 4. Mapping to neural activity patterns in the human brain

#### Encoding neural imaging data

- Notebooks

- Code for plotting Figure 7: `mri_encoding_figure.ipynb`
  - Encoding results for all three subjects:
    - `mri_encoding_data.ipynb`
    - `mri_encoding_corr_figure.ipynb` (Figure S10)

- Results
  - Prediction performance of LLaMA3-70B: `{project_root}/dolphin/notes/encoding/`

#### Variance partitioning with baseline representations

- Notebooks

- Comparisons with baseline embeddings via variance partitioning based on full–reduced model comparisons (the primary method reported in the main text): `mri_var_part_comp_things.ipynb`
  - Comparisons with baseline embeddings via orthogonalization-based variance partitioning: `mri_var_part_things.ipynb`
  - Code for plotting variance partitioning results:
    - Encoding and variance partitioning results reported in the main text: `mri_encoding_figure.ipynb` (Figure 7)
    - Overview:
      - Variance partitioning based on full–reduced model comparisons: `mri_var_part_comp_figure_si.ipynb` (Figure S11)
      - Variance partitioning based on orthogonalization: `mri_var_part_figure_si.ipynb` (Figure S16)
    - Results for all three subjects:
      - Variance partitioning based on full–reduced model comparisons: `mri_var_part_comp_plot_baseline.ipynb` (Figures S12–S15)
      - Variance partitioning based on orthogonalization: `mri_var_part_plot_baseline.ipynb` (Figures S17–S20)

- Results
  - Variance partitioning with baseline embeddings: `{project_root}/dolphin/notes/encoding/var_part/{baseline}`

#### Regression of similarity embeddings onto LLM-derived conceptual representations

- Notebooks

- Regression results: `reg_sim_ana.ipynb`
  - Relationship between regression and error profile in behavioral predictions: `rep_error_correlation.ipynb` (Figure S5)

- Results
  - Regression results: `{project_root}/dolphin/output/encoding/reg_ana/spose`

#### Noise ceilings

- Notebooks

- Noise ceiling data: `mri_nc_data.ipynb`
  - Noise ceiling plot: `mri_nc_plot.ipynb` (Figure S21)

- Results
  - noise_ceiling: `{project_root}/dolphin/notes/encoding/noise_ceiling`

llm_concept

llm_concept Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

llm_concept

llm_concept Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README