ThinkARM

KO | EN

gitlite — search

#react #typescript #go #android #machine-learning #deep-learning #docker #macos #java #windows #kubernetes #security #pytorch #openai

(★ 28)

Schoenfeld’s Anatomy of Mathematical Reasoning by Language Models

ThinkARM 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Schoenfeld’s Anatomy of Mathematical Reasoning by Language Models

📖 [Schoenfeld’s Anatomy of Mathematical Reasoning by Language Models](https://github.com/MingLiiii/ThinkARM)

🔥🔥 This is the repo for the **ThinkARM** project, which provides systematic analysis on Mathematical Reasoning by Large Language Models, grounded by Schoenfeld’s Episode Theory.

The repo contains:

- All the responses generated by the models mentioned in our paper and corresponding annotations, including a corpus of 410,991 sentences, 150 responses, generated by 15 models solving 100 math problems.
- The human annotated gold episode labels, including 7,067 sentences. 
- The code of ThinkARM, which can be directly used for annotating episodes.
- Some of the analysis code used in our paper.

(Feel free to email Ming ([Homepage](https://mingliiii.github.io/), [Email](minglii@umd.edu)) for any questions or feedback.)

## Contents
- [Overview](#overview)
- [Key Findings](#key-findings)
- [Setup](#setup)
- [Data Organization](#data-organization)
- [Automatic Labeling](#automatic-labeling)
- [Correctness Evaluation](#correctness-evaluation)
- [Fine-grained Analysis](#fine-grained-analysis)
- [Run Code](#run-code)

## Overview

Large language models increasingly expose reasoning traces, yet their underlying cognitive structure and steps remain difficult to identify and analyze beyond surface-level statistics. We adopt Schoenfeld's Episode Theory as an inductive, intermediate-scale lens and introduce **ThinkARM** (Anatomy of Reasoning in Models), a scalable framework that **explicitly abstracts reasoning traces into functional reasoning steps** such as Analysis, Explore, Implement, Verify, etc. When applied to mathematical problem solving by diverse models, this abstraction reveals reproducible thinking dynamics and structural differences between reasoning and non-reasoning models, which are not apparent from token-level views. We further present two diagnostic case studies showing that exploration functions as a critical branching step associated with correctness, and that efficiency-oriented methods selectively suppress evaluative feedback steps rather than uniformly shortening responses. Together, our results demonstrate that episode-level representations make reasoning steps explicit, enabling systematic analysis of how reasoning is structured, stabilized, and altered in modern language models.

## Key Findings

* 🔎 When reasoning traces are analyzed at the episode level, **a functional progression from abstract reasoning to concrete execution, and finally to evaluative control, consistently emerges.** Episodes associated with analysis and exploration use more abstract, conceptual language and **decrease** steadily as reasoning progresses, while execution-oriented episodes **dominate** the middle of the trace through sustained concrete operations. In contrast, verification-related episodes are characterized by evaluative and meta-level language and **increase** toward the end of the reasoning process.

* 🔎 Comparing reasoning and non-reasoning models, the difference is not merely how many tokens they generate, but how reasoning is structured. **Non-reasoning models allocate most of their response trace to execution**, with episode transitions largely following a feed-forward pattern toward implementation. In contrast, **reasoning models distribute effort across analysis, exploration, execution, and verification, and exhibit frequent iterative Explore-Monitor/Verify loops.**
    
* 🔎 Through our correctness-oriented case study, we find that **exploration reflects uncertainty and serves as a critical branching point**: correct solutions more often route exploration into monitoring or re-analysis, whereas incorrect solutions tend to continue execution or terminate prematurely after exploration. 
    
* 🔎 Through our efficiency-oriented case study, we find that **different efficient reasoning methods selectively suppress evaluation-oriented episodes and feedback loops, leading to varying degrees of divergence** from the reasoning patterns of the base model. Episode-level analysis thus reveals which episodes can be removed to gain efficiency, beyond token-level pruning.

## Setup

```bash
pip install -r requirements.txt
```

## Data Organization

The data is organized into the following directories within `data/`:

- `raw/`: Raw model outputs. Each file (e.g., `DSQwen32B.json`) contains the problems and model responses.
- `ground_truth/`: Human-annotated labels. Organized by model name. Each JSON file (e.g., `QwQ32B/1.json`) corresponds to a problem and contains the reasoning steps with `human_label`.
- `label/`: Automatically generated labels. Organized by model name. Each JSON file (e.g., `DSQwen32B/1.json`) contains the reasoning steps with `sentence-category` (the assigned label) and `sentence-category-reason`.
- `correct/`: Correctness evaluation results. JSON files mapping problem indices to boolean values indicating if the answer was correct.

## Automatic Labeling

```bash
export OPENAI_API_KEY=...
export GOOGLE_API_KEY=...
python -m method.label --annotate_model [ANNOTATE_MODEL] --response_model [RESPONSE_MODEL]
```

## Correctness Evaluation

```bash
python -m analysis.correctness_eval --model [MODEL] --evaluator_model [EVALUATE_MODEL]
```

## Fine-grained Analysis

### Temporal Dynamics

```bash
python -m analysis.temporal 
```

### Word Cloud Analysis

```bash
python -m analysis.word_cloud
```

### Diagnostic Analysis

```bash
python -m analysis.diagnostic 
```

### Episode-level N-gram Analysis

```bash
python -m analysis.episode_ngram_preprocess
python -m analysis.episode_ngram_discriminate [FILE1] [FILE2]

```

ThinkARM

ThinkARM 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

ThinkARM

ThinkARM 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README