AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality

KO | EN

gitlite — search

#react #ai #go #deep-learning #linux #docker #nodejs #awesome #cli #kubernetes #security #pytorch #mcp #openai

AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality

(★ 10)

No description available.

AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MSR Mining Challenge 2026
**AI Builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality**

---

## Artifact Overview

This artifact accompanies the paper:

> **AI Builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality**

The artifact provides the **data, scripts, and analysis outputs** used to study the quality of **AI-generated build system code** (Maven, Gradle, CMake, and Make) produced by autonomous coding agents.

The goal of this artifact is to support:
- **Reproducibility** of the reported results
- **Transparency** of the data preprocessing and analysis steps
- **Reuse** by future researchers studying AI-generated build code, refactoring, or technical debt

---

## Artifact Scope and Claims

This artifact supports the following claims from the paper:

- **C1:** AI coding agents sometimes introduce build-related code smells, particularly maintainability and security smells.
- **C2:** AI agents can eliminate existing build code smells via recognized build refactoring patterns.
- **C3:** A majority of AI-generated build-related PRs are accepted and merged by developers with minimal human intervention.

---

## Artifact Contents

The artifact contains:

- A curated subset of **AI-generated pull requests** affecting build files including Maven, Gradle, Cmake, and Make systems.
- **Before/after snapshots** of modified build files.
- **Static analysis results** produced by the Sniffer tool.
- **Derived datasets** summarizing introduced and eliminated code smells.
- **Analysis scripts** used to aggregate results.

---

## Repository Structure

```
├── data/            -- Contains all generated data including results for RQ1, RQ2 and RQ3-- 
│   ├──Build-related-PRs.csv   -- contains the results of filtering the AIDev data to contain only build-related PRs--
│   ├──BuildFiles-before-after/    --Contains Before/after snapshots of modified build files--
│   ├──Build_smells.csv        -- code smells for each before/after pair --
│   ├──RQ1-Results.csv      -- Results of RQ1--
│   ├──RQ2-Results.csv      -- Results of RQ2--
│   ├──RQ3-Results.csv      -- Results of RQ3--
│   ├──final_dataset.csv      -- Results of RQ1 includng manual labeling for RQ2, RQ3--
│   ├──introduced_smells_per_agent.csv   -- Introduced Smells Per Agent --
│   ├──smell_quality_change.csv   -- All Smells --
│   ├──smell_sums_by_agent.csv    --Total of Smells per agent--
│   ├──smells_diff.csv      --Introduced/Eliminated, Unchanged smells per build file--
│   ├──summary_pivot_by_agent_type.csv   --Count of Introduced/Eliminated, Unchanged smells per Agent--
│
│
│── scripts/
│   ├──CodeSmells_before_after.py    -- Calculate the count of codesmells for each build file content pair (before/after AI changes)
│   ├──Extract_file_CodeSmells.py    -- Run Sniffer to identify code smells for a build file --
│   ├──analyse_results.py            -- Count by smell type and Severity --
│   ├──build-files-content-before_after.py   -- get the content of each build file before/after AI changes using GitHub API--
│   ├──categorize_smells_by_agent_and_type.py   --  Count of smells by agent and type --
│   ├──load_And_Preprocess_AIDev.ipynb  -- clean DEVAI data to get build-related PRs--
│   ├──merge_csv_files.py     -- merge code smells results with PRs metadata--
│
├── secure_linter/       --Contains all Python files related to sniffer analysis of maintainability and security smells in CMake, Gradle, Maven, and Make code--
│   ├──smell_diff_analysis.py    
│   ├──cmake_security_checks.py
│   ├──gradle_security_checks.py
│   ├──makefile_analyzer.py
│   ├──makefile_security_checks.py
│   ├──maven_parser.py
│   ├──maven_security_checks.py
│
│
├── README.md
```

---

## Data Description

### Source Dataset
- **AIDev Dataset**: A large-scale dataset of ~933K agent-authored pull requests from GitHub.
- AI agents included:
  - OpenAI Codex
  - GitHub Copilot
  - Devin
  - Cursor
  - Claude Code

### Filtered Dataset
After automated filtering and manual validation, the final dataset includes:
- **387 AI-generated PRs**
- **945 build files**
- Build systems:
  - Maven (`pom.xml`)
  - Gradle (`.gradle`, `.gradle.kts`)
  - CMake (`CMakeLists.txt`, `.cmake`)
  - Make (`Makefile`, `.mk`)

---

## Analysis Pipeline To Reproduce Results

1. PR Filtering
```bash
Run load_And_Preprocess_AIDev.ipynb steps
```
2. Build File Extraction
```bash
python3 build-files-content-before_after.py
```
3. Static Analysis using Sniffer  (python3 Extract_file_CodeSmells)
```bash
python3 Extract_file_CodeSmells.py
```
4. Smell Comparison (introduced/eliminated/unchanged)
```bash
python3 CodeSmells_before_after.py
```
5. Count Code Smells Per Type and Severity (RQ1)
```bash
python3 analyse_results.py
```
6. Merge smell Results with PR metadata (Agent, merged/not merged, closed not closed) to answer RQ1, RQ2, and RQ3)
```bash
python3 merge_csv_files.py
```
7. Count Introduced Smells Per Agent (RQ1)
   
```bash
python3 categorize_smells_by_agent.py
```
8. Qualitative Refactoring Labeling (RQ2)

9. PR Acceptance Analysis (RQ3)
``` Both step 8 and 9 were manually labeled by the reviewers and level of agreement was calculated. Please check RQ2-Results.csv, and RQ3-Results.csv for full results including manual labeling.
```

---

## Tooling and Dependencies

### Required Software
- Python ≥ 3.9
- Sniffer (build-system static analyzer)

### Python Dependencies
```
pandas
numpy
matplotlib
requests
```

---

## Ethical Considerations

- All data originates from public GitHub repositories
- No private or personal data is included

---

## Citation
Our paper is accepted in MSR Mining Challenge 2026 !
Ghammam, Anwar, and Mohamed Almukhtar. "AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality." arXiv preprint arXiv:2601.16839 (2026).

AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality

AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality

AI-Builds-We-Analyze-An-Empirical-Study-of-AI-Generated-Build-Code-Quality 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README