DataReaper

KO | EN

gitlite — search

#python #hacktoberfest #react #llm #go #machine-learning #rust #linux #nodejs #vue #security #mcp #openai #open-source

(★ 15)

DataReaper is a powerful Python tool designed to harvest data from publicly accessible HTTP servers. It combines the capabilities of Shodan search with web scraping techniques to efficiently gather information from targeted websites.

DataReaper 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DataReaper (DARE)

> **Disclaimer:** DataReaper is intended for authorized security research and educational purposes only. Users are solely responsible for ensuring they have proper authorization before scanning any systems. Unauthorized access to computer systems is illegal. The authors assume no liability for misuse of this tool.

DataReaper is a Python-based reconnaissance tool designed to discover and enumerate publicly accessible HTTP servers with directory listings exposed. It leverages the Shodan API to identify targets, performs automated directory traversal to catalog exposed files, downloads sensitive content, and can exploit discovered SSH keys to access remote hosts.

## Features

- **Shodan Integration**: Query Shodan for HTTP servers with open directory listings and store results in a local SQLite database
- **Automated Enumeration**: Recursively scan and catalog exposed directories and files
- **Sensitive File Detection**: Built-in detection for security-relevant files including SSH keys, credentials, configuration files, and more
- **Automated File Download**: Download files matching keyword lists from discovered targets
- **SSH Key Exploitation**: Parse downloaded SSH keys and public key files to extract usernames, then attempt SSH access against targets
- **Bash History Parsing**: Extract potential usernames from downloaded `.bash_history` files for use during SSH exploitation
- **Shadow File Processing**: Extract password hashes from downloaded `/etc/shadow` files and store them for offline cracking
- **Survey Logging**: SSH walker logs full directory trees of accessed hosts to per-IP log files with timestamps
- **Scan Result Export**: Export and filter database scan results to dated log files
- **Proxy Support**: Route traffic through a SOCKS proxy (such as Tor) for anonymized scanning
- **Noninteractive Mode**: Skip all prompts for automated or scheduled execution

## Requirements

- Python 3.6 or higher
- A paid Shodan API membership (required for API access)
- Optional: Tor (for anonymous scanning)

## Installation

### Step 1: Clone the Repository

```
git clone https://github.com/ice-wzl/DataReaper.git
cd DataReaper
```

### Step 2: Create a Virtual Environment

It is recommended to use a virtual environment to isolate dependencies.

**Linux/macOS:**
```
python3 -m venv venv
source venv/bin/activate
```

**Windows:**
```
python -m venv venv
venv\Scripts\activate
```

### Step 3: Install Dependencies

```
pip install -r requirements.txt
```

### Step 4: Configure Your Shodan API Key

1. Log in to your Shodan account at https://shodan.io
2. Navigate to your account page to obtain your API key
3. Create a file named `api.txt` in the DataReaper directory
4. Paste your API key as the only content in the file

```
echo "YOUR_API_KEY_HERE" > api.txt
```

### Step 5: Initialize the Database
This has already been done for you, including it here if you ever need to create a new database.
The database schema is located in `db/schema.sql`. If the database does not already exist or you need to reinitialize it:

```
sqlite3 db/database.db < db/schema.sql
```

On Windows (using SQLite command line):
```
sqlite3 db/database.db ".read db/schema.sql"
```

## Usage

DataReaper operates in five modes: **query**, **scan**, **exploit**, **shadow**, and **process targets**.

### Command Syntax

```
python DataReaper.py [-h] (-q | -s | -e | -sh | -pt) [-t TOR] [-p PORT] [-n] [-v]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `-q`, `--query` | Query Shodan for targets with directory listings on the specified port |
| `-s`, `--scan` | Scan all targets stored in the database and download matching files |
| `-e`, `--exploit` | Test downloaded SSH keys against discovered targets |
| `-sh`, `--shadow` | Process downloaded shadow files and extract password hashes |
| `-pt`, `--process_targets` | Export scan results from the database to log files |
| `-p PORT`, `--port PORT` | Target port number (required for query mode) |
| `-t IP:PORT`, `--tor IP:PORT` | SOCKS proxy address for anonymized requests (e.g., `127.0.0.1:9050`) |
| `-n`, `--noninteractive` | Run without prompts (skips IP confirmation and warnings) |
| `-v`, `--verbose` | Enable verbose output |

Note: The `-q`, `-s`, `-e`, `-sh`, and `-pt` options are mutually exclusive. You must choose one per execution.

### Workflow

**1. Query Shodan for Targets**

Query Shodan to populate your database with potential targets:

```
python DataReaper.py -q -p 8000
```

This searches Shodan for servers on port 8000 with exposed directory listings and stores the results in the SQLite database.

**2. Scan and Download from Targets**

Scan the discovered targets to enumerate their exposed files and automatically download anything matching the built-in keyword lists:

```
python DataReaper.py -s -p 8000
```

Downloaded files are saved to `downloads/` organized by target IP.

**3. Exploit Discovered SSH Keys**

After downloading files, attempt to authenticate to targets using any discovered SSH keys. DataReaper extracts usernames from public key files and bash history, then tries each key/username combination:

```
python DataReaper.py -e -t 127.0.0.1:9050
```

Without a proxy, you will be warned before connecting directly. Successfully accessed hosts are logged to the database and the SSH walker performs a full directory enumeration, writing results to `survey_results/`.

**4. Process Shadow Files**

Extract password hashes from any downloaded `/etc/shadow` files:

```
python DataReaper.py -sh
```

Hashes are written to `hashes.txt` and stored in the database for offline cracking.

**5. Export Scan Results**

Export scan results from the database to dated log files in `scan_results/`:

```
python DataReaper.py -pt
```

### Using Tor for Anonymity

To route your traffic through Tor, first ensure Tor is running on your system.

**Linux:**
```
sudo apt install tor
sudo systemctl start tor
```

**Windows:**

Download and install the Tor Expert Bundle from https://www.torproject.org/download/tor/

Then run DataReaper with the `-t` flag pointing to your Tor SOCKS proxy:

```
python DataReaper.py -s -p 8000 -t 127.0.0.1:9050
```

### Examples

Query Shodan for servers on port 80:
```
python DataReaper.py -q -p 80
```

Scan stored targets through Tor with verbose output:
```
python DataReaper.py -s -p 8000 -t 127.0.0.1:9050 -v
```

Exploit downloaded SSH keys through Tor:
```
python DataReaper.py -e -t 127.0.0.1:9050
```

Export scan results without prompts:
```
python DataReaper.py -pt -n
```

Process shadow files:
```
python DataReaper.py -sh
```

## Output

- **Database**: All target data, scan results, download records, accessed hosts, and extracted hashes are stored in `db/database.db`
- **Downloads**: Files downloaded from targets are saved in `downloads/`, organized by IP address
- **Scan Results**: Exported scan data is saved to `scan_results/` as dated log files (e.g., `targets_output_2026-04-07.log`)
- **Survey Results**: SSH walker directory enumerations are saved to `survey_results/` as per-IP dated log files (e.g., `192.168.1.1_survey_2026-04-07.log`)
- **Hashes**: Extracted shadow file hashes are written to `hashes.txt`
- **Runtime Log**: Each execution is timestamped in `runtime.log`

## Detected File Types

DataReaper includes built-in detection for security-sensitive files and directories, including:

- SSH keys (`id_rsa`, `id_ed25519`, `id_ecdsa`, etc.)
- AWS credentials (`.aws`)
- Git repositories and credentials (`.git`, `.git-credentials`)
- Certificate files (`.pem`, `.pfx`, `.p12`, `.crt`, `.key`)
- Shell history files (`.bash_history`, `.zsh_history`, `.python_history`)
- System directories (`/home`, `/root`)
- Password files (`passwd`, `shadow`)
- Configuration files (`.env`, `config.yaml`, `docker-compose.yml`, Wireguard configs)
- Offensive security tools (Cobalt Strike, Metasploit, Sliver, Havoc, Mythic, etc.)
- Malware artifacts (ransomware, stealers, droppers, crypters)

## SSH Walker

The SSH walker (`executors/ssh_walker.py`) can also be used as a standalone tool for SSH directory enumeration:

```
python executors/ssh_walker.py -i <IP> -p <PORT> -u <USERNAME> (-P <PASSWORD> | -k <KEY>)
```

| Argument | Description |
|----------|-------------|
| `-i`, `--ip_addr` | Target IP address |
| `-p`, `--port` | Remote SSH port (default: 22) |
| `-u`, `--username` | Username for authentication |
| `-P`, `--password` | Password for authentication |
| `-k`, `--key` | Path to private key file |

The walker performs a full SFTP directory enumeration while skipping noisy system directories (`/proc`, `/sys`, `/snap`, `/dev`, `/usr/share`, `/usr/src`, `/usr/snap`). Results are logged to `survey_results/`.

## Database Parser

The database parser (`parsers/db_parser.py`) can also be run standalone to export scan results:

```
python parsers/db_parser.py [-f] [-t]
```

| Argument | Description |
|----------|-------------|
| `-f`, `--filter` | Filter out common junk file types (html, js, jpg, ts, svg) |
| `-t`, `--today` | Only export results from today's scans |

## Disclaimer

DataReaper is intended for authorized security research and educational purposes only. Users are solely responsible for ensuring they have proper authorization before scanning any systems. Unauthorized access to computer systems is illegal. The authors assume no liability for misuse of this tool.

## Resources

- Shodan API Documentation: https://developer.shodan.io/api
- Python Requests Library: https://docs.python-requests.org/
- Tor Project: https://www.torproject.org/

## License

See the LICENSE file for details.

DataReaper

DataReaper 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

DataReaper

DataReaper 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README