ScrapPY - GitLite

KO | EN

gitlite — search

#python #ai #llm #golang #machine-learning #rust #docker #java #cli #kubernetes #vue #awesome-list #pytorch #openai

ScrapPY

(★ 224)

ScrapPY is a Python utility for scraping manuals, documents, and other sensitive PDFs to generate wordlists that can be utilized by offensive security tools to perform brute force, forced browsing, and dictionary attacks against targets. The tool dives deep to discover keywords and phrases leading to potential passwords or hidden directories.

파일 탐색기

최종 버전 다운로드 (.zip)

- - pylint.yml
- ScrapPY.py
- test_pdf.pdf
.flake8
.gitignore
.pre-commit-config.yaml
LICENSE
pyproject.toml
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/RoseSecurity/ScrapPY

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd ScrapPY

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

$ python3 -m venv venv

파이썬 스크립트(또는 모듈)를 실행합니다.

$ pip3 install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# :dog2: ScrapPY: PDF Scraping Made Easy

<p align="center">
<img width=40% height=40% src="https://user-images.githubusercontent.com/72598486/200046477-94c17a93-2dc8-418b-96eb-2b554227dce2.png">
</p>

ScrapPY is a Python utility for scraping manuals, documents, and other sensitive PDFs to generate targeted wordlists that can be utilized by offensive security tools to perform brute force, forced browsing, and dictionary attacks. ScrapPY performs word frequency, entropy, and metadata analysis, and can run in full output modes to craft custom wordlists for targeted attacks. The tool dives deep to discover keywords and phrases leading to potential passwords or hidden directories, outputting to a text file that is readable by tools such as Hydra, Dirb, and Nmap. Expedite initial access, vulnerability discovery, and lateral movement with ScrapPY!

# Demo:

https://user-images.githubusercontent.com/72598486/201235531-6b037daf-d1f3-4d33-b256-8411e3a0b3da.mov

# Install:

Download Repository:

```
$ mkdir ScrapPY
$ cd ScrapPY/
$ sudo git clone https://github.com/RoseSecurity/ScrapPY.git
```

Install Dependencies:

```
$ python3 -m venv venv
$ source .venv/bin/activate
$ pip3 install -r requirements.txt
```

# ScrapPY Usage:

```
usage: ScrapPY.py [-h] [-f FILE] [-m {word-frequency,full,metadata,entropy}] [-o OUTPUT]
```

Output metadata of document:

```
$ python3 ScrapPY.py -f example.pdf -m metadata
```

Output top 100 frequently used keywords to a file name ```Top_100_Keywords.txt```:

```
$ python3 ScrapPY.py -f example.pdf -m word-frequency -o Top_100_Keywords.txt
```

Output all keywords to default ScrapPY.txt file:

```
$ python3 ScrapPY.py -f example.pdf
```

Output top 100 keywords with highest entropy rating:

```
$ python3 ScrapPY.py -f example.pdf -m entropy
```

ScrapPY Output:

```
# ScrapPY outputs the ScrapPY.txt file or specified name file to the directory in which the tool was ran. To view the first fifty lines of the file, run this command:

$ head -50 ScrapPY.txt

# To see how many words were generated, run this command:

$ wc -l ScrapPY.txt
```

# Integration with Offensive Security Tools:

Easily integrate with tools such as Dirb to expedite the process of discovering hidden subdirectories:

```
root@RoseSecurity:~# dirb http://192.168.1.123/ /root/ScrapPY/ScrapPY.txt

-----------------
DIRB v2.21
By The Dark Raver
-----------------

START_TIME: Fri May 16 13:41:45 2014
URL_BASE: http://192.168.1.123/
WORDLIST_FILES: /root/ScrapPY/ScrapPY.txt

-----------------

GENERATED WORDS: 4592

---- Scanning URL: http://192.168.1.123/ ----
==> DIRECTORY: http://192.168.1.123/vi/
+ http://192.168.1.123/programming (CODE:200|SIZE:2726)
+ http://192.168.1.123/s7-logic/ (CODE:403|SIZE:1122)
==> DIRECTORY: http://192.168.1.123/config/
==> DIRECTORY: http://192.168.1.123/docs/
==> DIRECTORY: http://192.168.1.123/external/
```

Utilize ScrapPY with Hydra for advanced brute force attacks:

```
root@RoseSecurity:~# hydra -l root -P /root/ScrapPY/ScrapPY.txt -t 6 ssh://192.168.1.123
Hydra v7.6 (c)2013 by van Hauser/THC & David Maciejak - for legal purposes only

Hydra (http://www.thc.org/thc-hydra) starting at 2014-05-19 07:53:33
[DATA] 6 tasks, 1 server, 1003 login tries (l:1/p:1003), ~167 tries per task
[DATA] attacking service ssh on port 22
```

Enhance Nmap scripts with ScrapPY wordlists:

```
nmap -p445 --script smb-brute.nse --script-args userdb=users.txt,passdb=ScrapPY.txt 192.168.1.123
```

## Future Development:

- [x] Allow for custom output file naming and increased verbosity
- [x] Integrate different modes of operation including word frequency analysis
- [x] Allow for metadata analysis
- [x] Search for high-entropy data
- [ ] Prepare packaging for `homebrew` installation
- [ ] Search for path-like data 
- [ ] Implement image OCR to enumerate data from images in PDFs
- [ ] Allow for processing of multiple PDFs