Neural-Crawler-Web-Crawler
BFS and DFS BASED Project.A neural network-powered web crawler that intelligently extracts, classifies, and processes web content using deep learning
File Explorer
- bug_report.md
- custom.md
- feature_request.md
- crawler_runner.cpython-312.pyc
- .gitkeep
- crawler-1.png
- crawler-2.jpg
- crawler3.png
- web-crawling.png
- __init__.cpython-313.pyc
- settings.cpython-313.pyc
- __init__.cpython-313.pyc
- crawling_spider.cpython-313.pyc
- __init__.py
- crawling_spider.py
- __init__.py
- items.py
- middlewares.py
- pipelines.py
- settings.py
- style.css
- index.html
- .gitattributes
- app.py
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- crawler_runner.py
- LICENSE
- output.json
- README.md
- scrapy.cfg
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
git checkout
View Details ▼
git checkout
Checkout a branch or paths to the working tree.
git checkout -b {{branch_name}}
Create and switch to a new branch:
git checkout -b {{branch_name}} {{reference}}
Create and switch to a new branch based on a specific reference (branch, remote/branch, tag are examples of valid references):
git checkout {{branch_name}}
Switch to an existing local branch:
git clone
View Details ▼
git clone
Clone an existing repository.
git clone {{remote_repository_location}} {{path/to/directory}}
Clone an existing repository into a new directory (the default directory is the repository name):
git clone --recursive {{remote_repository_location}}
Clone an existing repository and its submodules:
git clone {{[-n|--no-checkout]}} {{remote_repository_location}}
Clone only the `.git` directory of an existing repository:
git commit
View Details ▼
git commit
Commit files to the repository.
git commit
Open an editor to write a message and commit staged files to the repository:
git commit {{[-m|--message]}} "{{message}}"
Commit staged files to the repository with the specified message:
git commit {{[-F|--file]}} {{path/to/commit_message_file}}
Commit staged files with a message read from a file:
git push
View Details ▼
git push
Push commits to a remote repository.
git push
Send local changes in the current branch to its default remote counterpart:
git push {{remote_name}} {{local_branch}}
Send changes from a specific local branch to its remote counterpart:
git push {{[-u|--set-upstream]}} {{remote_name}} {{local_branch}}
Send changes from a specific local branch to its remote counterpart, and set the remote one as the default push/pull target of the local one:
pip install
View Details ▼
pip install
Install Python packages.
pip install {{package1 package2 ...}}
Install one or more packages:
pip install {{package1 package2 ...}} {{[-U|--upgrade]}}
Upgrade all specified packages to the latest version, installing any that are not already present:
pip install {{package}}=={{version}}
Install a specific version of a package:
python
View Details ▼
python
Python language interpreter.
python
Start a REPL (interactive shell):
python {{path/to/file.py}}
Execute a specific Python file:
python -i {{path/to/file.py}}
Execute a specific Python file and start a REPL:
scrapy
View Details ▼
scrapy
Web-crawling framework.
scrapy startproject {{project_name}}
Create a project:
scrapy genspider {{spider_name}} {{website_domain}}
Create a spider (in project directory):
scrapy edit {{spider_name}}
Edit spider (in project directory):
