QareceWebCrawler

KO | EN

gitlite — search

#go #android #machine-learning #deep-learning #linux #nodejs #macos #java #windows #kubernetes #awesome-list #vue #ios #open-source

QareceWebCrawler

(★ 8)

This web crawler gathers the latest details, variations, imagery and pricing informations of a catalog of products given their urls from their corresponding online stores and prepares files ready for upload to your e-commerce platfrom. It was built with the purposes of making product additions easier for e-commerce retailers.

QareceWebCrawler 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Qarece Web Crawler

## Author: [(EJ) Vivek Pandey](https://viveckh.com) / [Sujil Maharjan](https://sujilmaharjan.com) / Arjun Bastola

## Save hours spent on manually adding and updating products to your e-commerce site and do it in seconds!

### This web crawler gathers the latest details, variations, imagery and pricing informations of a catalog of products given their urls from their corresponding online stores and prepares files ready for upload to your e-commerce platfrom. It was built with the purposes of making product additions easier for e-commerce retailers.

## Supported Ecommerce Stores

- Macys
- Sephora
- 6pm (Future Enchancement)
- Amazon (Future Enhancement)

## Output Files Customized for Platforms

- Woocommerce (Default)

## Important Notes

- The internal category mapping, profit rates, and shipping rates are to be adjusted by you according to your business logic in the reference files located at `product_scraper/product_scraper/reference_data` folder. The current values are default placeholders.

## Custom Enhancements for Your Business

This crawler prototype was initially built as per the needs of Qarece - New York. If you want a custom implementation or enhancement for your e-commerce business, reach out to the author at `viveckh@hotmail.com`.

Development rates are on a hourly basis or project basis depending on your needs.

## How to Run

- Clone the Repo (Duh)
- Make sure you have Python, Scrapy and Pandas installed in your system
- Add the urls of products you want to crawl by going to `/product_scraper/product_scraper/input/urls.txt`. Check the supported e-commerce stores above.
- Navigate back to the base of the scraper where the scrapy.cfg file is. `cd /product_scraper`
- `python setup.py`
- `scrapy crawl products-spider`
- `python clean-scraped-results.py`
- The output file to upload in woocommerce is ready at `/product_scraper/product_scraper/output/Product-formatted-for-wp.csv`. Run it twice, once in a new products import mode, and again in update mode to update existing products
- The output file to hide products that are no longer available in market will be ready at `/product_scraper/product_scraper/output/products_to_unpublish.csv`. Run it once in update mode to hide products that exist in your catalog but are not available in the market anymore.
- Any products that could not be crawled successfully with associated errors will be detailed in `/product_scraper/product_scraper/output/errors.txt`

## TODO
- An automated way to update catalog in woocommerce using output files without having to upload manually
- An automated script to prepare `urls.txt` by exporting the catalog from woocommerce in csv, and making a list of product's market urls

QareceWebCrawler

QareceWebCrawler 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

QareceWebCrawler

QareceWebCrawler 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README