EZAI-Web-Scraper

KO | EN

gitlite — search

#javascript #ai #llm #golang #go #android #machine-learning #rust #deep-learning #awesome #awesome-list #pytorch #openai #open-source

EZAI-Web-Scraper

(★ 10)

An API that allows you to scrape blog posts and articles and get a list of notes or a summary back.

EZAI-Web-Scraper Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# EZAI-Web-Scraper

An API that allows you to scrape blog posts and articles and get a list of notes or a summary back.

### Recommendations

1. Use browserless.io for scraping instead of the headless chromium that comes with puppeteer. It's faster and more reliable.
2. For better results changed the language model to text-curie-001 or text-davinci-002. The default model is cheap, but not the greatest.
3. For easiest deployment, use the docker file in the dist folder (Make sure you add the .env variables).
4. If you make a cool feature or find a bug, please consider contributing!

### Enviroment Variables

OPENAI_API_KEY={YOUR API KEY} (Optional. Key can be provided in the request headers)

BROWSERLESS_API_KEY={YOUR API KEY} (Optional. Only needed if you plan on using browserless.io)

PORT={YOUR CHOSEN PORT} (Required)

### How To Run

#### Command Line

Developement: `npm run test`

Production: `npm run start`

#### Docker

_(Does not work on Apple M1 chips)_

CD into the "dist" folder and build the image.

Run the image, but make sure to include the enviroment variables.

I have tested this project on render.com and Google Cloud Run. Both work well and are a good choice.

## API ENDPOINTS

### /notes

- Method: POST
- Parameters:
  - Headers:
    - Key
      - Description: Your OpenAI API key (Only use if you didnt set the OPEN_API_KEY enviroment variable)
      - Type: String
      - Required: False
  - Body
    - URI
      - Description: A link to the website you would like to have notes made from.
      - Type: String
      - Required: True

### /summary

- Method: POST
- Parameters:
  - Headers:
    - Key
      - Description: Your OpenAI API key (Only use if you didnt set the OPEN_API_KEY enviroment variable)
      - Type: String
      - Required: False
  - Body
    - URI
      - Description: A link to the website you would like to have summarized
      - Type: String
      - Required: True

### Example Request

```
const notes = await fetch(https://myapi.com/notes, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Key: "Your OpenAI key"
  },
  body: {
    URI: "https://blog.com/1234"
  }
)
```

EZAI-Web-Scraper

EZAI-Web-Scraper Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

EZAI-Web-Scraper

EZAI-Web-Scraper Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README