url-metadata - GitLite

KO | EN

gitlite — search

#react #typescript #llm #go #rust #deep-learning #docker #java #kubernetes #awesome-list #ios #mcp #openai #open-source

url-metadata

(★ 172)

npm module: Fetch a url & scrape the metadata from its HTML with Node.js or the browser.

File Explorer

Download Latest Version (.zip)

- pull_request_template.md
- pre-commit
- index.js
- package.json
- - fetch.tsx
  - layout.tsx
  - page.tsx
- next-env.d.ts
- package.json
- tsconfig.json
- index.html
- index.ts
- metadata.html
- package.json
- tsconfig.json
- webpack.config.js
- - vite.svg
- - counter.ts
  - main.ts
  - style.css
  - typescript.svg
  - vite-env.d.ts
- .gitignore
- index.html
- metadata.html
- package.json
- tsconfig.json
- clean.js
- extract-canonical.js
- extract-charset.js
- extract-favicons.js
- extract-headers.js
- extract-headings.js
- extract-hreflang.js
- extract-img-tags.js
- extract-json-ld.js
- extract-meta-tags.js
- http-error.js
- metadata-fields.js
- parse.js
- resolve-fields.js
- utils.js
- basic.test.js
- canonical.test.js
- citations.test.js
- decode.test.js
- fail.test.js
- fields-gating.test.js
- fields-groups.test.js
- fields-network.test.js
- fields.test.js
- hreflang.test.js
- itemprop.test.js
- json-ld.test.js
- og.test.js
- omit-empty.test.js
- options.test.js
- proxy.test.js
- redirect-perf.test.js
- robots.test.js
- setup-env.js
- ssrf.test.js
- type-drift.test.js
- x402.test.js
.gitignore
browser.js
CHANGELOG.md
index.d.ts
index.js
LICENSE
main.js
package.json
README.md
ROADMAP.md

# Installation Guide

1. Get the code

git clone https://github.com/laurengarcia/url-metadata

Downloads the entire project code from GitHub to your computer.

cd url-metadata

Moves into the project folder you just downloaded.

2. Node.js

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Node.js Node.js must be installed to use npm. The LTS version is recommended.

npm install url-metadata --save

Downloads and installs the libraries listed in package.json.

✅ After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# url-metadata

Fetch a URL and scrape its metadata using Node.js or the browser. Optional mode to parse metadata from HTML strings or `Response` objects instead. You can also request just the fields you need for smaller responses and less extraction work, tuned for your use-case. First-class configurable security options available. Content is returned raw by design; sanitize your own output (so you control processing efficiency).

---
<div>
  👉 <i><strong>Looking for a quick hosted solution?</i> <a href="https://minifetch.com">Minifetch</a></strong> is a web page extraction & SEO toolkit built on top of this package by the same author. Get started free:
  <a href="https://www.npmjs.com/package/minifetch-api">npm install minifetch-api</a>
</div>

---

## **Features**
- **proxy mode** for routing thru unblocking services (optional)
- **headless javascript rendering** (in proxy mode)
- **screenshots** (in proxy mode)
- **parser mode** - pass in an html string or Response object (optional)
- automatic charset detection & decoding (optional)
- [x402](https://www.x402.org/) errors return payment requirements
- 📫 Actively monitored [Feedback Form](https://forms.gle/UyXmzp596ZYzWCEH8) | [Discord](https://discord.gg/BqVBeeGsc5) | [GitHub](https://github.com/laurengarcia/url-metadata)

## **Extracts:**
- redirects
- response headers
- performance metrics
- meta tags
- hreflang
- favicons
- citations, per the Google Scholar spec
- [Open Graph Protocol (og:) Tags](http://ogp.me/)
- [Twitter Card Tags](https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
- [JSON-LD](https://moz.com/blog/json-ld-for-beginners)
- h1-h6 tags
- img tags
- the full response body as a string of html (optional)

### Performance & Token-Efficiency
- Use `fields` option to request only what you need
- Shrinks the response size *and* skips the extra extraction work
- Use `omitEmpty: true` option to drop empty fields
- Customize for your specific use-case

### **🔒 Security** - Protects against:
- Infinite redirect loops: `maxRedirects` option defaults to 10.
- SSRF attacks via [request-filtering-agent](https://www.npmjs.com/package/request-filtering-agent) in Node.js v18+ (custom options also available)
- Memory-exhaustion attacks (gzip bombs, oversized responses): set `size` option. Pair with `timeout` to prevent slow/ connection-holding responses.
- Leaking sensitive headers (auth, cookies) if target URL redirects to a diff host.

More details below. To report a bug or request a feature please open an issue or pull request in [GitHub](https://github.com/laurengarcia/url-metadata). Please read the `Troubleshooting` section below *before* filing a bug.

## Install
Works with Node.js versions `>=18.17` or in the browser when bundled. Example build configs available in the [GitHub repo](https://github.com/laurengarcia/url-metadata) `/example-*` dirs: Next.js, Vite and Webpack (see `/example-typescript`).

```
npm install url-metadata --save
```

## Usage
In your project file:
```js
// Use 'import' in .mjs/ .ts files or if your package.json
// has "type": "module", otherwise use 'require'.
// This package supports both:
import urlMetadata from 'url-metadata';
// const urlMetadata = require('url-metadata');

async function getMetadata(url) {
  try {
    return await urlMetadata(url); // pass options as 2nd argument
  } catch (err) {
    console.error(err);
  }
}

const metadata = await getMetadata('https://en.wikipedia.org/wiki/WHATWG');
```

### Options & Defaults
To override the default options, pass in a second options argument. The default options are the values below.
```javascript
const options = {

// Customize the default fetch request headers:
  requestHeaders: {
    'User-Agent': 'url-metadata (+https://www.npmjs.com/package/url-metadata)',
    From: 'example@example.com'
  },

// Route the fetch through a proxy/ unblocking service.
  // In proxy mode, other fetch-related options apply to the proxy
  // fetch, not the target URL. See  "Proxy Mode" section below.
  // Presence of proxyUrl alone triggers proxy mode.
  proxyUrl: undefined,
  // Optional vendor-specific query params passed thru as-is,
  // exactly as named in their docs. Get residential proxies,
  // headless javascript rendering, and screenshots.
  proxyParams: undefined,

// Alternate use-case: pass `Response` object to be parsed
  // See example usage below.
  parseResponseObject: undefined,

// Request only what you need. Accepts atomic field names or named groups:
  // ['network', 'meta', 'og', 'twitter']
  // Also accepts specific meta tags with the "meta:<name>" syntax, ex:
  // ['meta:description'] or ['meta:og:url']
  // See "Performance Tuning" section below.
  fields: undefined,

// Drop empty fields (undefined, null, '', [], {}) from the returned
  // object. Shallow: only top-level fields are removed.
  omitEmpty: false,

// (Node.js v18.17+ only)
  // To prevent SSRF attacks, the default option blocks fetch
  // requests to private network & reserved IP addresses via:
  // https://www.npmjs.com/package/request-filtering-agent
  // Browser security policies prevent SSRF automatically.
  requestFilteringAgentOptions: undefined,

// (Node.js v18.17+ only)
  // Pass in your own custom `agent` to override the
  // built-in request filtering agent above
  // https://www.npmjs.com/package/node-fetch/v/2.7.0#custom-agent
  agent: undefined,

// Maximum redirects in request chain, defaults to 10
  maxRedirects: 10,

// `fetch` timeout in milliseconds, default is 10 seconds.
  // Time-bounds slow and connection-holding (Slowloris-class) responses.
  // Auto-bumped to 60 seconds in proxy mode (see "Proxy mode" below)
  // unless you explicitly pass your own `timeout`.
  timeout: 10000,

// (Node.js v18.17+ only) max size of response in bytes (decompressed).
  // Aborts before limit is exceeded so oversized upstreams can't
  // exhaust process memory. Pair with `timeout` option above.
  // Default set to 0 disables max size:
  size: 0,

// (Node.js v18.17+ only) compression defaults to true
  // Support gzip/deflate content encoding, set `false` to disable
  compress: true,

// Force-rewrites favicon and img url strings returned to use https://,
  // valid for images with absolute & protocol-relative urls.
  // Relative urls pass thru untouched:
  ensureSecureImageRequest: true,

// Charset to decode response with (ex: 'auto', 'utf-8', 'EUC-JP')
  // defaults to auto-detect in `Content-Type` header or meta tag
  // if none found, default `auto` option falls back to `utf-8`
  // override by passing in charset here (ex: 'windows-1251'):
  decode: 'auto',

// (Browser only) `fetch` API cache setting
  cache: 'no-cache',

// (Browser only) `fetch` API mode (ex: 'cors', 'same-origin', etc)
  mode: 'cors',

// Number of characters to truncate description to
  descriptionLength: 750,

// Include raw response body as string
  includeResponseBody: false
};

// Options usage:
const metadata = await urlMetadata(url, options);

// Alternately, parse a Response object instead:
try {
  // fetch the URL in your own code
  const response = await fetch('https://en.wikipedia.org/wiki/WHATWG');
  // ...do your own thing
  // then pass the `response` object to be parsed for metadata:
  const metadata = await urlMetadata(null, {
    parseResponseObject: response
  });
  console.log(metadata);
} catch (err) {
  console.error(err);
}

// Similarly, if you have a string of html you can create
// a Response object and pass the html string into it:
const html = `
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Metadata page</title>
    <meta name="author" content="foobar">
    <meta name="keywords" content="HTML, CSS, JavaScript">
  </head>
  <body>
    <h1>Metadata page</h1>
  </body>
</html>
`;
const response = new Response(html, {
  headers: {
    'Content-Type': 'text/html'
  }
});
const metadata = await urlMetadata(null, {
  parseResponseObject: response
});
console.log(metadata);
```

### Proxy mode for blocked web pages (403 errors)
This package is vendor-neutral. Any proxy (unblocking) service works via `proxyUrl` + `proxyParams`. Two vendors are documented below, affiliate links support the author. Want another added? Ask in the [Discord support channel](https://discord.gg/BqVBeeGsc5).

`proxyUrl` triggers proxy mode. Proxy calls route through a third party doing its own upstream fetch, which takes longer than a direct fetch — so `options.timeout` defaults to 60 seconds instead of the usual 10, or you can customize.

`proxyParams` is a flat passthru, sent verbatim as query params exactly as named in your vendor's docs - no allowlist, no translation on our side. Some vendors authenticate via a header instead of a query param (ex: an `x-api-key` header) — for those, pass it in `requestHeaders` instead and skip `proxyParams` entirely if the vendor needs no other params.

In proxy mode, `requestHeaders` are sent to the proxy vendor, not the target URL — the vendor makes its own request to the target URL and won't include them unless it has a passthrough param for it (ex: ScraperAPI's `keep_headers: true`).

**Note: HTML-only.** Some vendors offer structured-data or merchant-specific endpoints that return JSON or CSV, not HTML — this package can't parse those and will throw `unsupported content type` if your proxy config is pointed at one. That's expected; those are a different kind of tool. Stick to each vendor's plain HTML-fetching endpoint (ex: ScraperAPI's https://api.scraperapi.com/, ScrapingAnt's https://api.scrapingant.com/v2/general).

**Note:** `redirects`, `responseHeaders` and `performance` timing reflect the proxy call, not the target server's actual response.

**More code examples** and notes available in our [Github test suite](https://github.com/laurengarcia/url-metadata/blob/master/test/proxy.test.js).

#### 👉 ScraperAPI.com

[Grab your ScraperAPI key (affiliate link supports the author)](https://docs.scraperapi.com/getting-started/quick-start/grab-your-api-key?fp_ref=lauren37). It's a premium choice for hard-to-get pages.

ScraperAPI's [proxy params list](https://docs.scraperapi.com/control-and-optimization/supported-parameters?fp_ref=lauren37) offers headless javascript rendering via `render: true`, also supports `screenshot: true`. Activate residential and mobile IPs by setting `premium: true`. `ultra_premium: true` activates advanced bypassing mechanisms - can exceed the 60s default, so pass a higher `timeout` option explicitly in that case.

ScraperAPI [status codes and error list](https://docs.scraperapi.com/ai-parser/status-codes-and-errors?fp_ref=lauren37) is here.

```javascript
const metadata = await urlMetadata('https://hardto.get', {
  proxyUrl: 'https://api.scraperapi.com/',
  proxyParams: {
    api_key: '<YOUR_API_KEY>'
  }
});
// Find out how many credits the request cost you:
console.log(metadata.responseHeaders['sa-credit-cost']);
// Get screenshot URL when you set `proxyParams: { screenshot: true }`
console.log(metadata.responseHeaders['sa-screenshot']);
```

#### 👉 ScrapingAnt.com

[Grab your ScrapingAnt API Key (affiliate link supports the author)](https://scrapingant.com/?ref=ztg1mgz). Good choice if you are more price-sensitive.

It works the same way, with its own `proxyUrl` and `proxyParams`. Offers headless javascript rendering by default. Set proxy param `browser: false` to turn it off. You may pass your API key as proxy param `x-api-key` or send it as a header for extra security and privacy. Set `proxy_type` to `residential` for pages that are more difficult (default is `datacenter`).

ScrapingAnt [status codes and error list](https://docs.scrapingant.com/errors?ref=ztg1mgz) is here.

```javascript
const metadata = await urlMetadata('https://hardto.get', {
  proxyUrl: 'https://api.scrapingant.com/v2/general',
  proxyParams: {
    'x-api-key': '<YOUR_API_KEY>'
  }
});
```

### Performance Tuning

This package has powerful levers for optimizing the amount of processing that takes place for each target URL *and* for reducing response sizes. By default every metadata field is extracted and returned. Pass the `fields` option an array to narrow that down. The response only includes what you list, and the extractors for everything else never run.

`fields` accepts atomic field names ([any key from `lib/metadata-fields.js`](https://github.com/laurengarcia/url-metadata/blob/master/lib/metadata-fields.js)) plus these named groups:
- **`network`** — transport data only (status, headers, redirects, timing); see below, it's special
- **`meta`** — every meta tag, including page-specific ones not on the built-in list
- **`og`** — all [Open Graph](http://ogp.me/) (`og:`) tags
- **`twitter`** — all [Twitter Card](https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup) (`twitter:`) tags

Selecting a group keeps its empty fields so the shape stays predictable; add `omitEmpty: true` to drop empty fields entirely for further streamlining. An unknown token or an empty `fields: []` throws.

For a single page-specific meta tag, use the `meta:<name>` syntax — the part after `meta:` is the exact tag name, colons and all (`meta:og:url` is valid):

```js
const metadata = await urlMetadata(url, {
  fields: ['title', 'og', 'meta:dc.creator']
});
```

#### The `network` group: a fast, universal probe

`network` resolves as soon as the response **headers** arrive and **never downloads the response body**. A cheap way to check status, redirect chain, and timing without fetching the whole page.

Because it doesn't read the body, **`network` works on any URL regardless of content type** — HTML, PDFs, images, JSON, binaries where a normal fetch would otherwise throw `unsupported content type`:

```js
// Probe any URL — no body downloaded, any content type works:
const probe = await urlMetadata('https://pdfobject.com/pdf/sample.pdf', {
  fields: ['network']
});
probe.responseStatusCode; // 200
probe.responseHeaders;    // { 'content-type': 'application/pdf', ... }
probe.redirects;          // { count, chain }
probe.performance;        // { ttfbMs, redirectTimeMs, ... }
```

Since the body is never read, `performance.responseTimeMs` is `undefined` in this mode; there's no body-read to measure.

**Note:** `fields: ['network']` (only) and `proxyUrl` throw an error when used together, since no body-read happens in `network`-only mode, and the proxy executes the fetch on your target URL and returns the result in the body.

## Returns
Returns a promise resolved with a JSON object. Note that the returned `url` field will be the last hop in the request chain if there are redirects.

A basic template for the object can be found in `lib/metadata-fields.js`. Any additional meta tags found on the page are appended as new fields to the object. Extractor details live in `/lib/extract-*`.

Content is returned raw, by design, as found on the page without sanitizing it. Only you know its output context, so sanitization is your call (e.g. DOMPurify before rendering to a DOM; parameterized queries when storing to SQL). Treat all extracted values as untrusted.

The object consists of key/value pairs as strings, with exceptions:
- `redirects` is an object with `count` (number) and `chain` (array of `{ order, url, statusCode }`)
- `canonical` is a string, and returns the first `<link rel="canonical">` found on the page (empty string if none). The raw href of every canonical tag found — including if there's only one, or none, or many — is also returned as `canonicalUrls`, an array of strings in the order encountered while parsing the page, so you can reason about this yourself (e.g. detect and fix multiples).
- `hreflang`, `favicons`, and `responseHeaders` is an array of objects containing key/value pairs of strings
- `jsonld` is an array of objects
- `meta` tags with multiple or duplicate tags on the page are returned as strings with comma-separated values since these tags don't typically contain commas, except:
- meta tags that begin with `citation_` (ex: `citation_author`) return with keys as strings and values that are an array of strings conforming to the [Google Scholar spec](https://scholar.google.com/intl/en/scholar/inclusion.html#indexing) which allows for multiple citation meta tags with different content values, including commas. So if the html contains:
```
<meta name="citation_author" content="Arlitsch, Kenning">
<meta name="citation_author" content="OBrien, Patrick">
```
... it will return as:
```
'citation_author': ["Arlitsch, Kenning", "OBrien, Patrick"],
```

### TypeScript

See `index.d.ts` for the full field catalog and the other exported interfaces: `Options`, `HreflangTag`, `FaviconTag`, `Heading`, `ImgTag`, `RedirectHop`, and `UrlMetadataError`.

The default return type `Result` is intentionally loose (collapses to `any`) so that existing codebases, mocks, and defensive scraping code compile untouched. If you want a documented, autocompleting shape, opt in with a cast. No runtime change needed:

```ts
import urlMetadata from 'url-metadata';

// Known fields fully typed; any extra page-specific
// meta tags fall through to `any`:
const metadata = await urlMetadata(url) as urlMetadata.KnownFields;
metadata.title;      // string
metadata.favicons;   // FaviconTag[]
metadata.redirects;  // { count: number, chain: RedirectHop[] }
metadata.foobar;     // any (arbitrary meta tag found on page; mostly returns as string)

// Strictest variant: known fields ONLY, closed shape.
// Catches typos at compile time, but arbitrary page-specific
// meta tags (incl. `citation_*`) are compile errors:
const strict = await urlMetadata(url) as urlMetadata.KnownFieldsStrict;
```

When you pass `fields` or `omitEmpty: true` options, the return type narrows to `Partial<urlMetadata.KnownFields>` automatically. Known fields become optional (any of them may be filtered out), and no cast is needed:

```ts
const partial = await urlMetadata(url, { fields: ['og', 'network'] });
partial['og:title'];        // string | undefined
partial.responseStatusCode; // number | undefined
```

### Troubleshooting

**⛔ Issue: Returns `404`, `403` errors or a CAPTCHA form.** Your fetch request may have been blocked by the target server because it suspects you are a bot or scraper. This package has a built-in proxy mode you can use for hard-to-get pages, see "Proxy Mode" section above.

**⛔ Issue: Proxy mode throws `unsupported content type` errors.** This is expected behavior when you use a proxy url or proxy params that produce anything other than HTML. See "Proxy Mode" section above.

**⛔ Issue: Proxy mode with `render: true` param (headless browser rendering) throws `unsupported content type: text/x-component`.** Some JS-rendered sites (esp Next.js App Router sites) can return a React Server Component payload — a serialized component tree — instead of the rendered HTML page. This package can't parse that as HTML, so it throws instead of returning corrupted metadata. This is a quirk of the target site (and sometimes intermittent, tied to CDN/caching behavior), not something fixable from this package's side; retry or test without `render: true`. You may also try a different proxy.

**⛔ Issue: `No fetch implementation found`.** You're in either an older browser that doesn't have the native `fetch` API or a Node.js environment that doesn't support `node-fetch` (Node.js <18.17). File a GitHub issue or try dowgrading to `url-metadata` version 2.5.0 which uses the now-deprecated `request` module.

**⛔ Issue: `DNS Lookup` errors.** The SSRF filtering agent defaults on this package prevent calls to private ip addresses, link-local addresses and reserved ip addresses. To change or disable this feature you need to pass custom `requestFilteringAgentOptions`. More info [here](https://www.npmjs.com/package/request-filtering-agent).

**⛔ Issue: `Response status code 0` or `CORS` errors.** The `fetch` request failed at either the network or protocol level. Possible causes:

- CORS errors. Try changing the mode option (ex: `cors`, `same-origin`, etc) or setting the `Access-Control-Allow-Origin` header on the server response from the url you are requesting if you have access to it.

- Trying to access an `https` resource that has invalid certificate, or trying to access an `http` resource from a page with an `https` origin.

- A browser plugin such as an ad-blocker or privacy protector.

---

👉 You may also want to try the hosted version of this package: [Minifetch](https://www.npmjs.com/package/minifetch-api).

---
📫 Get in touch: [Feedback Form](https://forms.gle/UyXmzp596ZYzWCEH8) | [Discord](https://discord.gg/BqVBeeGsc5) | [Github Issues](https://github.com/laurengarcia/url-metadata/issues)