mark-it-down - GitLite

KO | EN

gitlite — search

#linux #windows #privacy #nlp #animation #reactjs #pdf #large-language-models #web #hacktoberfest

mark-it-down

(★ 21)

Convert any web page to clean Markdown

File Explorer

ZIP Download

- - ci.yml
- web-to-markdown.ts
- - icon128.png
  - icon16.png
  - icon48.png
- - copy-static.js
- - convert.ts
  - converter.ts
  - extractor.ts
  - mdx-extractor.ts
  - popup.ts
  - types.ts
- .gitignore
- manifest.json
- package-lock.json
- package.json
- popup.css
- popup.html
- promo-tile-440x280.png
- screenshot-1280x800.png
- screenshot-kubernetes-1280x800.png
- screenshot-react-1280x800.png
- tsconfig.json
- tsup.config.ts
- web-to-markdown.zip
- privacy.html
- cli.ts
- converter.ts
- errors.ts
- extractor.ts
- fetcher.ts
- index.ts
- mdx-extractor.ts
- ssrf.ts
- turndown-plugin-gfm.d.ts
- types.ts
- converter.test.ts
- extractor.test.ts
- index.test.ts
- mdx-extractor.test.ts
- ssrf.test.ts
.editorconfig
.gitignore
.nvmrc
.prettierrc
eslint.config.js
LICENSE
package-lock.json
package.json
README.md
tsconfig.json
tsup.config.ts
vitest.config.ts

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

Command Glossary

Commands referenced in this DOCs, explained below.

🔍

`eval`

View Details ▼

Execute arguments as a single command in the current shell and return its result.

eval "{{echo foo}}"

Call `echo` with the "foo" argument:

eval "{{foo=bar}}"

Set a variable in the current shell:

🔍

`npm install`

View Details ▼

Install Node packages.

npm {{[i|install]}}

Install dependencies listed in `package.json`:

npm {{[i|install]}} {{package_name}}@{{version}}

Download a specific version of a package and add it to the list of dependencies in `package.json`:

npm {{[i|install]}} {{package_name}} {{[-D|--save-dev]}}

Download the latest version of a package and add it to the list of dev dependencies in `package.json`:

🔍

`npx`

View Details ▼

This command is an alias of `npm exec`.

tldr npm exec

View documentation for the original command:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Convert any web page to clean Markdown. Built for developers and LLM pipelines.

Handles documentation sites (Mintlify, GitBook, Docusaurus, Next.js docs), blogs, news articles, and more. Automatically extracts the main content, strips navigation chrome, and produces readable Markdown with GFM support.

## Quick Start

```bash
# No install needed
npx web-to-markdown https://example.com

# Save to a file
npx web-to-markdown https://example.com -o page.md
```

## Install

```bash
# Global CLI
npm install -g web-to-markdown

# Project dependency
npm install web-to-markdown
```

Requires Node.js 20 or later.

## CLI Usage

```bash
web-to-markdown <url> [options]
```

### Options

| Flag | Description |
|------|-------------|
| `-o, --output <file>` | Write to file instead of stdout |
| `-b, --browser` | Force headless browser rendering (for SPAs) |
| `-r, --raw` | Convert full HTML without content extraction |
| `-f, --frontmatter` | Include YAML frontmatter with metadata |
| `--no-images` | Strip images from output |
| `--timeout <ms>` | Timeout for page loading (default: 30000) |

### Examples

```bash
# Convert a blog post
web-to-markdown https://example.com/blog/post

# Include title, author, date as YAML frontmatter
web-to-markdown https://example.com/blog/post -f

# Strip all images
web-to-markdown https://example.com --no-images

# Convert full page HTML without content extraction
web-to-markdown https://example.com -r

# Pipe to another tool
web-to-markdown https://example.com | head -20
```

### JavaScript-Rendered Pages (SPAs)

Some pages (React, Vue, Angular apps) render content with JavaScript. Use `--browser` to launch a headless Chromium instance:

```bash
# One-time setup
npm install playwright
npx playwright install chromium

# Use the --browser flag
web-to-markdown https://myapp.com --browser
```

Playwright is an optional peer dependency and is only needed for JS-rendered pages.

## Programmatic API

```typescript
import { convert } from "web-to-markdown";

const { markdown, metadata, warnings } = await convert("https://example.com", {
  frontmatter: true,
  noImages: false,
  timeout: 30_000,
});

console.log(metadata.title);
console.log(markdown);
```

## How It Works

The converter tries multiple extraction strategies in order, using the first one that succeeds:

### Strategy 1: Raw Markdown Endpoint

Many documentation sites serve raw markdown when `.md` is appended to the URL path. The converter tries this first (e.g., `/docs/guide` becomes `/docs/guide.md`). Works with Mintlify, Stripe Docs, GitHub Docs, React docs, Next.js docs, and others.

### Strategy 2: MDX Extraction (Next.js / Mintlify)

For sites built with Next.js and Mintlify, the converter extracts MDX content directly from React Server Component payloads embedded in the HTML. It parses YAML frontmatter, strips JSX components (Accordion, Tab, Callout, Card, etc.) into clean Markdown equivalents, and resolves relative URLs.

### Strategy 3: Readability + Turndown

The general-purpose fallback:

1. **Extract** -- Mozilla Readability removes boilerplate (nav, footer, ads), isolating the main content
2. **Convert** -- Turndown converts the cleaned HTML to Markdown with GFM support (tables, task lists, strikethrough, fenced code blocks)

If Readability drops significant content (detected via `<pre>` block comparison), the converter falls back to extracting from `<article>` or `<main>` elements directly.

### Site-Specific Handling

**GitBook** -- Detects GitBook pages via `data-gb-*` attributes and removes navigation chrome before extraction: site header, sidebar TOC, breadcrumbs, SVG icons in headings, dark-mode duplicate images, and "Was this helpful?" boilerplate.

**Code blocks** -- Language detection from `class="language-X"`, `data-lang` attributes, and Sphinx/Pygments span classes. Adaptive fence length for code containing backticks.

## Security

- **SSRF protection** -- Blocks requests to private IPs, localhost, link-local, and cloud metadata endpoints. Handles obfuscated IPs (hex, octal, decimal encoding, IPv6-mapped IPv4, NAT64). DNS is resolved once and pinned to prevent rebinding attacks.
- **Dangerous protocol filtering** -- Strips `javascript:`, `data:`, `vbscript:`, and other dangerous URL schemes from links and images, with control character and zero-width character normalization.
- **Content limits** -- 10MB response size limit, 100K DOM element cap, 500-level nesting depth guard.
- **Redirect validation** -- Each redirect hop is re-validated for SSRF (max 5 redirects).
- **Sandboxed MDX evaluation** -- Compiled MDX is executed with a mock React runtime; dangerous patterns (`require`, `import`, `eval`, `fetch`, `process`) are blocked.
- **Typed errors** -- All failures throw specific error classes (`ValidationError`, `SSRFError`, `NetworkError`, `ContentError`) with ES2022 error cause chaining.

## License

MIT