ai-daily-pulse

KO | EN

gitlite — search

#python #data-science #anthropic #command-line #github #docker #html #llm #csharp #chatgpt

(★ 9)

Your AI radar for AI news — scores, filters, and summarizes 47+ RSS feeds into a daily Obsidian digest with trends and builder insights.

파일 탐색기

ZIP 다운로드

- - daily.yml
- .gitkeep
- feed_health.json
- pushed.json
- .gitkeep
- digest-2026-05-18.json
- digest-2026-05-19.json
- digest-2026-05-21.json
- digest-2026-05-22.json
- digest-2026-05-23.json
- digest-2026-05-24.json
- digest-2026-05-25.json
- digest-2026-05-26.json
- digest-2026-05-27.json
- digest-2026-05-28.json
- digest-2026-05-29.json
- digest-2026-05-30.json
- digest-2026-05-31.json
- digest-2026-06-01.json
- digest-2026-06-02.json
- digest-2026-06-03.json
- digest-2026-06-04.json
- digest-2026-06-06.json
- digest-2026-06-07.json
- digest-2026-06-08.json
- digest-2026-06-09.json
- digest-2026-06-10.json
- digest-2026-06-11.json
- digest-2026-06-12.json
- digest-2026-06-13.json
- digest-2026-06-14.json
- digest-2026-06-15.json
- digest-2026-06-16.json
- digest-2026-06-17.json
- digest-2026-06-18.json
- digest-2026-06-19.json
- digest-2026-06-20.json
- digest-2026-06-21.json
- digest-2026-06-22.json
- digest-2026-06-23.json
- digest-2026-06-24.json
- digest-2026-06-25.json
- digest-2026-06-26.json
- digest-2026-06-27.json
- digest-2026-06-29.json
- digest-2026-06-30.json
- digest-2026-07-01.json
- digest-2026-07-02.json
- digest-2026-07-03.json
- digest-2026-07-04.json
- digest-2026-07-05.json
- digest-2026-07-06.json
- digest-2026-07-07.json
- digest-2026-07-08.json
- digest-2026-07-09.json
- digest-2026-07-14.json
- digest-2026-07-23.json
- digest-2026-07-24.json
- digest-2026-07-25.json
- digest-2026-07-28.json
- digest-2026-07-29.json
- digest-2026-07-30.json
- digest-2026-07-31.json
- digest-2026-08-01.json
- digest-2026-08-02.json
- digest-2026-08-03.json
- digest-2026-08-04.json
- digest-2026-08-05.json
- digest-2026-08-06.json
- digest-2026-08-07.json
- digest-2026-08-08.json
- digest-2026-08-09.json
- digest-2026-08-10.json
- digest-2026-08-12.json
- digest-2026-08-13.json
- digest-2026-08-14.json
- digest-2026-08-15.json
- digest-2026-08-16.json
- digest-2026-08-17.json
- digest-2026-08-18.json
- digest-2026-08-19.json
- digest-2026-08-20.json
- digest-2026-08-21.json
- digest-2026-08-22.json
- digest-2026-08-23.json
- digest-2026-08-24.json
- digest-2026-08-25.json
- digest-2026-08-27.json
- digest-2026-08-28.json
- digest-2026-08-29.json
- digest-2026-08-30.json
- digest-2026-08-31.json
- digest-2026-09-01.json
- digest-2026-09-02.json
- digest-2026-09-03.json
- digest-2026-09-04.json
- digest-2026-09-05.json
- digest-2026-09-06.json
- digest-2026-09-07.json
- digest-2026-09-08.json
- digest-2026-09-09.json
- digest-2026-09-10.json
- digest-2026-09-11.json
- digest-2026-09-12.json
- digest-2026-09-13.json
- digest-2026-09-14.json
- latest.json
- - __init__.py
  - base.py
  - rss.py
- __init__.py
- config.py
- dedup.py
- feed_health.py
- feeds.py
- file_utils.py
- history.py
- insights.py
- llm.py
- models.py
- scorer.py
- sync_insights.py
- trends.py
- writer.py
- __init__.py
- test_config.py
- test_dedup.py
- test_feed_health.py
- test_feeds.py
- test_file_utils.py
- test_history.py
- test_llm.py
- test_models.py
- test_scorer.py
- test_scrapers.py
- test_smoke.py
- test_trends.py
- test_writer.py
- - globals.css
  - icon.png
  - layout.tsx
  - page.tsx
  - robots.ts
  - sitemap.ts
- - AuthButton.tsx
  - AuthModal.tsx
  - BuilderOverview.tsx
  - BuilderProjectCard.tsx
  - DateSwitcher.tsx
  - ExploreClient.tsx
  - FeaturedItem.tsx
  - FeaturedSection.tsx
  - FinalCta.tsx
  - HeaderNav.tsx
  - InsightCard.tsx
  - InsightClient.tsx
  - InsightShowcase.tsx
  - InsightSpotlightCard.tsx
  - LandingHero.tsx
  - LandingHeroReveal.tsx
  - LocaleSwitcher.tsx
  - MobileMenu.tsx
  - PipelineSection.tsx
  - ProductHero.tsx
  - ProjectCard.tsx
  - ScoreBadge.tsx
  - SearchBar.tsx
  - SectionReveal.tsx
  - SiteFooter.tsx
  - StatsHero.tsx
  - SummaryBlock.tsx
  - Tag.tsx
  - ThemeProvider.tsx
  - ThemeSwitcher.tsx
  - VoteBar.tsx
- - request.ts
  - routing.ts
- - api-client.ts
  - api.ts
  - auth.tsx
  - metadata.ts
  - motion.ts
  - topicLabels.ts
  - topics.ts
  - types.ts
- - en.json
  - zh-CN.json
  - zh-TW.json
- - _worker.js
  - favicon.ico
  - index.html
  - logo.png
  - og-image.png
- - copy-digests.js
  - inject-layout.js
- .env.example
- .gitignore
- eslint.config.mjs
- next.config.mjs
- package-lock.json
- package.json
- postcss.config.mjs
- README.md
- tailwind.config.ts
- tsconfig.json
- - auth.ts
  - db.ts
  - ideas.ts
  - index.ts
  - middleware.ts
  - vote.ts
- .gitignore
- package-lock.json
- package.json
- schema.sql
- tsconfig.json
- wrangler.toml
.gitignore
CLAUDE.md
config.toml
feeds.toml
LICENSE
main.py
PROMPTS.md
publish.py
README-zh.md
README.md
requirements.txt

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
  <a href="README.md">English</a> · <a href="README-zh.md">中文</a>
</p>

> 🌐 **[ai-daily-pulse.top](https://ai-daily-pulse.top)** — 51 RSS feeds → AI scoring → semantic dedup → trend detection → build directions → bilingual daily digest. Open once a day, read only what matters.

---

## What is this?

Every day, massive amounts of AI news is published across platforms. Manually keeping up is exhausting and you'll miss things.

**AI Daily Pulse** automates this:

1. Fetches from 51 RSS sources (English + Chinese)
2. **Full-text extraction**: when RSS content is thin (< 400 chars), falls back to trafilatura to fetch the full article body
3. **Rule-based prefiltering**: drops low-quality articles (saves AI costs)
4. AI scores each article (0-10), keeps only high-quality content
5. **Jaccard title dedup**: coarse similarity check before LLM precise dedup
6. **Bilingual summaries**: Chinese + English summaries for every article
7. **why_now anti-hallucination**: lightweight fact-token grounding check blanks commentary that isn't backed by source content (zero extra LLM cost)
8. **Trend detection**: cross-source clustering based on LLM tags
9. **Digest quotas**: per-topic and total caps keep the daily digest balanced (no single topic flooding)
10. **Build directions (Insight)**: AI-generated project ideas with difficulty, MVP days, monetization — community-voted
11. **Social media copy**: auto-generated X/Twitter posts + Threads

Fully automated. You just **open it once a day**.

### Focus Areas

- OPC / solopreneur / AI monetization cases
- AI + ecommerce
- AI tools & Agent workflows
- AI new tech / new models
- Funding & investment news

---

## Web Frontend

Online at [**ai-daily-pulse.top**](https://ai-daily-pulse.top)

| Page | Description |
|------|-------------|
| **Home** | Today's pulse + featured articles |
| **Explore** | Topic filtering, keyword search, date range picker, 15 per page |
| **Insight** | All-time build directions with community voting (Good / Maybe / Don't), 30 per page |
| **Detail** | Full article summary, AI commentary, tags, source link |
| **About** | Project background & author info |

### Features

- **Insight Voting**: Sign in with GitHub / Google / Email to vote on project ideas. One vote per idea per user.
- **Date Range Picker**: Filter articles and ideas by date range (replaces old date dropdown)
- **i18n**: zh-CN / zh-TW / English (default: English)
- **Dark mode**: auto-follows system or manual toggle
- **Pagination**: Explore 15/page, Insight 30/page
- **Responsive**: two-column card layout on desktop

---

## Quick Start (5 min)

### 1. Fork this repo

Click **Fork** in the top-right corner.

### 2. Get an API Key

Works with any OpenAI-compatible API (DeepSeek, OpenAI, OpenRouter, Ollama, etc.).

> Recommended: [DeepSeek](https://platform.deepseek.com) — cheap, reliable, free credits on signup.

### 3. Add API Key to GitHub

Go to your fork → **Settings** → **Secrets and variables** → **Actions** → **New repository secret**

| Name | Value |
|------|-------|
| `API_KEY` | Your API key |

### 4. Enable GitHub Actions

Go to the **Actions** tab, click **"I understand my workflows, go ahead and enable them"**.

**Done.** Runs daily at 09:00 Beijing time. Results are committed to `output/`.

---

## Local Development

### Python Pipeline

```bash
git clone https://github.com/YOUR_USERNAME/ai-daily-pulse.git
cd ai-daily-pulse

pip install -r requirements.txt
export API_KEY=sk-...
python main.py

# Fetch multiple days
LOOKBACK_DAYS=3 python main.py
```

Output:
- `output/digest-YYYY-MM-DD.json` — structured data for web frontend
- `output/latest.json` — latest digest copy

### Web Frontend

```bash
cd web
npm install
npm run dev
```

Opens at [http://localhost:3000](http://localhost:3000).

### Worker API (Insight backend)

```bash
cd worker
npm install
npm run db:init
npx wrangler dev --local --port 8787
```

Then start web with: `NEXT_PUBLIC_API_URL=http://localhost:8787 npm run dev`

---

## Configuration

All settings in [config.toml](config.toml):

```toml
[api]
api_key = ""                          # or use API_KEY env var
base_url = "https://api.deepseek.com"
scoring_model = "deepseek-v4-flash"
summary_model = "deepseek-v4-flash"
price_in_per_m = 0.14
price_out_per_m = 0.28
max_concurrency = 0                  # 0 = unlimited; set >0 to cap in-flight LLM calls (eases 429s)

[pipeline]
lookback_days = 1
dedup_window_days = 90
content_cap = 4000
output_dir = "output"
fetch_timeout = 15                   # mirror of [timeouts].fetch (kept for back-compat)
fetch_workers = 8
score_workers = 4

[timeouts]                           # unified timeout config (prefix-mapped)
fetch = 15                           # RSS fetch
extract = 20                         # trafilatura full-text extraction
llm = 60                             # LLM API calls

[digest]                             # daily digest quotas
max_per_topic = 8                    # cap articles per topic (score desc)
min_per_topic = 1                    # guarantee minimum if available
max_total = 60                       # global cap, truncate by score desc

[insight]
sync_enabled = false                  # set to true after deploying Worker
```

Environment variables override config.toml:

| Variable | Purpose |
|----------|---------|
| `API_KEY` | API key |
| `BASE_URL` | API endpoint |
| `SCORING_MODEL` | Scoring model |
| `SUMMARY_MODEL` | Summary model |
| `LOOKBACK_DAYS` | Lookback days |
| `LLM_MAX_CONCURRENCY` | Cap in-flight LLM calls (0 = unlimited) |
| `TIMEOUTS_FETCH` / `TIMEOUTS_EXTRACT` / `TIMEOUTS_LLM` | Per-stage timeouts (seconds) |
| `DIGEST_MAX_PER_TOPIC` / `DIGEST_MIN_PER_TOPIC` / `DIGEST_MAX_TOTAL` | Digest quotas |
| `INSIGHT_API_URL` | Worker API URL for sync |
| `INSIGHT_SYNC_KEY` | Worker API sync key |

---

## Scoring System

Each article is scored 0-10 by AI. Core criteria: **information density × actionability**.

- Specific numbers, product names, technical details → high information density
- Enables action or better decisions → high actionability
- Vague opinions, marketing fluff, unsupported predictions → ≤ 4

**Keep threshold**: ≥ 5 (≥ 4 for GitHub Trending) AND topic is relevant.

Same event covered by multiple sources → only the highest-scored article is kept (Jaccard title similarity pre-filter + LLM precise dedup).

---

## Cost

Built-in cost controls:

| Mechanism | Description |
|-----------|-------------|
| Rule prefilter | Drop titles < 5 chars, content < 100 chars without AI keywords |
| 150+ keywords | AI tech, business models, growth, ecommerce, open source |
| Full-text extraction | Only triggered when RSS content < 400 chars (avoids unnecessary fetches) |
| Jaccard dedup | Title similarity > 0.4 before LLM check |
| 90-day history dedup | Already-pushed URLs are skipped |
| Content cap | 4000 chars max per article |
| why_now fact-check | Regex-based grounding check, no extra LLM call |
| LLM concurrency cap | Optional semaphore (default off) proactively reduces 429 storms |

Typical cost per run: **~$0.01-0.02** (DeepSeek V4 Flash).

---

## Reliability

- **tenacity retry**: exponential backoff for 429/5xx/timeout (401 fails immediately); 429s are retried, and the optional concurrency semaphore proactively reduces their occurrence
- **Provider fallback**: configurable primary → fallback model/base_url so a single provider outage doesn't kill the run
- **Atomic writes**: all output files written via temp-file + atomic rename, so a crashed run can't leave a half-written digest
- **Pydantic score validation**: LLM scoring responses are schema-validated before use, rejecting malformed JSON
- **Feed health monitoring**: tracks per-feed success/failure, warns at ≥ 3 consecutive failures
- **Empty run protection**: 0 articles passing quality filter → exits without overwriting `latest.json`
- **Same-day URL dedup**: keeps the highest-scored article per URL within a single run

---

## Project Structure

```
ai-daily-pulse/
├── main.py                      # Entry: fetch → dedup → score → trends → insights → write
├── config.toml                  # AI model & pipeline config
├── feeds.toml                   # RSS feed list (51 sources)
├── requirements.txt             # Python dependencies
│
├── src/                         # Python pipeline
│   ├── config.py                # Config loader (env overrides, section prefixes)
│   ├── models.py                # Pydantic ContentItem + ScoreResult models
│   ├── llm.py                   # LLMClient: retry, provider fallback, concurrency semaphore
│   ├── scrapers/                # Scraper abstraction
│   │   ├── base.py              # BaseScraper interface
│   │   └── rss.py               # RSS fetch + trafilatura full-text fallback
│   ├── feeds.py                 # Feed loading + concurrent fetch orchestration
│   ├── feed_health.py           # Feed health monitoring
│   ├── history.py               # URL history dedup (90-day window)
│   ├── dedup.py                 # Jaccard + LLM precise dedup
│   ├── scorer.py                # AI scoring, bilingual summaries, why_now + fact-check
│   ├── trends.py                # Trend detection: LLM tag cross-source clustering
│   ├── insights.py              # Build directions + social media copy (bilingual)
│   ├── file_utils.py            # Atomic write helpers
│   ├── sync_insights.py         # Sync directions to Worker API
│   └── writer.py                # Digest JSON output + quota enforcement
│
├── web/                         # Next.js 16 frontend
│   ├── app/[locale]/
│   │   ├── page.tsx             # Home: today's pulse + featured
│   │   ├── explore/             # Explore: topic filter + search + date range
│   │   ├── insight/             # Insight: all-time build directions + voting
│   │   ├── item/[date]/[id]/    # Article detail
│   │   └── about/               # About page
│   ├── components/              # InsightCard, VoteBar, AuthModal, ExploreClient, etc.
│   ├── lib/                     # api.ts, api-client.ts, auth.tsx, types.ts
│   └── messages/                # i18n (zh-CN, zh-TW, en)
│
├── worker/                      # Cloudflare Worker (Hono + D1)
│   ├── src/
│   │   ├── index.ts             # Entry point
│   │   ├── auth.ts              # GitHub/Google/Email OAuth
│   │   ├── ideas.ts             # Ideas CRUD + sync
│   │   ├── vote.ts              # Voting endpoints
│   │   ├── middleware.ts        # JWT auth middleware
│   │   └── db.ts                # D1 helpers + schema init
│   ├── schema.sql               # D1 table definitions
│   └── wrangler.toml            # Worker config
│
├── output/                      # Pipeline output (CI commits daily)
│   ├── digest-YYYY-MM-DD.json   # Structured data for web
│   └── latest.json              # Latest digest copy
│
├── data/
│   ├── pushed.json              # Pushed URL history
│   └── feed_health.json         # Feed health status
│
├── tests/                       # pytest tests
└── .github/workflows/
    ├── daily.yml                # Daily pipeline (09:00 CST)
    └── deploy-worker.yml        # Worker deploy on push
```

---

## Pipeline Flow

```
fetch (8 concurrent) → fulltext extract (if content thin) → prefilter (rules) → history dedup (URL) → score (4 concurrent) → dedup (Jaccard+LLM) → summarize_zh (4 concurrent) → summarize_en (4 concurrent) → why_now (score≥7, fact-checked) → trend_detect (tags) → insights (build directions + social posts) → write (JSON, quota-enforced) → sync to Worker API
```

---

## Tech Stack

**Pipeline**: Python 3.11+ · feedparser · OpenAI SDK · GitHub Actions

**Frontend**: Next.js 16 · React 19 · TypeScript · Tailwind CSS · next-intl

**Backend API**: Cloudflare Workers · Hono · D1 (SQLite) · JWT (jose) · Resend (email)

---

## License

MIT

ai-daily-pulse

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README