Substack Summarizer

Key results

• 7–8/day — Posts fetched and summarized from subscribed accounts
• 100/hr — Ingestion capacity when publication volume spikes
• 24/7 — Selected Substack accounts monitored
• 0 inbox — Manual monitoring/reading removed from the loop

TL;DR

• A custom Node.js scraper API continuously monitors specific Substack accounts around the clock.
• Cookie authentication is used to bypass restrictions and access freshly published posts.
• LLMs summarize each downloaded post into clear, structured, and actionable points.
• Teams get a fast digest of relevant content without manual monitoring and reading.

Project Overview

Automates the workflow from detecting a new Substack publication to delivering a concise summary-ready output.

The system was built for teams that needed fresh Substack insights without manually tracking every account and digesting each article.

It continuously scans configured accounts, downloads posts as they go live, and sends each article through an AI summarization workflow.

• 24/7 monitoring of selected Substack accounts
• Cookie-authenticated scraping to reliably fetch new posts
• LLM summarization into structured, reusable insights
• Reduced manual review burden with faster intake

Business Challenge / Problem Statement

Manual Substack reading did not scale with publication velocity.

Important posts were easy to miss when monitoring large account lists.

Restricted content paths required resilient session/auth handling.

Insights were delayed when teams depended on manual summarization.

• High Monitoring Cost — Teams spent too much time polling feeds and reading long texts manually.
• Access Barriers — Cookie/session restrictions complicated automation consistency.
• Slow Consumption — Decision-ready summaries were not available in real time.

Objectives & Goals

Reduce manual content ingestion and create structured outputs for immediate use.

Build a reliable ingestion and summarization service for Substack publications from selected accounts.

Cut monitoring-to-summary latency and keep pace with frequent post bursts without adding analyst headcount.

• Continuous Feed Monitoring — Run scraping checks continuously and trigger summaries as soon as posts go live.
• Reliable Access Layer — Use cookies and robust request handling to pass platform restrictions with stable daily coverage.
• LLM Summarization — Convert long-form posts into concise, actionable summaries.
• Throughput at Scale — Sustain high-volume publication periods while keeping outputs structured, timely, and ready for immediate consumption.

Key Features & Automation

A practical content intelligence loop for teams.

Node.js Scraper API

A dedicated backend service monitors account configuration and captures each newly published post the moment it appears.

Cookie Access Control

• Session-aware scraping: Uses cookie-based authentication to access protected publication endpoints.
• Scheduled polling: Runs continuously with alert-driven scheduling to reduce missed posts during busy release windows.

LLM Summarization Engine

• Structured points: Summaries are organized into digestible, decision-oriented outputs.
• Actionability: Highlights key takeaways and practical items for quick follow-up.

Operational Delivery

Output is prepared for quick team distribution and consumption.

Volume Resilience

The system is designed to handle high-volume publication periods while keeping outputs structured, timely, and ready for immediate consumption.

Consistency Controls

• Access failover: Fallback token/session handling preserves continuity when auth state changes.
• Structured output: Consistent formatting ensures summaries are always readable and easy to action.

Results & Impact

Subscribed accounts typically yielded about 7–8 new posts per day depending on publish cadence; the pipeline summarized each into structured digests.

The scraper was built to handle bursts up to roughly 100 posts per hour when accounts published in waves.

Cookie-based handling improved consistency for restricted posts, and LLM summaries made it easier to spot what to act on quickly.

Technologies & Tools Used

Node.js automation, resilient ingestion, and summarization stack

Node.js scraping engine with cookie/session handling, queue-aware ingestion, and AI summarization prompts for structured output.

Built for high-volume publication streams while preserving reliability, formatting consistency, and delivery speed.

Technology stack

• Node.js — Custom API layer to manage monitoring, scraping, and orchestration.
• Cookies Authentication — Session/cookie-based bypass handling for consistent post downloads.
• LLM — AI model layer to summarize long-form text into structured bullets and action points.
• Scheduler + Job Queue — Continuous polling and processing pipeline to ensure publish events are handled in order and without drift.
• Resilience Middleware — Retry and fallback mechanisms for transient network/auth failures, keeping ingestion stable.
• Data Validation — Post-content normalization and sanitization before summarization to reduce hallucinated or malformed outputs.
• Structured Delivery Layer — Standardized output format so summaries can be consumed quickly by downstream teams or tools.