Facebook Marketplace Scraping

Key results

• 15 min — Keyword re-search cadence, 24×7
• 4/hr — Captures per keyword from the 15-min poll
• Dozens — Keywords monitored each cycle
• First → sold — Per-item lifecycle tracking from first-seen

TL;DR

• We built a scheduled scraping pipeline for Facebook Marketplace that re-searched dozens of keywords every 15 minutes around the clock.
• The system collected listing snapshots into a normalized database and created durable time series per item, including first seen, last seen, sold status, and price-change history.
• This enabled trend analysis such as sell-through speed, average time on market, and pricing adjustments across categories, cities, and keyword segments.
• The outcome was a reliable data product for market intelligence that replaced manual monitoring and unsupported ad-hoc exports.

Project Overview

The project captured how active listings moved over time on Facebook Marketplace using high-frequency polling, producing an evidence-rich dataset for trend forecasting and pricing strategy.

The internal marketplace team needed continuous monitoring across a large keyword list to answer business questions that short-lived manual exports could never solve reliably.

We designed a scraper that re-runs every 15 minutes and records each listing state at every poll, storing snapshots in a relational model to preserve listing lifecycle events.

• Keyword-level search orchestration for dozens of categories
• 24-hour polling window with periodic retry logic and gap filling
• Item deduplication and fingerprinting to track lifecycle changes
• Price and availability trend analytics computed directly from snapshots

The result is not just a one-time dump but a dataset designed for longitudinal analysis: how long an item stays active, when pricing changes, and how fast demand cycles complete.

Business Challenge / Problem Statement

What blocked real-time marketplace intelligence before this project?

Manual checks and infrequent scraping missed listing updates, causing blind spots in inventory movement and price evolution.

Facebook listing pages changed frequently and could not be polled with a single static workflow, creating inconsistent capture and duplicate records.

The team needed reliable trend answers across many keywords, but lacked the database model and scheduling discipline to preserve each listing's lifecycle.

• Low-Frequency Data Collection — Infrequent checks failed to capture timely listing events like quick sell-outs or fast price changes.
• Volatile Availability State — Listings moved fast, required deduplicated tracking, and needed persistent item identity handling.
• No Time-Series Insight Layer — Without per-item snapshots over time, questions about sell-through speed and price behavior were answerable only by intuition.

Objectives & Goals

What success looked like for this implementation?

Establish continuous listing ingestion for a broad keyword set, at 15-minute intervals, and maintain a clean, queryable dataset that tracks each item from first detection to sold status.

• Continuous Capture — Run scheduled searches every 15 minutes across all configured keywords for uninterrupted marketplace visibility.
• Lifecycle Tracking — Build canonical item IDs so each listing can be updated as sold, relisted, price-adjusted, or removed.
• Trend Analytics Readiness — Materialize business metrics for time-on-market, sell-out window, and dynamic pricing behavior.

Key Features & Innovations

The scraper combined high-frequency scheduling, snapshot persistence, and lifecycle analysis for operationally useful insights.

Keyword-Driven Search Orchestration

A configuration-driven search list powered the crawler, allowing the team to add or reprioritize keywords without code changes while preserving execution traceability.

Cadence & Reliability Controls

• 15-min scheduler: Cron-style orchestration with controlled concurrency prevented overlap and preserved predictable polling windows.
• Retry strategy: Transient failures, timeouts, and temporary blocks were retried with controlled backoff and failure logging.

Item Time-Series Modeling

• Snapshot history: Every poll writes a normalized state row for each listing, enabling chronological lifecycle analysis.
• Status transitions: Available, sold, removed, and relisted transitions were derived from recurring snapshots and stored for downstream KPI calculations.

Price Movement Intelligence

Dedicated fields captured current price, previous price, percentage change, and elapsed time between changes, allowing pattern detection for urgency and demand signaling.

Actionable Output Layer

The final outputs fed directly into CSV/FDV files and database tables for BI reporting, including retention-ready indexing for fast analytical queries.

Governance and Data Hygiene

• Deduplication: Canonical signatures prevented duplicate insertion and protected trend computations from repeated fetch noise.
• Field standardization: Location, category, condition, and currency fields were normalized at ingestion for clean joins across keywords and time windows.

Results & Impact

The pipeline transformed one-off listing collection into a continuous intelligence feed, enabling operational decisions based on actual posting velocity and price movement patterns.

Teams can now answer questions that were previously impossible: how long a listing usually stays active, where quick discounts occur, and which keywords produce the shortest sell-through times.

The model is built to scale by keyword and geography without redesigning the data contract.

What our clients say

The Facebook Marketplace monitoring system finally gave us continuous visibility, not weekly snapshots. We can now quantify price behavior and sell-through speed with confidence.

Sarah Khan — Marketplace Analyst · Internal Team

Technologies & Tools Used

Technology stack powering continuous Facebook Marketplace capture

This solution combines resilient web automation, Node.js orchestration services, database-backed event tracking, and analytical exports designed for long-horizon marketplace trend reporting.

The stack emphasizes stable scheduling, durable logging, and consistent normalization so listing events can be trusted as the source of business insights.

Technology stack

• Python — Core orchestration language for the polling loop, parsing, and transformation.
• Node.js — Used for job API orchestration, webhook hooks, and monitoring endpoints around scrape cycles.
• Selenium — Browser automation to interact with dynamic Facebook Marketplace search pages.
• BeautifulSoup — Resilient HTML parsing and field extraction with fallback rules.
• PostgreSQL — Relational store for listings, snapshots, keyword jobs, and lifecycle states.
• Redis — Job queue and run-state coordination for high-frequency execution.
• Pandas — Data normalization, deduplication, and time-series feature generation.
• Cron / Scheduler — Execution orchestration to run jobs every 15 minutes, continuously.