Divi Scraper + n8n Blog Automation Pipeline

The Brief: Automate Blog Content from Divi

The client (Mikas) needed a system to automatically scrape articles from the Elegant Themes / Divi website and republish them on their own blog including images, layout, and formatting via an n8n workflow. Two problems had to be solved: (1) getting past Cloudflare bot protection, and (2) preserving image positions in the extracted text so the blog post could be reconstructed accurately.

Part 1: The Divi Scraper API

The scraper is a FastAPI microservice that exposes two endpoints: /scrape/blog for article pages and /scrape/docs for documentation pages. The core innovation is the image position marker system.

bash

# Example n8n HTTP Request node call
POST /scrape/blog
{
  "url": "https://www.elegantthemes.com/blog/divi-resources/example-post"
}

# Returns structured JSON:
# BLOG CONTENT: "...intro text... [[IMAGE_1]] ...more text... [[IMAGE_2]]..."
# IMAGES: ["https://cdn.elegantthemes.com/img1.webp", "https://...img2.webp"]
# PUBLISHED DATE: "2025-03-15T10:00:00+00:00"
# YOUTUBE URL: []

Beating Cloudflare: curl_cffi Impersonation

Standard libraries like requests or httpx are immediately blocked by Cloudflare's bot detection on the Elegant Themes site. The solution: curl_cffi, which impersonates a real Chrome 110 browser at the TLS fingerprint level, not just at the User-Agent header level.

python

from curl_cffi import requests

def fetch_html(url: str) -> bytes:
    # Impersonates Chrome 110 at TLS level, bypasses Cloudflare JS challenges
    response = requests.get(url, impersonate="chrome110", timeout=15)
    if response.status_code != 200:
        raise Exception(f"Failed to fetch {url}, status: {response.status_code}")
    return response.content

This approach works because Cloudflare's bot protection analyses the TLS handshake fingerprint (JA3/JA4), not just HTTP headers. curl_cffi replicates Chrome's exact TLS signature, making the request indistinguishable from a real browser visit.

The Image Position Marker System

The key technical challenge was preserving where images appeared in the article, not just what images existed. A naive approach collects all image URLs at the end, losing position context. The solution: replace each <img> tag in-place with a text marker before extracting text.

python

def extract_images_with_markers(content_soup):
    image_urls = []
    img_tags = content_soup.find_all("img")  # static list - won't change during iteration

    for img in img_tags:
        src = img.get("src") or img.get("data-src") or img.get("data-lazy-src") or ""
        if src.startswith("data:image"):  # skip tracking pixels
            src = ""

        marker_index = len(image_urls) + 1
        marker_text = f"[[IMAGE_{marker_index}]]"
        image_urls.append(src)

        # Replace <img> in the live soup tree with a plain-text marker
        img.replace_with(NavigableString(f" {marker_text} "))

    # get_text() now contains [[IMAGE_N]] at exact image positions
    marked_text = content_soup.get_text(separator="\n", strip=True)
    return image_urls, marked_text

The n8n workflow receives IMAGES[0] = URL for [[IMAGE_1]], IMAGES[1] = URL for [[IMAGE_2]], and so on. The blog automation nodes can reconstruct the full post layout with images in exactly the right positions.

Content Type Detection

The scraper handles two distinct page types on the Elegant Themes site, each with different DOM structures:

Page Type	Primary Selector	Fallbacks
BLOG	.entry-content	<article> → .et_pb_post
DOCUMENTATION	<article>	.entry-content → <main>

Part 2: The Image Converter Microservice

The companion microservice handles image format conversion as part of the blog pipeline. When the n8n workflow fetches images from Elegant Themes (WebP, AVIF, etc.), this service converts them to formats suitable for the target WordPress blog.

FastAPI endpoint accepting image uploads or URLs for conversion

Pillow handles JPEG, PNG, WebP conversion with configurable quality settings

Multi-stage Docker build - builder stage compiles libjpeg/libpng/libwebp, runtime stage is a lean production image

Runs as a non-root user for security; health check endpoint for Proxmox/Docker monitoring

2-worker Uvicorn setup sized for single-core Proxmox LXC containers

API key authentication (optional), leave empty for LAN-only dev use

The Complete n8n Pipeline

The 73-connection n8n workflow (ExecutedWorkflow.JSON) ties both microservices together into a full blog automation pipeline:

1.Fetch Sitemap : fetch_sitemap.py discovers all article URLs from the Elegant Themes sitemap

2.URL Extraction : extract_urls.py filters to blog/documentation URLs for scraping

3.Divi Scraper API Call : n8n HTTP Request node calls /scrape/blog with each URL

4.Content Processing : n8n Code nodes parse the BLOG CONTENT and reconstruct post structure

5.Image Download & Conversion : Image Converter microservice converts images to target format

6.WordPress Publish : n8n WordPress nodes create the draft post with content and media

Deployment