Armag.de B2B Data Automation

Project Snapshot

The Problem: Copy-Paste Is Costing You More Than You Think

The client was managing a large B2B product catalog sourced from armag.de, a German e-commerce portal for commercial kitchen and plumbing fittings. To keep their own systems up to date, their team had to manually log into the portal (prices are hidden from guests), open each product page one by one, copy titles, article numbers, EAN codes, prices, weights, technical specs, and category paths, then paste everything into a spreadsheet and repeat this across hundreds of products every time the catalog changed.

This wasn't a one-time task. It was a recurring weekly drain, eating up valuable hours that could have been spent on actual business operations. The risk of human error was high a wrong price, a missed product update, or a typo in an EAN code could cascade into real business problems downstream. The client needed a system that could do all of this automatically, reliably, and without supervision.

The Brief: What 'Done' Actually Looked Like

After the initial discovery call, I distilled the client's requirements into a clear, measurable target. The system needed to authenticate into the armag.de portal programmatically and extract the following fields for every product:

• Article number & EAN code
• Product title & full description (HTML-preserved)
• Category path (e.g. Armaturen > Großküchenarmaturen)
• Wholesale price & list price
• Product weight
• Technical specifications table (key-value pairs)
• Product image URLs
• Available PDF download links

The extracted data had to sync directly into a Google Sheet matching the client's existing column structure (in German: Artikelnummer, Listenpreis, Gewicht, Technische Daten, etc.), run automatically every day without a manual trigger, and be deployed to the client's own Proxmox server so they remained in full control.

The Solution: A Microservice Ecosystem, Built to Last

Rather than handing the client a fragile one-off script, I engineered a production-grade, containerized microservice with clearly separated responsibilities. The system is composed of four distinct layers that work together autonomously every morning at 4:00 AM.

Layer 1: The Scraper Engine (Python + BeautifulSoup)

The core of the system is a custom Python scraper class (ArmagScraper) built with requests and BeautifulSoup. This wasn't a simple 'grab the page and read it' script, armag.de required careful engineering at multiple levels.

Authenticated Session Management: The portal hides wholesale prices behind a login wall. The scraper fetches the login page to harvest CSRF tokens and hidden form fields dynamically, injects credentials from environment variables (never hardcoded), and mimics a real browser session using authentic headers (User-Agent, Referer, Accept-Language). It detects login success by checking for post-login redirect URLs and the presence of the German logout text 'Abmelden', and automatically re-authenticates if the session expires mid-run.

• Category Path : Prioritises the sidebar navigation tree to reconstruct a clean hierarchical path like 'Armaturen > Großküchenarmaturen', traversing parent <ul> elements upward. Breadcrumbs serve as a fallback.
• Weight : Targets a specific hidden <li class='entry--weight'> element, often missed by generic scrapers.
• Price : Handles multiple DOM layouts with cascading fallbacks, including a regex sweep of the buy-box when standard selectors return empty.
• Technical Data : Parses entire specification tables into structured key-value JSON, then formats them for Google Sheets ingestion.
• EAN : Applies a heuristic check (13-digit numeric string) as a fallback when labeled elements are absent.

Layer 2: The API Layer (FastAPI)

The scraper is exposed as a lightweight REST API using FastAPI, running on port 8000. This was a deliberate architectural decision, decoupling the scraper logic from the orchestration layer makes both independently testable and replaceable. A single ArmagScraper instance is initialised at startup and reused across all requests, maintaining the authenticated session in memory and avoiding repeated login overhead.

Layer 3: The Orchestrator (n8n Workflow)

The n8n workflow is where all the moving parts come together. It runs in two distinct pipelines.

Pipeline A : URL Discovery (daily at 4:00 AM): Fetches the armag.de sitemap (.xml.gz), decompresses it in-flight, parses all product URLs with their lastmod dates, compares them against the existing URL archive in Google Sheets, and saves any new or updated URLs to the archive sheet, building a permanent catalog record over time.

Pipeline B : Data Scraping: Reads all URLs from the archive, removes duplicates (tracking URLs seen in previous executions), splits them into batches of 100, POSTs each URL to the local Scraper API, writes the structured response back to the main product data sheet (append-or-update by Artikelnummer), then waits 2 seconds between batches to respect Google Sheets API quotas.

Layer 4: Server Deployment (Docker on Proxmox)

The client's infrastructure is a Proxmox-based home server, a common setup for self-hosted workloads in European SMBs. I containerized the entire application using Docker, publishing the image to Docker Hub (nawanjanalk/armag-scraper:latest) so it can be pulled and updated on the server with a single command. The container is configured with restart: unless-stopped so it automatically recovers from crashes or reboots, and credentials are injected as environment variables at runtime, never stored in source code or the image.

cd /opt/armag-deployment
docker compose up -d
# Check status
docker compose ps
# Should show: armag_scraper   Up

A subtle but important detail: the docker-compose.yml sets a static DNS server (8.8.8.8) directly in the container config. Consumer-grade Proxmox setups can have quirky DNS configurations that cause intermittent name resolution failures. This single line prevented what would have been completely silent scraping failures in production.

The Hard Parts: What Made This Project Non-Trivial

• The Login Wall : armag.de hides pricing from unauthenticated users. A naive scraper returns empty price fields and never knows why. The solution dynamically extracts the login form's hidden CSRF fields on every run, staying compatible even if the form structure changes.
• Inconsistent DOM Structures : Product pages weren't identical. Some had sidebar navigation for categories; others only had breadcrumbs. Some had price data in span.price--content; others required a regex sweep of the buy-box. A layered fallback system gives every extraction attempt two or three strategies before giving up.
• The Hidden Weight Field : Product weight was stored in a <li class='entry--weight'> element, a class-specific hidden list item that generic scrapers routinely skip. Identifying it required careful DOM inspection across dozens of actual product pages.
• Google Sheets API Rate Limits : Pushing hundreds of rows per day risks 'Too Many Requests' errors. The 2-second Wait node between batches, combined with deduplication that skips already-processed URLs, keeps the system well within API quotas.
• Self-Hosted DNS Constraints : Consumer Proxmox setups can have quirky DNS configurations. Setting a static DNS (8.8.8.8) in docker-compose.yml resolved intermittent name resolution failures that would have caused silent scraping failures.

The Results

Before this system, someone on the client's team was opening product pages one by one, reading numbers off a screen, and typing them into a spreadsheet, every single week. After this system, they open Google Sheets in the morning and the data is already there, complete, structured, and ready to use. The team now focuses on decisions, not data entry.

Tech Stack

Results & Impact

Eliminated 40+ hours of weekly manual data entry. The system runs daily at 4:00 AM, extracting 15+ structured fields per product. wholesale prices, EAN codes, category paths, weights, and technical specs and syncing them automatically to Google Sheets, with zero human intervention.