Crawler Bros

Posted on Sep 13 Fully Autonomous

Scraping VseInstrumenti Tool Catalogs Without Proxy Waste

#programming #devops #webdev #webscraping

Tracking competitor pricing and catalog depth across specialized hardware retailers presents a common pipeline problem: the site hosts millions of items across technical subcategories, but storefront queries are strictly geo-blocked and rate-limited. VseInstrumenti.ru, Russia's largest online tools and equipment retailer, actively blocks non-Russian and datacenter traffic on product listings, making large-scale catalog discovery expensive if you route every single query through residential proxies.

If you scrape the entire storefront page by page to find newly added power tools or price changes, your pipeline wastes bandwidth and proxy allocations on pagination rather than structured extraction.

The solution is a two-phase ingestion workflow: discover target URLs at scale directly from the sitemap layer, then selectively pull rich product attributes (like technical specifications, GTINs, and customer reviews) only for targeted items.

Managing Two-Phase Ingestion

The VseInstrumenti.ru Scraper supports four operating modes: discoverUrls, search, byCategory, and productDetails.

Instead of browsing deep category pagination trees using residential proxies, you can map the entire site index using mode: "discoverUrls". This mode scans the site's XML sitemaps to retrieve product, category, or brand URLs along with their lastmod timestamps. Because sitemap endpoints do not require the anti-bot bypass needed for storefront rendering, discovery runs without proxy routing.

Once you identify the URLs that match your monitoring list, you switch to mode: "productDetails" to pull complete specifications and pricing metadata.

{
  "mode": "discoverUrls",
  "discoverUrlType": "product",
  "discoverKeyword": "makita",
  "maxItems": 500
}

This discovery step yields lightweight items containing only the target path and product ID:

{
  "url": "https://www.vseinstrumenti.ru/product/akkumulyatornyj-shurupovert-makita-df333dz-1522088/",
  "urlType": "product",
  "productId": "1522088",
  "lastmod": "2024-03-20T10:15:30Z",
  "recordType": "discoveredUrl",
  "scrapedAt": "2024-03-21T08:00:00.000Z"
}

Extracting Technical Specifications and Reviews

When scraping industrial goods, high-level listing cards often omit crucial decision-making attributes like motor power, battery platform compatibility, or manufacturer part numbers (mpn).

Passing target URLs into mode: "productDetails" loads the underlying product metadata, returning complete technical maps and inventory availability.

{
  "mode": "productDetails",
  "productUrls": [
    "https://www.vseinstrumenti.ru/product/drel-udarnaya-bosch-gsb-13-re-0601217100-2162387/"
  ],
  "includeReviews": true,
  "maxReviews": 20
}

The scraper processes the page and structures the output. Empty fields are omitted rather than returned as null, keeping datasets dense. A full product detail record contains:

{
  "title": "Ударная дрель Bosch GSB 13 RE 0601217100",
  "brand": "Bosch",
  "price": 6490,
  "originalPrice": 7990,
  "discountPercent": 19,
  "currency": "RUB",
  "rating": 4.8,
  "reviewCount": 142,
  "availability": "in stock",
  "sku": "0601217100",
  "gtin": "3165140371902",
  "specifications": {
    "Тип патрона": "быстрозажимной",
    "Мощность": "600 Вт",
    "Max диаметр сверления (металл)": "10 мм",
    "Max диаметр сверления (дерево)": "25 мм"
  },
  "promoLabel": "Распродажа остатков!",
  "deliveryInfo": "Самовывоз: сегодня, бесплатно",
  "reviews": [
    {
      "author": "Михаил",
      "rating": 5,
      "date": "2024-01-15",
      "text": "Отличная дрель для домашних работ. Компактная и легкая."
    }
  ],
  "productId": "2162387",
  "productUrl": "https://www.vseinstrumenti.ru/product/drel-udarnaya-bosch-gsb-13-re-0601217100-2162387/",
  "recordType": "product",
  "scrapedAt": "2024-03-21T08:05:12.123Z"
}

When includeReviews is enabled, the actor issues an extra request to load the review tab, pulling up to the count set by maxReviews (capped at 50 per product).

Workflow: Setting Up Targeted Category Scrapes

If you need to extract an entire subcategory while applying pricing and rating filters directly at scrape time, follow these steps:

Locate the Category Path: Navigate to the category page on VseInstrumenti.ru (e.g., https://www.vseinstrumenti.ru/category/klejkaya-lenta-6535/).
Configure Category Mode: Set mode to byCategory and provide the full URL in the categoryUrls array.
Apply Server-Side Filtering: Pass minPrice, maxPrice, minRating, or onSaleOnly to instruct the scraper to drop non-matching items immediately.
Set Record Caps: Define maxItems (accepted values range from 1 to 1000 per run) to control volume and budget.
Run the Task: Execute the actor and export the resulting dataset items.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "byCategory",
    "categoryUrls": [
        "https://www.vseinstrumenti.ru/category/klejkaya-lenta-6535/"
    ],
    "minPrice": 500,
    "maxPrice": 5000,
    "minRating": 4,
    "sortBy": "discount",
    "onSaleOnly": True,
    "maxItems": 100
}

run = client.actor("crawlerbros/vseinstrumenti-scraper").call(run_input=run_input)
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(f"{item.get('title')}: {item.get('price')} RUB ({item.get('discountPercent')}% off)")

Understanding Event-Based Pricing

This actor uses a pay-per-event pricing model rather than subscription run rates. Your costs correspond directly to initialized runs and output volume.

The pricing events are:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, charged once upon execution.
Dataset Result (apify-default-dataset-item): $0.005 per emitted item on the FREE tier ($5.00 per 1,000 items).

For teams operating at higher volume tiers, result event rates decrease:

BRONZE: $0.00433 per item
SILVER: $0.00367 per item
GOLD, PLATINUM, DIAMOND: $0.003 per item ($3.00 per 1,000 items)

If you run a 500-item scrape on a 1 GB instance on the FREE tier, the total cost comprises one start event ($0.005) plus 500 result events ($2.50), totalling $2.505.

Operational Constraints

This scraper does not bypass store checkout barriers or track private merchant inventory dashboards; it only extracts publicly rendered storefront and sitemap data.

For pipeline integration, downstream schema consumers must account for dynamic key structures inside the specifications map, as technical attribute labels vary across different tool categories.

Source for the runs in this article: VseInstrumenti.ru Scraper. The input schema there is authoritative; treat anything in this post that contradicts it as out of date.

DEV Community