<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Syntrovia Data</title>
    <description>The latest articles on DEV Community by Syntrovia Data (@syntrovia_data).</description>
    <link>https://dev.to/syntrovia_data</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4245319%2Fa4abbd23-bfc3-4cac-9112-722fb9e81442.jpg</url>
      <title>DEV Community: Syntrovia Data</title>
      <link>https://dev.to/syntrovia_data</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/syntrovia_data"/>
    <language>en</language>
    <item>
      <title>Filtering SAM.gov's daily Contract Opportunities extract by NAICS, set-aside and state</title>
      <dc:creator>Syntrovia Data</dc:creator>
      <pubDate>Sun, 15 Nov 2026 03:18:40 +0000</pubDate>
      <link>https://dev.to/syntrovia_data/filtering-samgovs-daily-contract-opportunities-extract-by-naics-set-aside-and-state-22d0</link>
      <guid>https://dev.to/syntrovia_data/filtering-samgovs-daily-contract-opportunities-extract-by-naics-set-aside-and-state-22d0</guid>
      <description>&lt;p&gt;&lt;strong&gt;Disclosure:&lt;/strong&gt; I'm the developer. This is my own tool on Apify, it is paid per result (prices below), and it is unofficial: not affiliated with or endorsed by GSA, SAM.gov or any U.S. government agency. I drafted this post with AI help and checked the numbers against the tool's README.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why I built it.&lt;/strong&gt; I wanted a plain spreadsheet of new notices for a few NAICS codes every morning, without logging in and without writing code against the API. GSA publishes a public extract of active Contract Opportunities once a day on sam.gov/data-services (a CSV of about 220 MB). The tool reads that file from the newest notices backwards and stops as soon as it passes your date window, so a 2-day window reads about 15 MB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Filters:&lt;/strong&gt; notice type, NAICS prefix, PSC prefix, set-aside code, place-of-performance state, agency, and keywords (matched in the title and description; the description itself is not returned).&lt;/p&gt;

&lt;p&gt;Example: IT services (NAICS 5415), small business or SDVOSB set-asides, work in VA/MD/DC, posted in the last 7 days, only notices an earlier run has not returned yet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"postedWithinDays"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"noticeTypes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Solicitation"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Combined Synopsis/Solicitation"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"naicsPrefixes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"5415"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"setAsideCodes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"SBA"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SDVOSBC"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"placeOfPerformanceStates"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"VA"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"MD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"DC"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cloud"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cybersecurity"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"keywordMode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"any"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"onlyNewSinceLastRun"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"stateName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"it-services-dmv"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each row has the notice ID and link, title, solicitation number, notice type, posted date, response deadline, department, sub-tier and office, NAICS, PSC, set-aside, place of performance (city, state, ZIP; no street address), and award number, date and amount on award notices.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it leaves out on purpose:&lt;/strong&gt; contracting officer names, e-mails, phone numbers, street addresses and the description text. Those are details of individual people, and the description often contains them. You click through to the notice on SAM.gov for the full text and attachments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Limits, so nobody is surprised:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The file is updated once a day, so today's notices usually show up tomorrow. If you need alerts minutes after posting, this is not the tool.&lt;/li&gt;
&lt;li&gt;Active notices only. No archived notices, no attachments.&lt;/li&gt;
&lt;li&gt;If SAM.gov's own saved searches already work for you, keep using them. This is for people who want the rows in a sheet, in Slack or in their own database.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cost:&lt;/strong&gt; $0.005 per run plus $0.002 per notice saved. A daily run that returns about 20 new notices comes to about $1.35 a month. A run that matches nothing costs $0.005, and if the day's file is empty or fails the layout check on its header and first 200 rows, the run stops before charging anything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Where I'd like your input:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Do you filter by posted date, or by response deadline (for example "due in 10 to 30 days")? Which one should the tool support first?&lt;/li&gt;
&lt;li&gt;Which notice types do you actually watch? I can't tell whether Sources Sought and Special Notices are noise or early signal for most of you.&lt;/li&gt;
&lt;li&gt;Is there a field in the extract you rely on that I should add?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Link: &lt;a href="https://apify.com/syntrovia/federal-opportunities-sam-gov-extract?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=govcon-feedback" rel="noopener noreferrer"&gt;https://apify.com/syntrovia/federal-opportunities-sam-gov-extract?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=govcon-feedback&lt;/a&gt;&lt;/p&gt;

</description>
      <category>opendata</category>
      <category>government</category>
      <category>automation</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How to audit every URL in your sitemap for on-page SEO issues (no headless browser)</title>
      <dc:creator>Syntrovia Data</dc:creator>
      <pubDate>Sun, 15 Nov 2026 03:17:06 +0000</pubDate>
      <link>https://dev.to/syntrovia_data/how-to-audit-every-url-in-your-sitemap-for-on-page-seo-issues-no-headless-browser-56de</link>
      <guid>https://dev.to/syntrovia_data/how-to-audit-every-url-in-your-sitemap-for-on-page-seo-issues-no-headless-browser-56de</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Disclosure:&lt;/strong&gt; I build and maintain the Apify Actor used in this post (published as Syntrovia Data). It is a paid, pay-per-event tool; the prices are listed below. This post was drafted with AI assistance, and every input, output field and price was checked against the Actor's README.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A sitemap tells search engines which URLs you care about. It does not tell you whether those URLs still answer 200, whether they redirect, or whether half of them ship without a meta description. Desktop crawlers handle one-off audits well. This post covers the other case: a repeatable check that runs in the cloud, reads robots.txt and the sitemap first, and gives you one row per page that you can compare week to week.&lt;/p&gt;

&lt;p&gt;The tool is &lt;a href="https://apify.com/syntrovia/sitemap-seo-check-http?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=sitemap-audit" rel="noopener noreferrer"&gt;Sitemap &amp;amp; On-Page SEO Checker - HTTP Only, No Browser&lt;/a&gt; on Apify Store. It sends plain HTTP GET requests, so runs are cheap, but it does not run JavaScript.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it checks
&lt;/h2&gt;

&lt;p&gt;Before it requests a page, the Actor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;reads &lt;code&gt;robots.txt&lt;/code&gt; and follows its rules and &lt;code&gt;Crawl-delay&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;finds the sitemaps (the &lt;code&gt;Sitemap:&lt;/code&gt; lines in robots.txt, or &lt;code&gt;/sitemap.xml&lt;/code&gt; and &lt;code&gt;/sitemap_index.xml&lt;/code&gt;), including sitemap indexes up to 3 levels deep, gzip files and plain-text sitemaps;&lt;/li&gt;
&lt;li&gt;sends at most 2 requests at a time and 60 requests per minute by default.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For each page it records the status code, redirect target, title and meta description (with their lengths), H1, canonical, robots meta tag and &lt;code&gt;X-Robots-Tag&lt;/code&gt;, word count, link and image counts, and a list of issue codes. There are more than 35 codes, for example &lt;code&gt;missing-meta-description&lt;/code&gt;, &lt;code&gt;duplicate-title&lt;/code&gt;, &lt;code&gt;canonical-points-elsewhere&lt;/code&gt;, &lt;code&gt;redirect-chain&lt;/code&gt;, &lt;code&gt;noindex&lt;/code&gt;, &lt;code&gt;thin-content&lt;/code&gt; and &lt;code&gt;not-in-sitemap&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It stops when a site answers 401, 403, 407 or 429, or shows a CAPTCHA or a bot check instead of the content. A single login page is skipped and not charged; if most pages ask for a login, the site sits behind a login wall and the run stops with &lt;code&gt;stopReason: login-wall&lt;/code&gt;. Use it on sites you own or are authorised to audit; the input has a checkbox for exactly that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: list the sitemap first (cheap)
&lt;/h2&gt;

&lt;p&gt;Start with sitemap mode. No page is requested; you only get what the sitemaps say.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"startUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"authorizedToAudit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"sitemap"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"maxSitemapUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20000&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each row has &lt;code&gt;url&lt;/code&gt;, &lt;code&gt;sitemapUrl&lt;/code&gt;, &lt;code&gt;lastmod&lt;/code&gt;, &lt;code&gt;changefreq&lt;/code&gt;, &lt;code&gt;priority&lt;/code&gt;, &lt;code&gt;allowedByRobots&lt;/code&gt; and &lt;code&gt;issues&lt;/code&gt;. The sitemap-mode codes catch problems that are easy to miss by eye:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;blocked-by-robots&lt;/code&gt;: the sitemap lists a URL that robots.txt disallows;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;other-host&lt;/code&gt; and &lt;code&gt;insecure-http&lt;/code&gt;: URLs on another host, or plain-http URLs on an https site;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;invalid-lastmod&lt;/code&gt;, &lt;code&gt;invalid-changefreq&lt;/code&gt;, &lt;code&gt;invalid-priority&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Listing 5,000 URLs costs $0.005 + $1.50 = $1.505.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: audit the pages
&lt;/h2&gt;

&lt;p&gt;Switch &lt;code&gt;mode&lt;/code&gt; to &lt;code&gt;audit&lt;/code&gt; and cap the run with &lt;code&gt;maxPages&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"startUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"authorizedToAudit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"audit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"maxPages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"discoverFrom"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"sitemaps-and-links"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"includePageText"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"maxRequestsPerMinute"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A result row looks like this (trimmed from the example in the README):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com/pricing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"statusCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"foundIn"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"sitemap"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"inSitemap"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Pricing - Example"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"titleLength"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;17&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"metaDescription"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"h1"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Pricing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"h1Count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"canonicalStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"self"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"indexable"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"indexability"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"indexable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"wordCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;481&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"images"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"imagesWithoutAlt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"issues"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title-too-short"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"missing-meta-description"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"images-missing-alt"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A redirect gets its own row, with &lt;code&gt;statusCode&lt;/code&gt; 301 and the target in &lt;code&gt;redirectUrl&lt;/code&gt;. A same-site target is checked as a separate row, which is how &lt;code&gt;redirect-chain&lt;/code&gt; and &lt;code&gt;redirect-loop&lt;/code&gt; are found.&lt;/p&gt;

&lt;p&gt;One more code worth knowing: &lt;code&gt;not-in-sitemap&lt;/code&gt;. It marks an indexable page that the crawl reached through links but that no sitemap lists. It is only set when every sitemap was read to the end, so a partial read does not produce false alarms.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: pull the results and group them by issue
&lt;/h2&gt;

&lt;p&gt;You can download JSON, CSV or Excel from the Apify Console. For scripts, the Apify API has an endpoint that runs the Actor and returns the dataset in one request. It waits up to 300 seconds, so keep &lt;code&gt;maxPages&lt;/code&gt; small for this call: at 60 requests per minute, 100 pages take roughly two minutes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s -X POST \
  &lt;span class="s2"&gt;"https://api.apify.com/v2/actors/syntrovia~sitemap-seo-check-http/run-sync-get-dataset-items"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"Authorization: Bearer $APIFY_TOKEN"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"startUrl": "https://www.example.com/", "authorizedToAudit": true, "mode": "audit", "maxPages": 100}'&lt;/span&gt; \
  -o audit.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then count the issues and list the pages to fix first:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"audit.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"issues"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{n:5}  {code}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\nIndexable pages without a meta description:"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"indexable"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"missing-meta-description"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"issues"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each run also writes a &lt;code&gt;RUN_SUMMARY&lt;/code&gt; record to its key-value store: totals per issue, the robots.txt rules it applied, the sitemap files it read, and why it stopped (&lt;code&gt;max-pages&lt;/code&gt;, &lt;code&gt;all-urls-checked&lt;/code&gt;, &lt;code&gt;blocked&lt;/code&gt; and so on).&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: make it weekly
&lt;/h2&gt;

&lt;p&gt;Save the input as a Task, add a Schedule (for example every Monday at 07:00 in your time zone), and send the results on with an Apify integration such as Slack, e-mail or Google Sheets. Every Monday you get the same columns, so comparing with last week shows new 404s, new redirects and pages that lost their canonical.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Pay-per-event with three events; platform usage is included.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Price&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Run start (charged with the first saved result)&lt;/td&gt;
&lt;td&gt;$0.005&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Page checked and saved (audit mode)&lt;/td&gt;
&lt;td&gt;$0.0015&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sitemap URL listed and saved (sitemap mode)&lt;/td&gt;
&lt;td&gt;$0.0003&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The prefilled 20-page audit costs $0.035, and 1,000 pages cost $1.505. You are not charged for URLs that robots.txt disallows, for requests that got no answer, or for a run that saves nothing. You can also set a maximum cost per run.&lt;/p&gt;

&lt;h2&gt;
  
  
  When it is the wrong tool
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pages built with JavaScript.&lt;/strong&gt; The Actor reads the HTML your server sends. Titles, meta tags, canonicals and status codes are usually in it; body text built in the browser is not, and those pages get the &lt;code&gt;javascript-rendered&lt;/code&gt; code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Very large sites in one run.&lt;/strong&gt; One audit run handles up to 5,000 pages. With the default 1-hour timeout and 60 requests per minute, that is about 3,400 pages unless you raise the timeout.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sites you do not control.&lt;/strong&gt; It is meant for your own sites and for clients who asked you to audit theirs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Open the &lt;a href="https://apify.com/syntrovia/sitemap-seo-check-http?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=sitemap-audit" rel="noopener noreferrer"&gt;Actor page&lt;/a&gt;, keep the prefilled example or enter your own site, tick the authorisation box only if it is true, and press Start. If an issue code is unclear or a check you need is missing, open an issue on the Actor's Issues tab. I aim to answer within 72 hours.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>python</category>
    </item>
  </channel>
</rss>
