Submitting a product to SaaS and AI directories is the kind of task that looks perfect for browser automation: same five fields, same "Submit" button, fifty different sites. I spent a day driving 22 of them with Playwright for SongStory, a personalized song generator, and only 5 submissions actually went through.
The other 17 failed. Almost none of them failed for the reason I expected.
Here is the real distribution, because I think the failure modes are more useful than another "how to automate forms" tutorial.
The scoreboard
| Outcome | Count | Notes |
|---|---|---|
| Submitted successfully | 5 | 3 fully automated, 1 needed a human for reCAPTCHA, 1 by email |
| Hard paywall | 9 | No free tier at all, or free tier explicitly strips the link |
| Platform changed shape | 5 | Registration closed, site became app-only, field removed |
| Site-side bug or hard block | 2 | Broken form, Cloudflare |
| Bad neighborhood / wrong category | 3 | Would have hurt more than helped |
Note that "the automation broke" is not a row in that table. Playwright handled every form I pointed it at. What killed submissions was the state of the sites themselves.
Failure mode 1: the free tier that quietly removes your link
This was the most common paywall pattern, and the most interesting one.
One directory, bai.tools, offers a free listing with the line "Submit without backlink for free!" — a paid tier at $19 gets you the dofollow link. My first instinct was to skip it. Free tier, no link, no point.
That instinct was wrong, and it is worth explaining why. I checked an existing listing on that same site with two different user agents: a normal browser and a Googlebot UA fetched via page.request.get:
const r = await page.request.get(listingUrl, {
headers: { 'User-Agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' }
});
const html = await r.text();
// pull every outbound anchor and its rel attribute
const links = [...html.matchAll(/<a([^>]*?)href="(https?:\/\/[^"]+)"([^>]*)>/gi)]
.map(m => ({ href: m[2], attrs: (m[1] + m[3]).trim() }));
The outbound links on live listings came back as bare URLs with rel="dofollow", visible in the server-rendered HTML that Googlebot receives. The marketing copy and the actual markup did not agree.
The lesson generalizes: verify what the page renders, not what the pricing page claims. A site's own description of its free tier is marketing, not a measurement. Fetching one existing listing takes ten seconds and settles the question.
Failure mode 2: rendered DOM vs. what a crawler sees
The inverse trap is worse, because it looks like success.
I registered on a large music platform (2.3M monthly visits, 20 years old — on paper an excellent link) and got to a profile page where the outbound website links showed rel empty in DevTools. Dofollow, apparently.
Then I fetched the same URL with a Googlebot UA and found this in the raw HTML:
<a class="lhr2 text-white" ng-href="{{:: website.url}}" target="_blank"
ng-repeat="website in ctrl.websites | limitTo:(ctrl.limit || 4)">
An unrendered AngularJS template. The href attribute does not exist in the server response — it is produced client-side. Google does render JavaScript, but "probably fine" is not the same as "verified," and I had no way to confirm it.
That platform ended up failing for an unrelated reason anyway: the current version of its profile editor only accepts Spotify and Apple Music URLs. There is no free-form website field anymore, and the "Public Profile" menu item reads Coming Soon for new accounts. The dofollow section I had been looking at lives on legacy. subdomain profiles created years ago.
If you check rel only in the rendered DOM, you will report links that may not exist for a crawler. Check both. It costs one extra request.
Failure mode 3: forms that nobody can submit
One directory consumed thirty minutes before I understood it was unwinnable.
Every field filled cleanly. Clicking Submit did nothing — no error, no toast, no redirect. Following the "dump the response instead of guessing" rule, I attached a listener before clicking:
page.on('request', r => { if (r.method() === 'POST') console.log(r.url(), r.postData()); });
page.on('response', async r => {
if (r.request().method() === 'POST') console.log(r.status(), (await r.text()).slice(0, 300));
});
Zero POST requests. The click never produced a network call at all, which meant client-side validation was rejecting the form silently.
The cause turned out to be in the markup:
<input type="radio" name="form-name" value="audio_generators"> <!-- category -->
<input type="radio" name="form-name" value="freemium"> <!-- pricing -->
Two logically separate radio groups sharing one name. Browsers enforce mutual exclusivity within a name, so selecting a pricing model deselects your category and vice versa. Both are required. The form can never be completed — not by a script, not by a person. I confirmed by watching document.querySelectorAll('input[type=radio]:checked') flip back and forth.
Worth internalizing: when a button produces no network activity, stop retrying and start listening. Retrying a click ten times tells you nothing; one request listener tells you everything.
Failure mode 4: the link neighborhood
Two sites had genuinely good metrics — decent traffic, reasonable domain age, healthy organic share — and I walked away from both.
One had roughly twenty outbound links to Vietnamese gambling sites in its footer. Whatever that domain's metrics say, it is selling links to a gambling PBN, and a link from it sits in that neighborhood.
Another was a clean, free, no-login submission via Typeform. I got to step two before reading the category list: all 35 options were B2B SaaS operations tools — Analytics, DevOps, HR, Sales (B2B), QA Testing. A consumer gift product has no honest home there. Forcing it in would produce either a rejection or a listing filed under something irrelevant.
Both of those are judgment calls a script cannot make for you, and both require actually opening the page. No amount of metadata would have caught either one.
What actually worked
Five submissions landed. The three fully automated ones shared a shape: a real free tier, a website field, and either no captcha or an invisible one that passes on its own.
A few implementation notes that saved time:
Upload files by targeting the input directly. Do not click the upload button and handle a native file chooser — if the page navigates mid-dialog you lose every field you filled.
const input = await page.$('input[type="file"]');
await input.setInputFiles('/path/to/screenshot.png');
Watch the preview src. On WordPress listing themes it flips to something like /wp-content/uploads/listing-uploads/... once the async upload registers. If input.files has an entry but no preview appears, the front-end state never registered the file and it will not be submitted.
Rich text editors need their own API. fill() on the underlying textarea does nothing when TinyMCE is mounted on top:
await page.evaluate(html => {
window.tinymce.get('job_description').setContent(html);
}, descriptionHtml);
Select2 and similar widgets need real mouse events. Dispatching a synthetic click() inside page.evaluate will not register the selection. Use Playwright's locator click, which drives the actual input pipeline:
await searchBox.fill('music');
await page.waitForTimeout(2000); // async option load
await page.locator('.select2-results__option')
.filter({ hasText: /^Music$/ })
.first()
.click(); // real click, not evaluate
Watch for re-renders that wipe your work. One registration form re-rendered when I toggled an account-type radio, clearing every field I had filled and then rejecting the submit with "Please provide a valid email address." Order matters: set the option that causes the re-render first, then fill.
React controlled inputs ignore .value =. Use the native setter so React's onChange fires:
const setter = Object.getOwnPropertyDescriptor(
window.HTMLInputElement.prototype, 'value'
).set;
setter.call(el, value);
el.dispatchEvent(new Event('input', { bubbles: true }));
The part that stays manual
One submission required a human to tick a reCAPTCHA v2 checkbox. Clicking #recaptcha-anchor inside the anchor iframe left aria-checked="false" and produced no token, which is exactly what should happen.
The workable pattern is not to defeat it. Fill every other field first, scroll the widget into view, hand the window over, and submit the moment the token appears. Polling for it is trivial:
const token = () => page.evaluate(
() => document.querySelector('#g-recaptcha-response')?.value?.length || 0
);
Filling first matters more than it sounds. Captcha tokens expire, and some forms invalidate them on validation failure — solve it before the form is complete and you may burn it for nothing.
One more manual-ish case: a directory whose only submission channel was an email address. Its "Submit a tool" button did nothing and every guessable path 404'd, so the submission went out as a plain email with the name, URL, category and description. Not everything needs a form.
What I would tell myself before starting
The automation was never the hard part. Playwright filled every form correctly on the first or second attempt. What consumed the day was the gap between what sites advertise and what they actually do — free tiers that turn out to include the link, "dofollow" that only exists after JavaScript runs, forms shipped broken, and directories whose metrics look fine until you read the footer.
So: open the page. Fetch one existing listing with a crawler UA. Attach a request listener before you click. And when a site's copy disagrees with its markup, believe the markup.
If you want to see the product all this was for, it is SongStory — you describe a person and an occasion, it writes the lyrics for you to edit, then sings them. Which, unlike directory submission, turned out to be the easy part of the day.
Top comments (1)
The row that jumps out of your table is the one that isn't there: Playwright itself never being the failure. That matches what I've seen driving real logins through headless browsers — selectors and form-filling are the easy 10%, the other 90% is site state nobody documents (paywalls that contradict their own pricing page, flows that silently changed, blocks that look like bugs).
The Googlebot-UA audit of outbound rel attributes is going straight into my toolbox. One follow-up though: did you circle back weeks later to check whether the dofollow links you verified actually got indexed? I've run into directories that serve clean dofollow markup to crawlers but have such low crawl priority that Google never touches the listing pages — the markup audit passes and the link is still worthless. The markup is verifiable in ten seconds; indexing is a property you only observe after the fact.
Curious how you'd structure this at higher volume. With 22 sites, manual failure classification fits in a day. At 200, does the UA-check become a pipeline stage, and do you just blacklist the paywall cluster instead of re-attempting monthly?