DEV Community: Vignesh

Faceted navigation on B2B catalogs: keep filters useful for buyers and cheap for crawlers

Vignesh — Sat, 07 Nov 2026 07:40:24 +0000

B2B product catalogs love filters. Material, size, thread, voltage, brand, certification. Buyers need them, because nobody scrolls through four thousand fasteners. But every filter combination can become its own URL, and crawlers will happily try them all.

A category with 8 filters and a handful of values each can produce many thousands of URLs that show almost the same products. Google's own guidance on managing crawling of faceted navigation URLs names the two costs: overcrawling, and slower discovery of new pages, because time spent on useless URLs is time not spent on your new products.

Here is how to keep filters for buyers without handing crawlers an endless maze.

Step 1: decide which filter pages deserve to exist

Most filter combinations are not pages anyone searches for. A few are. "Stainless steel M8 bolts" might be a real query with real buyers behind it. "Stainless steel, M8, 40 mm, zinc-free, sold in boxes of 200" is not.

Make a short list of the combinations that match real searches. Those become proper landing pages with a clean URL, their own title and a short introduction:

/bolts/stainless-steel-m8/

Everything else stays a filter for people, not a page for search engines.

Step 2: keep crawlers out of the rest

Google gives two straightforward options for filter URLs you don't need indexed.

Option A: block the filter parameters in robots.txt. This follows the pattern in Google's documentation:

user-agent: *
disallow: /*?*material=
disallow: /*?*size=
disallow: /*?*brand=

If you also want certain filtered views crawled, add specific allow: rules for them, and test the file before you ship it.

Option B: put filters in the URL fragment. Google Search generally doesn't support URL fragments in crawling and indexing, so a filter state like this stays with the visitor:

/bolts/#material=stainless-steel&size=m8

A small amount of client-side code reads the fragment and filters the list:

function readFilters() {
  const params = new URLSearchParams(location.hash.slice(1));
  return Object.fromEntries(params.entries());
}

function setFilter(key, value) {
  const params = new URLSearchParams(location.hash.slice(1));
  if (value) params.set(key, value); else params.delete(key);
  location.hash = params.toString();
}

window.addEventListener('hashchange', () => renderProducts(readFilters()));

Fragments suit catalogs where filtering happens in the browser. If filtering needs the server, Option A is usually simpler.

Step 3: if filter URLs must be crawlable, make them predictable

Some sites need filtered URLs indexed. In that case, Google's advice is to keep them tidy:

Use the standard & separator between parameters.
Keep the filters in the same order every time, so ?material=steel&size=m8 and ?size=m8&material=steel never both exist.
Return a real 404 when a combination has no products, instead of an empty page with a 200 status.

Step 4: know what canonical and nofollow can and can't do

Pointing filtered pages to the unfiltered category with rel="canonical" may reduce crawling of those URLs over time, according to Google. It is a hint, not a wall. rel="nofollow" on filter links can help too, but only if every link to that URL carries it, which is hard to guarantee across a whole site.

Use them as support for Step 2, not instead of it.

Step 5: check the result

After the change, watch the Crawl stats report in Search Console for a few weeks. You want to see fewer requests for parameter URLs and more for product and category pages. If you have server logs, filter them by Googlebot and count requests that contain your filter parameters.

Also check that your chosen landing pages from Step 1 are linked from the category page in plain HTML, so crawlers can still find them.

The short version

Decide which filter combinations are real landing pages, and build those properly.
Keep crawlers out of everything else with robots.txt rules or URL fragments.
If filter URLs must be crawlable, keep a fixed parameter order and return 404 for empty results.
Treat canonical and nofollow as hints.

My team at VTurnU covers more catalog basics in a guide to SEO for ecommerce stores.

Quarantine, Don't Delete: A Simple Spam Score for Contact Forms in Node.js

Vignesh — Fri, 06 Nov 2026 08:48:43 +0000

Most advice about contact form spam ends with "block it". Add a CAPTCHA, reject anything that looks odd, move on.

That works until a real buyer gets blocked. Or until the opposite happens: everything gets through, and the sales team spends its time calling phone numbers that don't exist. I saw the second problem on a manufacturer's website. The sales team was chasing fake enquiries instead of real buyers.

The approach we used there was to hold suspected fake leads in a quarantine instead of deleting them. Real leads go straight to sales. Suspected junk waits in a separate list, with the reasons attached. If a real enquiry lands there by mistake, someone can release it.

This post builds a version of that idea in plain Node.js, with no dependencies.

The idea

form submit ──► score it ──► score below 5 ──► deliver (CRM, inbox)
                         └─► score 5 or more ─► quarantine (kept, with reasons)

Three rules make it work:

Never delete. A quarantine costs almost nothing. A lost buyer costs a lot.
Every point has a reason a person can read. "Honeypot field filled" is useful. A bare number is not.
Bots get the same response either way. If spam got an error message, the sender would adjust until it stopped getting one.

Five cheap signals

None of these need a third-party service, and none of them track the visitor.

1. A honeypot field

Add a field people never see. Bots that fill every input will fill this one too.

<div aria-hidden="true" style="position:absolute;left:-9999px">
  <label>Leave this empty <input name="website" tabindex="-1" autocomplete="off"></label>
</div>

aria-hidden keeps it away from screen readers, and tabindex="-1" stops keyboard users from tabbing into it. Browser autofill can still fill it now and then. That is fine here, because a filled honeypot sends the lead to quarantine, not to the bin.

2. A signed time trap

People need more than a couple of seconds to type a message. Many bots post the moment the page loads.

Put a timestamp in a hidden field when you render the form, and sign it so nobody can fake an older one:

// spam-score.js
import { createHmac, timingSafeEqual } from 'node:crypto';

const secret = () => {
  const s = process.env.FORM_SECRET;
  if (!s || s.length < 32) throw new Error('Set FORM_SECRET to a long random string');
  return s;
};

const sign = (value) => createHmac('sha256', secret()).update(value).digest('base64url');

// Put this in a hidden input when you render the form.
export function issueFormToken(now = Date.now()) {
  const ts = String(now);
  return `${ts}.${sign(ts)}`;
}

// On submit: seconds since the form was rendered, or null if the token is missing or forged.
export function secondsSinceRender(token, now = Date.now()) {
  const [ts, sig] = String(token ?? '').split('.');
  if (!ts || !sig || !/^\d+$/.test(ts)) return null;
  const given = Buffer.from(sig);
  const expected = Buffer.from(sign(ts));
  if (given.length !== expected.length || !timingSafeEqual(given, expected)) return null;
  return (now - Number(ts)) / 1000;
}

A missing or forged token returns null, and that counts as suspicious. timingSafeEqual compares the signatures without leaking information through timing.

3. Links in the message

Real enquiries rarely contain more than one link. Spam often has several, or forum-style [url=...] markup. One link adds a point. Three or more add four.

4. A phone number that can't exist

Under the ITU E.164 standard, international phone numbers have at most 15 digits. A number with fewer than 7 digits is almost never reachable. This is the check that speaks most directly to the problem I described at the start.

5. Bursts from the same sender

The same email and IP sending three times in ten minutes is worth a few points. A person might send twice by accident. A script sends twenty times.

The scoring function

// spam-score.js (continued)
const LINK = /https?:\/\/|www\./gi;

export function scoreSubmission(form, { secs, recentFromSameSender = 0 } = {}) {
  const reasons = [];
  let score = 0;
  const add = (points, reason) => {
    score += points;
    reasons.push(reason);
  };

  // 1. Honeypot: a field people never see.
  if (String(form.website ?? '').trim()) add(6, 'honeypot field filled');

  // 2. Time trap: people need a few seconds to type a message.
  if (secs === null || secs === undefined) add(5, 'form token missing or forged');
  else if (secs < 3) add(5, `sent ${secs.toFixed(1)}s after the form loaded`);
  else if (secs > 60 * 60 * 24) add(2, 'form token older than 24 hours');

  // 3. Content: links and link markup are rare in real enquiries.
  const message = String(form.message ?? '');
  const links = (message.match(LINK) ?? []).length;
  if (links >= 3) add(4, `${links} links in the message`);
  else if (links > 0) add(1, `${links} link(s) in the message`);
  if (/\[url=|<a\s/i.test(message)) add(3, 'link markup in the message');

  // 4. Field sanity: international numbers have at most 15 digits (E.164).
  const digits = String(form.phone ?? '').replace(/\D/g, '');
  if (form.phone && (digits.length < 7 || digits.length > 15)) {
    add(3, `phone number has ${digits.length} digit${digits.length === 1 ? '' : 's'}`);
  }

  // 5. Bursts: the same email or IP sending again within a few minutes.
  if (recentFromSameSender >= 2) add(3, `${recentFromSameSender} earlier sends in 10 minutes`);

  return { score, reasons };
}

export const QUARANTINE_AT = 5;

export function decide({ score }) {
  return score >= QUARANTINE_AT ? 'quarantine' : 'deliver';
}

Treat the weights as a starting point. A filled honeypot or a post under three seconds is enough on its own to reach quarantine. Softer signals, like a single link, need company before they matter.

Wiring it into a form endpoint

Here is the POST handler from a minimal node:http server. The full file renders the form with a fresh token on GET.

// server.js (POST /contact)
let body = '';
for await (const chunk of req) body += chunk;
const form = Object.fromEntries(new URLSearchParams(body));
const now = Date.now();
const sender = `${String(form.email ?? '').toLowerCase()}|${req.socket.remoteAddress}`;

const result = scoreSubmission(form, {
  secs: secondsSinceRender(form.form_token, now),
  recentFromSameSender: countRecent(sender, now), // a Map with a 10-minute window; use Redis or your database in production
});

const lead = { id: randomUUID(), receivedAt: new Date(now).toISOString(), ...form, ...result };
delete lead.form_token;

if (decide(result) === 'deliver') await deliver(lead);   // your normal path: CRM, inbox, autoresponder
else await quarantine(lead);                              // append to a table or file, with the reasons

// Same answer either way, so bots learn nothing from the response.
res.writeHead(303, { location: '/thanks' });
res.end();

Two details matter more than they look:

The response is identical for both paths. The bot gets the same redirect as a real buyer.
The quarantined record keeps the whole submission plus the reasons. That is what makes a release possible later.

A quick run against this server shows the difference. An instant, link-stuffed post with a filled honeypot scored 18, and the log read: honeypot field filled; sent 0.0s after the form loaded; 3 links in the message; phone number has 1 digit. A normal enquiry sent a few seconds after the page loaded scored 0 and was delivered.

Testing it

Node's built-in test runner is enough:

// spam-score.test.js
import test from 'node:test';
import assert from 'node:assert/strict';
import { scoreSubmission, decide } from './spam-score.js';

const human = {
  name: 'Priya Raman',
  email: 'priya@example.com',
  phone: '+44 20 7946 0958',
  message: 'We need 5,000 waterproof labels for bottles. Can you share a quote?',
  website: '',
};

test('a normal enquiry is delivered', () => {
  const result = scoreSubmission(human, { secs: 48 });
  assert.equal(result.score, 0);
  assert.equal(decide(result), 'deliver');
});

test('a filled honeypot goes to quarantine, not the bin', () => {
  const result = scoreSubmission({ ...human, website: 'https://cheap-seo.example' }, { secs: 48 });
  assert.equal(decide(result), 'quarantine');
  assert.ok(result.reasons.includes('honeypot field filled'));
});

test('one link alone does not quarantine a real enquiry', () => {
  const message = 'Our current labels are here: https://example.com/products. Can you match them?';
  assert.equal(decide(scoreSubmission({ ...human, message }, { secs: 40 })), 'deliver');
});

Run it with FORM_SECRET=<a long random string> node --test. I'd add a test for every false positive you find in real life, so the same mistake can't come back.

Running the quarantine

The code is the easy part. The habit is what keeps it working.

Check the quarantine once a day, or send yourself a short daily digest. It takes a minute.
Make "release" run the normal path. A released lead should reach the CRM, the inbox and the autoresponder like any other lead.
Count two numbers every week: leads you had to release (false positives) and junk that reached sales (false negatives). Adjust the weights when either one grows.
Expire quarantined records, for example after 30 days, and keep only the personal data you need. Make sure this matches your privacy policy.

These signals catch bots. Some junk comes from real people, such as fake "brand partnership" offers that pass every automated check. I covered how to spot that kind, along with the other common types of junk lead, in a playbook on stopping junk leads.

What I'd avoid

Blocking by country or language. If you sell to buyers in many countries, an enquiry from an unexpected place or in an unfamiliar script may be your best lead of the month.
A CAPTCHA on every form as the first step. It adds friction for every real visitor. Try the cheap signals first, and add a challenge later only for traffic that keeps scoring high.
Silently dropping anything. If you can't explain why a lead disappeared, you can't fix the rule that took it.

Wrapping up

Filtering lead form spam isn't about catching every bot. It's about making sure no real buyer gets lost, while your team stops chasing fake ones. A score, a readable reason for every point, and a quarantine someone actually checks will get you most of the way there.

If you run something similar, I'd like to know which signals work best on your forms.

Strong reasons why your Mobile App needs Video Integration

Vignesh — Thu, 23 May 2019 11:14:24 +0000

Nearly 3.6 million mobile apps is readily available in Google Play Store and 2.2 million in apple store for downloads. Most of the new comers get confused during the selection process of a mobile apps and this immediately raises a question among app developers. What should be featured within a Mobile Apps to make it attractive and sustainable over the app store?

Mobile App Developers are working towards integrating videos in mobile apps and then upload self-learning, intuitive videos to make aware the user with the existing features. The next thing about evolving mobile apps is its value proposition derived among customers and current prospects. This is because video integration engages the customers to provide a multidimensional feedback and then later enable brands to be built.

Video Integration in mobile apps proves to be an excellent way of communication with all kinds of customers. It is so powerful that it literally eliminates the need for content marketing strategies. It has defined the best way of reaching to customers and create the brand awareness.
Given here are the 5 reasons that affirms to video Integration in mobile apps.

User Interface

Generally mobile apps requires the best functional UI and UX design to run the videos swiftly. This video integration within apps gives the user the much needed introduction towards accessing mobile app features. Small Advertisements can also be used to serve users understand the functionalities of mobile app. This should certainly improve in the overall user satisfaction and engagement.

E.g. Many of the home appliance stores have started to present their products through an explainer video to make aware of the buyer intrinsically. This should enable any mobile users to easily install their dishwasher, clothes dryer and stoves without any difficulty.

Statistics

Statistics continuous to play a major role in no of mobile adoptions and usage while influencing the strategy for mobile apps. Moreover the no of mobile users getting addicted towards viewing videos using mobile devices is on the raise. It is an opportunity for every mobile app developers to integrate videos into their mobile apps at once and get benefitted largely.

E.g. Now you have a lot of time-tracking mobile apps that integrates videos for conveying easy thoughts. New Users will find it very much useful as it has the set reminder on time logs for frequenting.

Promotion

One of the reasons for videos to be built-in mobile apps is affirmatively for the promotional purpose. This could be a 30 seconds video consisting of screenshots explaining about app’s functionality and making you aware about brands. Promotional Videos can be seen in Google Play and iOS Stores freely. There are many app promoters who can sort the best of video integration in mobile apps for achieving easier presentation.

E.g. Generally any gaming apps requires the integration of a video to explain its core parts and features. One of them is Angry Birds gaming apps;

Social Media

Social Media in this current-age is vastly flooded with number of video ads that eventually attracts many smartphone and tablet users to view them instantly. It should be a valid reasons to develop integrated video within mobile apps and then linked to various social media platforms.

E.g. Generally mobile apps have video content that’s part of their excerpts linked to social media. Studying from the user’s behaviour it is observed to be that these videos are viewed multiples times and learnt fast.

Customer Expectation

When there are 6.5 million apps available in the play store and ensuring that your mobile app have its presence is going to be a tedious task. It requires the app developers to be experts in creating video contents that’s appropriately fits into with the mobile app and make it sustainable.

E.g. Most often the Net Banking apps requires the form of video integration to comfort the mobile user and then offers proof of financial receipts.

Conclusion
Every reasons about the Video Integration in Mobile Apps certainly proves that the best way of conveying message is through Videos and Images. Thereafter any app development process should first confirm to the excellence in app design and quality before Video Integration part. Alas it is going to be the responses that’s created in the play store that should decide about video integration in mobile apps.