DEV Community

build996
build996

Posted on

How to tell whether your DEV post is noindexed, and what decides it

I checked all 74 of my published posts on this site today. Seventy of them carry a noindex meta tag. Two don't. Two refused the request while I was checking.

Nothing told me this. The posts render normally, sit in the feed, have working URLs, and quietly instruct crawlers to skip them. I found out because I fetched my own page and read the HTML.

Here is how to check yours, and what the mechanism underneath actually is.

Checking

curl -s https://dev.to/<user>/<slug> | grep -o '<meta name="robots"[^>]*>'
Enter fullscreen mode Exit fullscreen mode

On my posts that returns three tags, in this order:

<meta name="robots" content="max-snippet:-1, max-image-preview:large, max-video-preview:-1">
<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">
Enter fullscreen mode Exit fullscreen mode

The noindex is the second one. The first is a perfectly innocuous snippet-policy tag that appears on healthy pages too. If you open "view source", hit Ctrl-F, find the first match and stop, you get a clean bill of health that is wrong. I did exactly that once and believed it for days. Read every match, not the first.

What decides it

From app/models/article.rb in the forem/forem repo:

def skip_indexing?
  !published ||
    (score < Settings::UserExperience.index_minimum_score && !featured) ||
    published_at.to_i < Settings::UserExperience.index_minimum_date.to_i ||
    score < -1
end
Enter fullscreen mode Exit fullscreen mode

So it comes down to the article's score. update_score sums about a dozen terms: reaction points, a spam adjustment, subscriber status, the author's featured count, context notes, a badge bonus, an organization baseline, and an automod label adjustment.

For an account with no reactions, no badges and no organization, every one of those terms is zero except the last. The article's score is effectively just the automod label:

AUTOMOD_SCORE_ADJUSTMENTS = {
  no_moderation_label: 0,
  great_and_on_topic: 20,
  very_good_and_on_topic: 15,
  okay_and_on_topic: 3,
  ok_but_offtopic_for_subforem: 0,
  likely_low_quality: -2,
  likely_spam: -5,
  clear_and_obvious_spam: -10,
  # ...
}
Enter fullscreen mode Exit fullscreen mode

The label itself comes from a classifier. The scoring table is open source; the prompt that picks which row you land on is not in the repo. You can read exactly what each label is worth and have no way to see which one you got.

Which means there is no recipe

That is the part I want to be straight about, because "how do I get the +20" is the question you actually came for, and I can't answer it. I tried three times:

  1. Rewrote a noindexed post in the exact style of the two of mine that were indexable. Twenty-four hours later: still noindexed.
  2. Cut the outbound links. My two indexable posts happened to have none in the body, and the failures had one each. Published a new post with zero links: still noindexed.
  3. Matched every observable feature at once — same length band, two fenced code blocks, same tags, no links. Forty-eight hours later: still noindexed.

Three surface-feature hypotheses, three refutations. My original evidence for the link theory was a 2-vs-2 split on four posts, which is exactly the kind of pattern that looks like a rule and isn't. If someone hands you the trick — short posts, code blocks, a particular tag — ask how many samples it rests on.

The two doors that don't involve guessing

Both live in the same update_score, and both are account-level, so they clear every post at once instead of one at a time:

established_user_adjustment =
  (user.score.to_i > 100 && !clear_and_obvious_spam? && !likely_spam?) ?
    Settings::UserExperience.index_minimum_score.to_i : 0
Enter fullscreen mode Exit fullscreen mode

And User#calculate_score:

calculated_score = (badge_achievements_count * 10) + user_reaction_points
Enter fullscreen mode Exit fullscreen mode

user_reaction_points counts vomit reactions, so it only ever pulls down. Which makes score > 100 mean one thing: 11 badges.

The second door is base_subscriber_adjustment — DEV++. That is no longer something you can buy; the page now describes it as awarded for challenges and other achievements.

I'm at 2 badges. I'm not going to pretend that's a strategy so much as a long walk.

This is not a complaint

The gate is a spam control and it is a reasonable one. An account that publishes seventy variations of the same post shape is precisely what it exists to catch, and that describes my own archive better than I'd like it to. I'd have flagged me too.

The part worth knowing isn't that the gate exists. It's that it is silent: there is no banner, no dashboard field, no email. Your post looks fine from every angle except the one that matters, and the only way to find out is to fetch the page yourself and read all three meta tags.

Worth thirty seconds on a post you assumed was working.

Top comments (0)