DEV Community: Colin McDermott

Telegram Channel Developer Cheatsheet

Colin McDermott — Wed, 09 Aug 2023 00:21:37 +0000

Setting Up a Telegram Channel

Start: Open Telegram and tap the hamburger menu (≡) or the pencil icon (on mobile).
Choose Channel Type: "New Channel".
Name & Photo: Set a name, add a photo, and write a brief description.
Type:
- Public: Unique username, can be found through Telegram search.
- Private: Generates a link; whoever has the link can join.

Managing Your Channel

Add Admins:
- Go to Channel Info > Administrators > Add Admin.
- Assign specific rights.
Manage Subscribers:
- View List: Channel Info > Subscribers.
- Remove: Tap on a member > Remove.
Post Messages:
- Simply type and send in the message bar.
- You can pin messages for visibility.
Edit Channel:
- Go to Channel Info > Edit (or the pencil icon).
- Change name, photo, description, or type.

Enhancements

Bots: Integrate bots to automate tasks.
- @ControllerBot: Schedule posts.
- @Manybot: Custom commands and posts.
Link Channels: Share one channel's post to another.
Discussion Groups: Link a group chat for members to discuss posts.
- Edit Channel > Discussion > Link Group.
Statistics: View detailed stats if your channel has over 1000 members.
- Channel Info > Statistics.

Safety & Privacy

Turn On Two-Step Verification:
- Settings > Privacy and Security > Two-Step Verification.
Backup Channel:
- Regularly back up content and subscriber lists.
Limit Spam:
- Use bots like @Shieldy_bot to prevent spam in linked discussion groups.
Check Links: Before sharing, ensure links are safe.
Restrict Forwarding: Limit the ability for subscribers to forward your posts to protect original content.
Reporting: Encourage subscribers to report suspicious activity or spam.
Silent Messages: Send a message without notification by holding the send button and selecting "Send without sound".

Promoting Your Channel

Share Link: Share the invite link on social media, websites, etc.
Cross-Promotion: Collaborate with other channel admins to promote each other.
Listing: Add your channel to Telegram directory websites like tlgrm.eu/channels.
Engaging Content: Regular, valuable, and interactive content retains and attracts members.

Charging for access to your Telegram channel

Whop Telegram Bot: Sell access to your channel.

Tools & Resources

Telegra.ph: Create rich articles to share on Telegram.
Telemetr: Submit your channel.

Useful Commands

@channel: Notify all subscribers in a post.
#hashtag: Categorize and find related content.
$ticker: For stock or crypto channels, indicates a ticker symbol.

REST API Key Concepts and Requirements

Colin McDermott — Fri, 21 Apr 2023 22:15:06 +0000

RESTful APIs are a type of web API, which can be understood as a gateway between clients and resources on the web. Clients, which could be a person or another software system, interact with resources, such as images, videos, text, numbers, or any type of data, through this API.

This exchange is managed by the server, which is the machine providing the resource. APIs are designed to maintain security, control, and authentication, and they determine which clients get access to specific internal resources.

REST, which stands for Representational State Transfer, is a software architectural style that provides guidelines for how an API should work.

It was initially created to manage communication on complex networks like the internet, offering support for high-performing and reliable communication at scale.

APIs that follow the REST architectural style are called REST APIs or RESTful APIs interchangeably. Web services implementing this architecture are referred to as RESTful web services.

Stateless

Each request from the client must contain all the necessary information for the server to process the request. The server should not store any information about the client's state between requests, making the API stateless. This principle allows for better scalability and easier maintenance.

Client-Server Architecture

A REST API is based on a client-server architecture, where the client is responsible for the user interface and the server handles data storage and processing. This separation of concerns allows for improved flexibility, scalability, and maintainability.

Cacheability

To optimize performance, REST APIs should support caching. Responses from the server should indicate whether they can be cached or not, so the client can reuse cached responses when appropriate. This reduces the load on the server and improves overall performance.

Layered System

A REST API should be built using a layered architecture, where each layer performs a specific function. This separation of concerns makes it easier to maintain and evolve the system over time. For example, an API might have a security layer, a business logic layer, and a data access layer.

Uniform Interface

A REST API should have a consistent and uniform interface, which simplifies its usage and makes it more intuitive. This involves using standard HTTP methods, clear resource naming conventions, and providing descriptive error messages.

Code on Demand (optional)

While not always implemented, REST APIs can support the ability to extend client functionality through downloadable code, such as JavaScript. This feature can be useful in specific scenarios but is not a mandatory requirement for a REST API.

Idempotency

Idempotent operations are those that can be performed multiple times without changing the result beyond the initial application. In the context of REST APIs, GET, PUT, and DELETE methods should be idempotent, ensuring that repeated requests have the same effect as a single request.

Resource Nesting

In some cases, it makes sense to nest resources within other resources to represent relationships. For example, you might represent a user's comments on a blog post with a URL like /posts/123/comments. Use nesting sparingly and only when it reflects the natural hierarchy of the data.

Filtering, Sorting, and Searching

In addition to basic filtering and sorting, consider implementing more advanced search capabilities for your API, such as full-text search or complex querying. This allows clients to quickly locate and retrieve relevant resources.

Also check out:

API Rate Limiting Cheat Sheet

Colin McDermott — Fri, 14 Apr 2023 23:10:21 +0000

Jump to a section:

Gateway-level rate limiting

Token bucket algorithm

Leaky bucket algorithm

Sliding window algorithm

Distributed rate limiting

User-based rate limiting

API key rate limiting

Custom rate limiting

Gateway-level rate limiting

Gateway-level rate limiting is a popular approach to rate limiting that allows developers to set rate limits at the gateway level.
Gateway-level rate limiting is typically implemented in API gateways such as Kong, Google's Apigee, or Amazon API Gateway.
Gateway-level rate limiting can provide simple and effective rate limiting, but may not offer as much fine-grained control as other approaches.

Token bucket algorithm

image source

The token bucket algorithm is a popular rate limiting algorithm that involves allocating tokens to API requests.
The tokens are refilled at a set rate, and when an API request is made, it must consume a token.
If there are no tokens available, the request is rejected.
The token bucket algorithm is commonly used in many rate limiting libraries and tools, such as rate-limiter, redis-rate-limiter, and the Google Cloud Endpoints.

More: Token Bucket vs Bursty Rate Limiter by @animir

Leaky bucket algorithm

image source

The leaky bucket algorithm is similar to the token bucket algorithm, but instead of allocating tokens, API requests are added to a "bucket" at a set rate.
If the bucket overflows, the requests are rejected.
The leaky bucket algorithm can be useful for smoothing out request bursts, and for ensuring that requests are processed at a consistent rate.

Sliding window algorithm

image source

The sliding window algorithm is a rate limiting approach that involves tracking the number of requests made in a sliding window of time.
If the number of requests exceeds a set limit, further requests are rejected.
The sliding window algorithm is commonly used in many rate limiting libraries and tools, such as Django Ratelimit, Express Rate Limit, and the Kubernetes Rate Limiting.

More: Rate limiting using the Sliding Window algorithm by @satrobit

Distributed rate limiting

For high-traffic APIs, it may be necessary to implement rate limiting across multiple servers.
Distributed rate limiting algorithms such as Redis-based rate limiting or Consistent Hashing-based rate limiting can be used to implement rate limiting across multiple servers.
Distributed rate limiting can help to ensure that rate limiting is consistent across multiple servers, and can help to reduce the impact of traffic spikes.

In this example, we'll create a simple Next.js application with a rate-limited API endpoint using Redis and Upstash. Upstash is a serverless Redis database provider that allows you to interact with Redis easily and cost-effectively.

First, let's create a new Next.js project:

npx create-next-app redis-rate-limit-example
cd redis-rate-limit-example

Install the required dependencies:

npm install upstash-redis@0.4.4 ioredis@4.27.6 express-rate-limit@5.3.0

Create a .env.local file in the project root to store your Upstash Redis credentials:

UPSTASH_REDIS_URL=your_upstash_redis_url_here

Replace your_upstash_redis_url_here with your actual Upstash Redis URL.

Create a new API route in pages/api/limited.js:

import { connectRedis } from '../../lib/redis';
import rateLimit from 'express-rate-limit';
import { createError } from 'micro';

const redisClient = connectRedis();

const rateLimiter = rateLimit({
  store: new RedisStore({
    client: redisClient,
  }),
  windowMs: 60 * 1000, // 1 minute
  max: 5, // limit each IP to 5 requests per minute
  handler: (req, res) => {
    res.status(429).json({ message: 'Too many requests, please try again later.' });
  },
});

export default async function handler(req, res) {
  try {
    await rateLimiter(req, res);
  } catch (error) {
    if (error instanceof createError.HttpError) {
      return res.status(error.statusCode).json({ message: error.message });
    }
    res.status(500).json({ message: 'Internal server error' });
  }

  res.status(200).json({ message: 'Success! Your request was not rate-limited.' });
}

export const config = {
  api: {
    bodyParser: false,
  },
};

Create a lib/redis.js file to handle Redis connections:

import Redis from 'ioredis';

let cachedRedis = null;

export function connectRedis() {
  if (cachedRedis) {
    return cachedRedis;
  }

  const redis = new Redis(process.env.UPSTASH_REDIS_URL);
  cachedRedis = redis;
  return redis;
}

Create a new RedisStore class in lib/redis-store.js:

import { connectRedis } from './redis';

export class RedisStore {
  constructor({ client } = {}) {
    this.redis = client || connectRedis();
  }

  async get(key) {
    const data = await this.redis.get(key);
    return JSON.parse(data);
  }

  async set(key, value, ttl) {
    await this.redis.set(key, JSON.stringify(value), 'EX', ttl);
  }

  async resetKey(key) {
    await this.redis.del(key);
  }
}

Now you can test your rate-limited API endpoint by starting the development server:

npm run dev

Visit http://localhost:3000/api/limited in your browser or use a tool like Postman or curl to make requests. You should see the Success! Your request was not rate-limited. message. If you make more than 5 requests within a minute, you'll receive the rate limit message:

Too many requests, please try again later.

User-based rate limiting

Some APIs may require rate limiting at the user level, rather than the IP address or client ID level.
User-based rate limiting involves tracking the number of requests made by a particular user account, and limiting requests if the user exceeds a set limit.
User-based rate limiting is commonly used in many API frameworks, such as Django Rest Framework, and can be implemented using session-based or token-based authentication.

API key rate limiting

For APIs that require authentication with an API key, rate limiting can be implemented at the API key level.
API key rate limiting involves tracking the number of requests made with a particular API key, and limiting requests if the key exceeds a set limit.
API key rate limiting is commonly used in many API frameworks, such as Flask-Limiter, and can be implemented using API key-based authentication.

Custom rate limiting

Finally, it's worth noting that there are many other rate limiting approaches that can be customized to suit the needs of a particular API.
Some examples include adaptive rate limiting, which adjusts the rate limit based on the current traffic load, and request complexity-based rate limiting, which takes into account the complexity of individual requests when enforcing rate limits.
Custom rate limiting approaches can be useful for optimizing the rate limiting strategy for a specific API use case.

For my latest project Pub Index API I am making use of an API gateway for rate-limiting.

More: RESTful API Design Cheatsheet

RESTful API Design Cheatsheet

Colin McDermott — Sun, 09 Apr 2023 11:04:40 +0000

RESTful API design is a complex process that involves many factors, including scalability, performance, security, and user experience. This cheat sheet provides a quick reference for best practices and common pitfalls to avoid when designing RESTful APIs.

HTTP request methods

HTTP request methods/HTTP verbs are used to perform operations on resources in a RESTful API. Each HTTP verb has a specific purpose, and using them correctly is essential for API consistency and security. Here are the most common HTTP verbs and their usage:

GET: retrieve a resource
POST: create a new resource or perform an action on a resource
PUT: update an existing resource
PATCH: partially update an existing resource
DELETE: delete a resource

Naming Conventions

Resource naming conventions are crucial for API readability and usability.

Use plural nouns for collections of resources (e.g., /users)
Use singular nouns for individual resources (e.g., /users/123)
Use lowercase letters and hyphens for resource names (e.g., /users/123/orders)
Use nouns instead of verbs in resource names (e.g., /orders/123 instead of /get-order/123)

Resource Pagination

Resource pagination is necessary when dealing with large collections of resources.

Use limit and offset parameters for pagination (e.g., /users?limit=10&offset=20)
Use a default limit and let the client control the offset
Include pagination metadata in the response headers (e.g., X-Total-Count)

Error Responses

Error responses are crucial for API usability and should be consistent across the API.

Use HTTP status codes to indicate the error type (e.g., 400 for bad request, 401 for unauthorized, 404 for not found, 500 for internal server error)

Include a descriptive error message in the response body
Use a consistent error format across the API

Authentication and Authorization

Authentication and authorization are essential for API security.

Use a secure authentication mechanism (e.g., OAuth 2.0, JWT)
Use HTTPS for secure communication
Use role-based access control (RBAC) to restrict access to resources
Use appropriate scopes to restrict access to specific actions

Versioning

API versioning is necessary when changes are made to the API, and backward compatibility needs to be maintained.

Use a version number in the URI (e.g., /v1/users)
Use a version header in the request (e.g., Accept-Version: v1)
Deprecate old versions and provide a migration path

Caching

Caching can improve API performance and reduce server load. Here are some best practices for API caching:

Use appropriate cache headers (e.g., Cache-Control, Expires)
Use ETags for resource validation
Consider using a content delivery network (CDN) for caching

HATEOAS

HATEOAS (Hypermedia as the Engine of Application State) is a RESTful API design principle that enables clients to discover and interact with resources dynamically.

Include links to related resources in the response
Use standard link relations (e.g., self, next, prev)
Use media types that support hypermedia (e.g., HAL, JSON-LD)

Filtering and Sorting

Use query parameters for filtering (e.g., /users?status=active)
Use query parameters for sorting (e.g., /users?sort=name_asc)

Rate Limiting

Use HTTP headers to communicate rate limits (e.g., X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset)
Apply rate limiting based on client IP, authenticated user, or API key

Content Negotiation

Use the Accept header to specify desired response format (e.g., application/json)
Use the Content-Type header to specify the format of the request payload (e.g., application/json)

API Documentation

Provide clear and comprehensive documentation
Use tools like Swagger/OpenAPI or RAML for API documentation
Include examples for different request scenarios

CORS (Cross-Origin Resource Sharing)

Implement CORS headers to control which domains can access your API
Use Access-Control-Allow-Origin, Access-Control-Allow-Methods, Access-Control-Allow-Headers, and Access-Control-Max-Age headers as needed
Ensure that preflight requests (OPTIONS) are properly handled

API Monitoring and Testing

I have recently been using a service called Checkly, combined with Instatus for my API monitoring/status alerting. You can view my status page here.

Monitor API performance and uptime
Use tools like Postman, Insomnia, or curl for manual testing
Implement automated testing for critical API functionality

I put together this REST API cheatsheet to assist with the production/planning of my own API documentation, hopefully it is useful for someone reading.

The 9 Key Questions for Google's Helpful Content Update

Colin McDermott — Sat, 27 Aug 2022 22:39:00 +0000

Google's new AI powered Helpful Content Update is rolling out right now.

SEO experts are predicting it could be quite a big update.

Planning a quick content audit is something you might want to consider.

Here are the 9 questions Google recommend you ask in relation to your content.

Is the content primarily to attract people from search engines, rather than made for humans?
Are you producing lots of content on different topics in hopes that some of it might perform well in search results?
Are you using extensive automation to produce content on many topics?
Are you mainly summarizing what others have to say without adding much value?
Are you writing about things simply because they seem trending and not because you'd write about them otherwise for your existing audience?
Does your content leave readers feeling like they need to search again to get better information from other sources?
Are you writing to a particular word count because you've heard or read that Google has a preferred word count? (No, we don't).
Did you decide to enter some niche topic area without any real expertise, but instead mainly because you thought you'd get search traffic?
Does your content promise to answer a question that actually has no answer, such as suggesting there's a release date for a product, movie, or TV show when one isn't confirmed?

If you run an ecommerce site we have put together a guide to the Helpful Content Update that you can read here.

Introduction to Amazon Advertising

Colin McDermott — Fri, 12 Aug 2022 19:48:00 +0000

Amazon dominates the ecommerce market with over 300 million active users. Amazon has also become the de-facto product search engine -- so much so that close to 75% of all USA consumers begin their product search journey on the Amazon.com marketplace.

So, it is no wonder that Amazon Advertising (formerly Amazon Marketing Services) is becoming an increasingly valuable tool for businesses of all sizes. A testimony to its popularity, Amazon Advertising generated a staggering £23 billion ($31 billion) in revenue in 2021 as brands vied to increase their visibility, product awareness, and sales.

Using Amazon Advertising, brands can promote their products through a plethora of sponsored campaign options - to gain visibility, brand awareness, and sales; and to make their product offerings stand out among their competitors.

However, for the uninitiated, Amazon Advertising can be very complicated and ever-changing. Therefore, companies must stay on top of the latest changes and develop winning Amazon advertising strategies for their business.

Read on for our introduction to Amazon Advertising.

Amazon Advertising: Types of Ads

The basic concept of Amazon Advertising is to increase visibility and sales on Amazon.

Tips for successfully setting up Amazon ads

Many factors go into successful Amazon advertising. First, of course, Keyword research helps identify winning keywords people use when looking for products. Based on this, marketers can create an optimized campaign to put things into action.

KEYWORD RESEARCH: Identify the right keywords to target the customers. Maintain a Search term catalog based on current trends, own branded search terms, and competitor analysis.
BIDDING: Use amazon's bidding system to get maximum returns on investment.
- Automated campaigns use product-level bids and Amazon's algorithm to match advertisers with customers.
- Manual campaigns provide more control as it allows advertisers to specify a bid for each keyword, competitor product, or keyword.
AD CREATIVES: Develop ad copy and images that stand out from the competition. Focus on A+ brand content and focus on Reviews and Reviews to drive growth.
REPORTING, ANALYTICS: Closely monitor the Amazon advertising campaigns to optimize results. And perform course correction and benchmarking as needed.

KPIs to focus on while advertising on Amazon

CTR (Click Through Rate)
ACoS (advertising cost of sales)
TACoS (total advertising cost of sales)
ROaS - Return on Advertising Spend
Search impressions
Conversion rate
CPC - Cost per Click

We often get caught up in a variety of shiny new metrics that marketers love but let us not forget classics like click-through rate (CTR).

A low CTR means that ads are not compelling enough for people who see them--and this could be leading companies down an unwanted advertising path!

CTR remains the single most important KPI for most marketers while measuring ad effectiveness. The Click-through rate (CTR) is calculated by dividing an advertisement's clicks by impressions. For example, if an ad campaign gets 100 clicks for every 1000 impressions, the CTR will be 100/1000, i.e., 10%.

When an ad does not successfully engage the buyer, it needs to be further optimized, or we should fix any lingering listing issues on the product detail page.

For example, a low click-through rate means buyers are not finding what they want, so keyword targeting will be essential to see how great of an offer it is on Amazon.

A few of the essential listing quality factors that affect Amazon CTR include:

Main image quality
Title quality
Number of reviews
Review score
Fulfillment method (FBA or FBM)
Pricing
Ad Positioning
Conversion Rate
Earn A Best-Seller Badge

Factors like A+ brand content, SEO optimization, reviews, and ratings are crucial factors -- and when they are optimized, Amazon ads tend to perform better. In addition, the higher a product ranks, the more effective the ad campaign becomes--- saving money and driving sales.

Remember that the first step to success at Amazon is creating a helpful product page. Also, a general keyword, such as "shampoo" will not work as effectively as a specific one like "shampoo for men."

However, the more detailed the keyword is, the higher the eventual search rank, CTR, and conversion rate.

Amazon Ads vs. Google Ads

Although every customer journey will differ from one brand to another, they are mostly made up of the following five stages: Awareness, Consideration, Retention, Decision, and Advocacy.

AMAZON: When someone searches Amazon, they are more likely to be at the end of their buyer's journey (i.e., close enough on a buying decision). This means that people who buy from Amazon already know what they want and need to find out whether any good deals are available!
GOOGLE: With Google, search queries cover diverse topics - covering all stages of a buyer's journey. As a result, the ad campaign strategy on Google can be much more flexible, targeting buyers at earlier stages of their journey. Nevertheless, not every buyer in the early part of the buyer's journey will be ready to buy in the near future (if at all).

Amazon is, therefore, a good (and perhaps optimal) advertisement tool for product-based companies because they can skip the initial stages and directly get in front of interested shoppers.

So what does this mean for ecommerce marketers?

First, Amazon advertising should be a top priority. By not utilizing Amazon's advertising platform, marketers are missing one of the world's largest marketplaces.

And by not being up to date on the latest changes and strategies, they are putting their brands at a disadvantage against competitors.

Related: check out our in-depth guide to ecommerce SEO.

How do you know if your ecommerce site is hit by a Google Core Update?

Colin McDermott — Sun, 24 Jul 2022 13:00:00 +0000

What are Core Updates?

Several times a year, we make significant, broad changes to our search algorithms and systems. We refer to these as "core updates." They're designed to ensure that overall, we're delivering on our mission to present relevant and authoritative content to searchers. These core updates may also affect Google Discover.

What site owners should know about Google's core updates @ Google

A few times a year Google push out Core Updates which shake up search results with the aim of improving results for Google users.

If you own or manage an ecommerce website's SEO, you may notice that your site has been impacted by a Google Core Update and wonder what you can do.

How do you know if your ecommerce website has been hit by a Core Update?

Check your analytics and Search Console to see if you can pinpoint the approximate date your rankings/traffic was hit.

Then see if this matches up with a date in a list of algorithm/Google Core Updates or a recent update announced by Google's Search Liaison on Twitter.

Google also recently released their own list of Core Updates here.

What are the symptoms of being hit by a Core Update?

Usually you will see your rankings and traffic to your products and categories go down, and will mostly likely see a fall in sales also.

What to do about it?

Make sure you are following all ecommerce SEO best practice by checking our guide to ecommerce SEO.

Google also have a guide to SEO basics you can read here.

What we would recommend is to:

Not panic
Improve the quality of your site
Make sure your site does not have any technical issues
Audit the quality of your content
Audit your site for low quality pages that are unlikely to answer the queries they are trying to rank for

We hope this quick guide was useful.

What is Indexifembedded?

Colin McDermott — Sat, 07 May 2022 22:52:51 +0000

What is Indexifembedded?

Indexifembedded is a new meta robots value announced by Google in January 2022.

The new robots value indexifembedded can be used to indicate that the contents of a resource should be indexed if they are embedded in a webpage, even if there is also a robots noindex present.

Today, we're introducing a new robots tag, indexifembedded, that brings you more control over when your content is indexed. With the indexifembedded tag, you can tell Google you'd still like your content indexed when it's embedded through iframes and similar HTML tags in other pages, even when the content page has the noindex tag.

The indexifembedded tag addresses a common issue that especially affects media publishers: while they may want their content indexed when it's embedded on third-party pages, they don't necessarily want their media pages indexed on their own. Because they don't want the media pages indexed, they currently use a noindex tag in such pages. However, the noindex tag also prevents embedding the content in other pages during indexing.

Example

Example usage using a meta tag:

<meta name="googlebot" value="indexifembedded" />

Code language: HTML, XML

Example usage using a HTTP header:

X-Robots-Tag: googlebot:noindex X-Robots-Tag: googlebot:indexifembedded ... OR ... X-Robots-Tag: googlebot:noindex,indexifembedded

Resources

New robots tag: indexifembedded

Introduction to Sentence-BERT (SBERT)

Colin McDermott — Thu, 14 Apr 2022 22:05:02 +0000

An introduction to Sentence-BERT by Ozan Yılmaz, an NLP engineer from Germany. This article was originally published on Search Candy.

In October 2019 Google announced a new AI-based technology called BERT to further improve their search results.

BERT stands for 'Bidirectional Encoder Representations from Transformers' and is a language representation model which was trained with 3.3M English words.

The huge difference between BERT and earlier versions of language models is that BERT 'understands' the context in which a word is used. To make this a bit clearer, let's look at the following two example sentences.

"Bob is running a marathon."
"Bob is running a company."

It is easy for a human reader to understand that running has a completely different meaning in these examples, but for machines this task is far from trivial.

Early models which were widely used until 2018 would have put both cases of 'running' into the same semantic box of 'walking fast', because it would be the most common usage.

BERT on the other hand looks at the context of words before encoding their meaning into ones and zeros. This makes it a much more accurate and powerful algorithm for encoding natural languages.

For BERT, it is clear that 'running' in example 2) implies that Bob leads a company and has nothing to do with the act of walking.

After understanding what makes BERT so special, it is easy to see the value behind this technology for Google's search algorithms.

Not only does it improve the understanding of words in user inputs, but it also helps Google understand more natural user queries. In 2022, you don't have to explain your grandparents keyword-based search anymore.

With the help of BERT, Google learns to understand whole sequences and their semantic connections.

Nowadays, you can ask a question like 'What is the name of the movie where the little boy sees ghosts?' and get 'The Sixth Sense' as the first result, which is a perfectly fitting, and in this case correct, answer.

Sentence-BERT (SBERT)

As good as it sounds, BERT is still not a solution for every kind of semantic search problem.

Although it is a superb way to encode the meanings of words in a query, it doesn't perform well when it comes to comparing similarities of whole sentences.

Reimers and his colleague Gurevych, the authors of the Sentence-BERT paper, realized this early on and explain the problem with the following statement:

"Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity [...]."

A few modifications and some more fine-tuning for semantic textual similarity help Sentence-BERT cut those 65 hours to about 5 seconds, which is an incredible performance boost.

The difference to regular BERT is the idea to go one abstraction level further and encode the semantic meaning of the whole sentence instead of only encoding the individual words.

The system's core still builds on the standard pre-trained BERT algorithm and derives its semantic power from it.

So, in a scenario where you want to find the most similar headlines to a given one, you are better off utilizing the SBERT algorithm.

At first glance this information might not seem valuable in the scope of SEO, but with new algorithms on the rise like SMITH, which works similar to BERT but looks at longer contexts on document scale, getting a feel for SBERT might be a good thing to do.

Fortunately, I conducted an experiment with SBERT to give you some insights and shed some light on the capacities of BERT-powered architectures.

Using SBERT to find similar asked questions

Given that we now know SBERT is much faster when it comes to comparing sentences, we will use this knowledge to identify questions from a database which are similar to a new question asked by the user.

To do this, I downloaded a huge data dump from reddit's well-known subreddit 'Explain Like I'm Five' (ELI5) using a script by Facebook. On ELI5 users can ask questions about all kinds of things and get layman-friendly answers. I filtered out the questions and encoded them with SBERT.

In the next step I built an algorithm which utilizes the commonly used vector similarity measure 'cosine-similarity' to compute 4 questions from my data dump that are most similar to my input question. To have a Baseline for comparison, I did the same experiment with an encoding strategy called TF-IDF.

TF-IDF encodings reflect how important a word is to a document in a collection or corpus. It is a quite naïve statistical method which operates on word-level and doesn't take semantics into account.

In the following section I will discuss the results for two questions I asked and explain the underlying reasoning process.

Example 1 -- 'Is there racism in animals?'

Resulting similar questions:

TF-IDF	SBERT
Why is there racism?	Do animals express discrimination / racism based on the colour of fur / skin?
Why is there racism?	Do animals get embarrassed or feel shame?
Why is racism still a thing?	Why is incest breeding bad for animals?
Why is racism so common?	Are there murderers, psychopaths or other behavioral deviations commonly associated with human beings among animals?

The top similar question for SBERT gives the user exactly what he asked for, although the wording of the question is different.

Here we can observe the power of SBERT: The sentences are compared using a deeper level of semantics, which helps the system understand the query in a much more advanced way, which is not possible for a simple TF-IDF model.

The naïve system gets very focused on the word 'racism' and provides two top answers containing 'is there racism'. It is obvious that the TF-IDF search mechanism drops the important phrase 'in animals' in order to maximize the similarity between the sequences.

This makes it clear that the baseline representation focuses on a word-level encoding structure and doesn't take the context into account.

Further, we can see that SBERT even understands that racism has a negative connotation. This is the reason why it also lists other question results like 'Do animals get embarrassed or feel ashamed?'.

This behavior is out of scope for a simple statistical model which was fitted on short questions.

Example 2 -- 'Why didn't we already fly to Mars?'

Resulting similar questions:

TF-IDF	SBERT
Could we see "someone" on Mars?	Why haven't we been able to land on Mars?
Why not mars?	Why haven't we put a man on Mars yet?
How high can a fly fly?	Why are we making our expedition to mars a one way trip?
Why does it sometimes cost more to fly from A → B than it does to fly from A → B → C?	Why do we need to go to Mars?

Once again the baseline model concentrates on the simple word-level concepts of 'flying' and 'mars', while SBERT gets the intention of the question and provides the user with fitting results regarding the topic 'travel of mankind to mars'.

It's interesting to observe the interchangeable usage of 'already' and 'yet' between the second SBERT-provided question and the user query. This again shows the semantic power of the BERT architecture.

As we can see, SBERT has the capacity to encode even granular semantic information on the sentence level and provides meaningful results when used for sentence similarity tasks.

Given the fact that algorithms like SMITH are currently being developed, it could be possible that a sentence-level language model like SBERT could be implemented in Google's algorithm as well. But what does this power of BERT and BERT-like technologies mean for SEO?

Cutting Edge Language Technologies and SEO

Natural Language Processing made a huge forward leap with the introduction of BERT and there is no sign of slowing down.

New papers are being released on a regular basis and the field is getting pushed more than ever by companies like Google, Facebook and Amazon.

With this in mind, the challenge of SEO will change from caring about backlinks, keyword optimization, meta descriptions, etc., to actually just generating quality content for users.

People want to find precise and compact information when they search for something, and you need to be able to deliver exactly that.

The times where we optimized for machines slowly comes to an end. In the future, machines will optimize for us.

Sources

What is a 308 Permanent Redirect?

Colin McDermott — Sun, 10 Apr 2022 21:36:12 +0000

What is a 308 Permanent Redirect HTTP response?

A 308 Permanent Redirect is a server HTTP response status code used for permanent redirection.

A 308 HTTP response is near identical to a 301, and has been confirmed to be treated as such by Google.

The only notable difference between a 301 and 308 HTTP response is that a 308 does not allow changing the request method from POST to GET.

Example 308 Permanent Redirect Response

HTTP/1.1 308 Permanent Redirect Content-Type: text/html; charset=UTF-8 Location: https://www.example.com/ Content-Length: 356

<!DOCTYPE HTML> <html> <head> <title>Permanent Redirect</title> </head> <body> <p> The document has been moved to <a href="https://www.example.com/">https://www.example.com</a> </p> </body> </html>

RFC

RFC7538, April 2015

Is a 308 response good for SEO?

A 308 HTTP response has been confirmed by Google to be treated as identical to a 301 by its bots/algorithms.

However a 308 response is still a newer and more unusual status code, so it is unknown how all search engines/bots around the world will treat it.

Non-Standard Usage of 308 Status Code

It should be noted that some sites may use a 308 status code in a non-standard way, such as in Google Drive - where a 308 Resume Incomplete status is used to indicate when an incomplete upload has stalled.

Ecommerce Navigation SEO Best Practice Guide

Colin McDermott — Thu, 02 Dec 2021 13:25:13 +0000

SEO is one of the most important channels in ecommerce, so it is vitally important you make your ecommerce site friendly to Google and other search engines if you want to reap some of that sweet organic traffic.

Essential reading: Google recently published additional guidance for ecommerce site owners relating to navigation which can be found here.

Why is navigation important in ecommerce?

The navigation menus and process of an ecommerce store can be super important: allowing both users and search engines to understand the structure/hierarchy of your store, enabling them to find their way around the different categories, sections and product offerings.

For this reason it is important to spend a solid amount of time planning out navigation systems across the site, and of course a strong focus the primary navigation menu to ensure it effectively links to all the key areas of the store. Also to conduct testing to make sure that your navigation systems are functionable, accessible, and easy to use for both customers and search engine bots.

A poorly designed, slow, or confusing navigation system can lead to both lower sales as well as worse indexing & performance in search engines.

Search Engine Friendly Ecommerce Navigation

From an SEO perspective there are a few things you want to avoid if you want bots to be able to easily use your navigation.

Most of the below tips are focused around the primary navigation menu. However we also drill down into a wide selection of specific navigation systems, providing advice and SEO considerations for each - as well as taking a look at all important mobile menus and Core Web Vitals.

Text Based Links

Navigation links should ideally always be text based, as opposed to using images, animations, or anything else too fancy that a search engine would struggle to read.

So for example your design team may think it looks creative and cool to link to your toys category using only an animation of a spinning yo-yo - but to simply use the word "Toys" would be far more descriptive.

Use Actual Links

Google finds it easiest to understand actual HTML based links, using the <a> tag.

Avoid using JavaScript based 'onclick' type links, and especially avoid using <buttons> as part of your navigation as these are designed for actions, not navigational links - and are not read by Google as links.

Content Hierarchy & Internal Linking

Google looks at how many links different pages get and uses this information to assess which content is most important on your site.

So it is a good idea to ensure the menu links to the most important areas of your site, and avoid pointing too many internal links towards areas which cannot be accessed or indexed by Google.

So for example linking to your privacy policy as the first link in the navigation on every page might not be sensible, compared to linking to your primary ecommerce category.

Similarly, as pleasing or easy to program as an alphabetical list may be, it may be more effective to order lists by popularity or choose the order manually.

It is important to think about your most important keywords and products, then to make sure they are linked to with sufficient priority.

Similarly think about sub-categories and any important areas. Can Google and users find their way to these pages quickly using the onsite navigation?

Overly Complex Navigation

Google recommends avoiding overly complicated navigation which forces users to click dozens of times to get to a specific area or product, and going overboard with segmenting your content into too many niche categories to the point where navigation becomes more difficult.

Avoid:

Creating complex webs of navigation links, for example, linking every page on your site to every other page.

Going overboard with slicing and dicing your content (so that it takes twenty clicks to reach from the homepage). Source: Google SEO Starter Guide

Core Web Vitals & Ecom Navigation

Following Google's page experience update - whereby fast and user-friendly sites are now eligible to get a boost in the search results - ensuring your navigation is fast loading and Core Web Vitals friendly is very important for SEO.

Cumulative Layout Shift (CLS)

Navigation menus that are slow to load (even by just a few split seconds) can be the cause of layout shifts that cause a bad experience for users.

Navigation menu causing CLS on a web page - From left to right

As ecommerce site navigation often appears at the very top of a page, if the navigation bar or an important element inside it (such as an image, search box or icon) is slow to load - then consequently 'pops' into place - it is possible it will push all the other content on the page below it downwards creating an unpleasant layout shift.

This could be caused by slow loading resources needed by the navigation, and/or by not properly instructing the browser how much space will be required by the navigation when it does eventually load.

First Input Delay (FID)

First Input Delay measures the delay from when a user attempts to interact with a webpage and the interaction starting.

For example if a user were to click a link in a navigation menu, FID measures the delay in how long before the browser begins the process of fulfilling the request to load the new page.

A delay may be caused by resources/elements on the page loading, or the 'main thread' being busy doing other work.

As well as affecting SEO, the faster users are able to interact with navigational elements the faster they will be able to browse products - which is certain to directly impact sales.

Spend a suitable amount of time testing, monitoring, and limiting the impact the navigation system has on your site's Core Web Vitals performance.

The below header example from British ecommerce site Next.co.uk includes a navigation menu and onsite search box, as well as links to key pages such as the account login, checkout, and a language selector.

See video at: https://www.searchcandy.uk/wp-content/uploads/2021/11/next-menu.mp4

Faceted Navigation & Product Filters

Faceted navigation is one of the most complex and difficult areas of ecommerce SEO to get right.

Whether to allow search engines to crawl and index all your product category filters and variations, and how to go about doing it, is a big decision for ecommerce sites - potentially opening up a site to hundreds of thousands, sometimes even millions of extra URLs/parameters for Google to assess.

Some product filters may be worth indexing - such as the first level of a colour filter - but many other filters may not.

There is no simple answer to recommend here and Google provide relatively scant resources to assist webmasters in this area, but we can point you towards this document published by Google which lists 5 of the best and worst practices when it comes to faceted navigation - and also this ecommerce URL structure best practice guide.

Breadcrumbs

Breadcrumbs are an essential navigational element for any website or ecommerce store.

Many ecommerce breadcrumb implementations fall down by either not existing at all, or by using a flat structure that misses out categories and sub-categories.

Home > Example Product

If your ecommerce breadcrumbs look like the above, they need work!

Making sure to use the correct breadcrumbs markup is important, with a good implementation potentially being rewarded with breadcrumbs direct in Google search results.

Breadcrumbs can be used on both product category pages and product pages.

Google recommend using breadcrumbs that reflect a typical user path to a particularly product or URL, rather than simply mirroring the URL structure.

So for example a bad breadcrumb structure (with multiple superfluous levels) might be:

Home > Shop > Products > ID > 343423 > Example Product

Mega Menus

Mega menus can be a great way of laying out your product categories and site architecture for users.

Try to keep to the general principles laid out above, avoiding: overly complex navigation; linking to an excessive number of resources; and considering Core Web Vitals or usability issues.

One of the primary SEO concerns with mega menus is undoubtedly the high risk of having an excessive number of linked resources.

For example a menu that has 10 primary categories, each with on average 50 sub-categories/links, would have 500 total links just inside the primary navigation - something which would recur on every single ecommerce page on the website.

Mega menus can also often be large and unwieldy, being almost always both powered by and heavily reliant on JavaScript resources which may be slow to load and render in the browser.

Many sites also rely too heavily on mega menus for navigation. Consider the following test: could your users find their way to a specific category or product without using the mega menu? If the answer is no - it is highly likely the site has improvements to be made in terms of navigation and internal linking.

Onsite Search

There are multiple ways potential customers may attempt to find their way around a website, and search based navigation is a method users may choose to utilise - especially if they feel like they are unable to quickly find their way to a resource using the existing navigational UI.

An effective onsite search engine can allow users to quickly and easily find their way to appropriate products or categories.

The Next search box functionality provides auto-complete suggestions before the user hits enter/taps to search - potentially reducing the user journey by one click or more. See the screenshot below for an example of this.

Another potentially good strategy is redirecting searches to specific products, ranges or category pages in some situations - which can save a step in the user journey if done effectively and without any frustrating errors.

For example on a grocery ecommerce store a search for [vegan] could redirect users directly to the primary vegan range/category, rather than simply listing all search results that include the word 'vegan'.

Blocking Search Results from Search Engines
From an SEO perspective it is important to ensure that onsite search results pages are blocked from indexing by offsite search engines, such as Google and Bing.

A robots.txt disallow rule is not strong enough here. A noindex based solution is going to be ideal for most situations - to fully prevent Google from indexing the results.

Why is it important to block Google/search engines from search results pages? Allowing search results pages to get indexed can create an attack vector for hackers/spammers, as well as potentially causing SEO issues.

Mobile menus & navigation

With the majority of visitors to ecommerce sites now being on mobiles for many sites and industries, and Google having moved firmly towards mobile first indexing, getting your mobile menus right is vitally important for both users and SEO.

There is a huge amount to consider when building and optimising a mobile navigation and menu system. It is worth considering that mobile navigation/UI includes more than just a standard mobile menu, especially on ecommerce sites.

For example let's take a look at the UK fashion etailer Missguided.co.uk. Note before we even look at what most would consider standard mobile menu accessed via the hamburger icon, we can already navigate using swipeable sub-menu of trending/featured categories, breadcrumbs, links to important sub-categories with images, and a product filtering system.

Having the mobile menu in a swipeable format in theory could save potential customers a click when accessing popular resources. It also means the links are in plain sight for both visitors and search engines.

HTML Sitemap

Internal linking is important for SEO, and a good HTML sitemap can potentially help with internal linking across an ecommerce site.

Static sitemaps can also be useful for users who by last resort have been forced to use a sitemap to find their way around a large website.

Tip: Neither users nor search engines should NEED to rely on HTML sitemaps to easily navigate a site. There should ideally already be an effective and intuitive to navigate UI, that doesn't require users to have to find what they need by browsing a large and cumbersome HTML sitemap.

Sitemaps can be designed to cover an entire site, or just specific sections or page types. For example the Asos.com site features HTML sitemaps listing brands (see below).

What to do if your Development Site gets Indexed in Google

Colin McDermott — Wed, 10 Nov 2021 23:08:41 +0000

We take a look at what to do if your development/staging/testing site has been indexed in Google, including expert advice direct from a Google employee.

Oops! Our Dev site has been indexed in Google

First of all, don't beat yourself up: a development site getting accidentally indexed in Google has happened at least once to every agency, developer, and inhouse team on the planet.

Perhaps you found out about it via an angry email from your client or their SEO consultant, or maybe you discovered it yourself whilst checking for indexed pages.

How did the dev site get indexed?

If Google is able to find its way to your development site unimpeded, and finds there no instruction not to crawl or index the URLs available to it, there is a high chance Google will store the pages in their search engine for users to find.

This can happen reasonably quickly - even if your dev site was only available for a few days or weeks - this could be enough time for Google to index the entire site.

So if your development site does get indexed in Google, what can you do about it, and are there any urgent solutions for situations where for example the client or management is upset?

How to quickly remove a dev site from the Google index

As recommended by Google's John Mu, if you find your staging site has been indexed and there is an urgent requirement to remove it, the quickest way to remove content from Google is to use the official 'Removals Tool' found in Search Console.

I'd do a site-removal request in search console - if the site is verified, it'll be hidden in search within less than a day. After that, you have time to figure out what to do for the long run.

John Mu via TechSEO subreddit

Official video: Removals in Search Console - Daniel Waisberg

To use the Removals tool you will first need to verify the specific domain you want to remove in Search Console if it is already verified.

John goes on to offer some footnotes and warnings regarding use of the tool:

If you make a mistake and need to cancel a removal request, this process should be fast.
Remember that removals apply to both www and non-www, and both http/https.
Using the tool properly should clear the URL from Google for around 6 months.

After temporarily removing URLs from Google, it is sensible to then work towards a permanent removal.

How to permanently remove a development or staging site from Google

The most effective way to request Google no longer indexes a page is to either use a noindex command, or ensure the resource responds with a 410/404 HTTP response to indicate it is no longer available.

Google have stated in the past that a noindex tag and 404/410 should work at the same speed.

If Google returns to a resource following a temporary removal request and finds a 404/410 or noindex tag they will cancel the removal request as it is no longer needed.

You could also set up authentication which would result in Google being unable to access a resource (eg with a 401 HTTP response).

Using a robots.txt block is not a good solution if your site has already been indexed. It can take a long time to have any impact and is not a direct instruction to remove content from the index, so Google can ignore it and leave the page indexed if they wish.

If your site is already indexed in Google, using a robots.txt rule to prevent Google crawling the site will also prevent them from seeing a noindex tag/header if you add one to a page.

How to remove a development site from Google's cache

Using the Removals Tool in Search Console will by default remove the URLs entirely, including the cache.

When using the tool you are given the option to remove the cached URL - which will clear the snippet shown in search results - until the resource is recrawled and a new snippet will be shown.

How to prevent a development site from getting indexed by Google

To prevent your dev site getting indexed in Google there are a variety of methods you can use:

Authentication (password, IP address, CMS/plugin based, etc)
Noindex tag or header
Robots.txt disallow rule (least recommended option) Google's John Mu recommends the use of server side authentication as the best method:

My recommendation is always to use server-side authentication for staging / dev sites, since it's obvious when it's blocked, and obvious when it's forgotten. Robots.txt and robots meta tags are easy to accidentally deploy to your live site.

Note: Robots.txt is not a good option because it can be ignored by Google and other search engines.

How to stop Google indexing a WordPress development site

You can use any of the standard methods to stop Google indexing a WordPress staging site - eg password protection, noindex or blocking Google from crawling the site with robots.txt.

The easiest method if you have access to the WordPress admin dashboard is to set WordPress to enable the 'Discourage search engines' option via Settings > Reading. This method should add a noindex tag to all your pages.

DEV Community: Colin McDermott

Telegram Channel Developer Cheatsheet

Setting Up a Telegram Channel

Managing Your Channel

Enhancements

Safety & Privacy

Promoting Your Channel

Charging for access to your Telegram channel

Tools & Resources

Useful Commands

REST API Key Concepts and Requirements

Stateless

Client-Server Architecture

Cacheability

Layered System

Uniform Interface

Code on Demand (optional)

Idempotency

Resource Nesting

Filtering, Sorting, and Searching

API Rate Limiting Cheat Sheet

Gateway-level rate limiting

Token bucket algorithm

Leaky bucket algorithm

Sliding window algorithm

Distributed rate limiting

User-based rate limiting

API key rate limiting

Custom rate limiting

RESTful API Design Cheatsheet

HTTP request methods

Naming Conventions

Resource Pagination

Error Responses

Authentication and Authorization

Versioning

Caching

HATEOAS

Filtering and Sorting

Rate Limiting

Content Negotiation

API Documentation

CORS (Cross-Origin Resource Sharing)

API Monitoring and Testing

The 9 Key Questions for Google's Helpful Content Update

Introduction to Amazon Advertising

Amazon Advertising: Types of Ads

Sponsored results on Amazon.co.uk

Tips for successfully setting up Amazon ads

KPIs to focus on while advertising on Amazon

Amazon Ads vs. Google Ads

How do you know if your ecommerce site is hit by a Google Core Update?

What are Core Updates?

How do you know if your ecommerce website has been hit by a Core Update?

What are the symptoms of being hit by a Core Update?

What to do about it?

What is Indexifembedded?

What is Indexifembedded?

Example

Resources

Introduction to Sentence-BERT (SBERT)

Sentence-BERT (SBERT)

Using SBERT to find similar asked questions

Example 1 -- 'Is there racism in animals?'

Example 2 -- 'Why didn't we already fly to Mars?'

Cutting Edge Language Technologies and SEO

Sources

What is a 308 Permanent Redirect?

What is a 308 Permanent Redirect HTTP response?

Example 308 Permanent Redirect Response

RFC

Is a 308 response good for SEO?

Non-Standard Usage of 308 Status Code

Ecommerce Navigation SEO Best Practice Guide

Why is navigation important in ecommerce?

Search Engine Friendly Ecommerce Navigation

Text Based Links

Use Actual Links

Content Hierarchy & Internal Linking

Overly Complex Navigation