DEV Community: Aaron Goldsmith

Twitch.tv API - Get live stream data from paginated results

Aaron Goldsmith — Mon, 29 Mar 2021 04:23:02 +0000

Recently, I wanted to work with the Twitch API to try to recreate the website twitchroulette.net, where you would be able to view a completely random live stream from all of the streams currently happening on the site. According to analytics from twitchtracker.com, there are currently an average of over 100,000 Twitch live streams at any given time.

When I went through the Twitch API documentation, I discovered that for the endpoint https://api.twitch.tv/helix/streams to get live streams, Twitch limits the response to a maximum of 100 streams per API call. However, the response includes a pagination field which contains a cursor value (a string) which is used in a subsequent requests to specify the starting point of the next set of results.

The response body for the GET request at https://api.twitch.tv/helix/streams?first=100 would include the top 100 most active live streams, and the data looks like this:

{
  "data": [
    {
      "id": "41375541868",
      "user_id": "459331509",
      "user_login": "auronplay",
      "user_name": "auronplay",
      "game_id": "494131",
      "game_name": "Little Nightmares",
      "type": "live",
      "title": "hablamos y le damos a Little Nightmares 1",
      "viewer_count": 78365,
      "started_at": "2021-03-10T15:04:21Z",
      "language": "es",
      "thumbnail_url": "https://static-cdn.jtvnw.net/previews-ttv/live_user_auronplay-{width}x{height}.jpg",
      "tag_ids": [
        "d4bb9c58-2141-4881-bcdc-3fe0505457d1"
      ]
    },
    ...
  ],
  "pagination": {
    "cursor": "eyJiIjp7IkN1cnNvciI6ImV5SnpJam8zT0RNMk5TNDBORFF4TlRjMU1UY3hOU3dpWkNJNlptRnNjMlVzSW5RaU9uUnlkV1Y5In0sImEiOnsiQ3Vyc29yIjoiZXlKeklqb3hOVGs0TkM0MU56RXhNekExTVRZNU1ESXNJbVFpT21aaGJITmxMQ0owSWpwMGNuVmxmUT09In19"
  }
}

If you wanted to retrieve the next 100 most active live streams, the subsequent API request URL would need to be:

https://api.twitch.tv/helix/streams?first=100&after=eyJiIjp7IkN1cnNvciI6ImV5SnpJam8zT0RNMk5TNDBORFF4TlRjMU1UY3hOU3dpWkNJNlptRnNjMlVzSW5RaU9uUnlkV1Y5In0sImEiOnsiQ3Vyc29yIjoiZXlKeklqb3hOVGs0TkM0MU56RXhNekExTVRZNU1ESXNJbVFpT21aaGJITmxMQ0owSWpwMGNuVmxmUT09In19

This includes as its after value the cursor value returned in the prior response.

It's not possible to sort the responses by least active, so in order to get results with streams with very few or no viewers, you would need results for the more active streams first.

It is also important to note that the Twitch API is rate-limited to 800 requests per minute, so the maximum number of livestreams we could retrieve in that time is 80,000, which is substantially lower than the current weekly average. It's therefore plausible that trying to get a truly complete list of results for live streams would run the risk of causing a HTTP 429 error (too many requests).

In order to try to retrieve as many live streams as possible, while keeping in mind the constraints of the rate-limit and a potentially impatient user, I approached this problem using recursion:

function getAllStreams (cursor, data = [], counter = 15) {
  while (counter !== 0) {
    const request = new Request('https://api.twitch.tv/helix/streams?first=100' + (cursor ? '&after=' + cursor : ''), { 
      method: 'GET' ,
      headers: {
        'Client-ID': clientId,
        'Authorization': `Bearer ${access_token}`,
        'Content-Type' : 'application/x-www-form-urlencoded; charset=UTF-8'
        }
      });

    return fetch(request).then((response) => response.json()).then((responseJson) => { 
      if (counter === 1) return data;
      data.push(...responseJson.data);
      return getAllStreams(responseJson.pagination.cursor, data, --counter);
    });
  }
}

I found that each request took about half a second to complete, so that meant I also needed to limit the number of requests made in order to keep the user engaged, and I specify that limit as a default argument counter. While 1500 streams might not seem like a big number, it does make it possible to recreate the experience of viewing a single random stream.

I would appreciate any suggestions or critiques of my approach, as this was the first time I've worked with and tried to 'crawl' a paginated API. I just wanted to share the way I went about using this endpoint in order to try to help other developers who attempt to do the same.

Thanks for reading!

Logical Assignment Operators

Aaron Goldsmith — Sat, 09 Jan 2021 05:58:54 +0000

Javascript (ES2021) now supports the ability to combine logical operations and assignment with the new operators &&=, ||=, and ??=. Previously, compound assignment operators were only possible with mathematical and bitwise operations.

Logical AND assignment `&&=`

The logical AND assignment x &&= y operator performs assignment only if x is truthy. For example:

let a = 1;
let b = 0;

a &&= 2;
console.log(a); // output: 2

b &&= 2;
console.log(b); // output: 0

Short-circuit evaluation

If the left-hand expression is falsy, it is short-circuit evaluated and the right-hand expression is not evaluated, so the value of b is not reassigned because 0 is falsy.

a &&= 2 is equivalent to a && (a = 2), and the value of a is reassigned because it was originally truthy.

Logical OR assignment `||=`

The logical OR assignment x ||= y operator only assigns if x is falsy.

const a = 50;
const b = '';

a ||= 10;
console.log(a);  // output: 50

b ||= 'string is empty.';
console.log(b); // output: "string is empty."

Short-circuit evaluation

From the above example, a ||= 10 is equivalent to a || (a = 10), but because the value of a is truthy, it is short-circuit evaluated and the right-hand assignment is not evaluated.

If you are checking against a falsy value such as an empty string or 0, ||= must be used, otherwise the logical nullish assignment operator ??= should be used to check for null or undefined values.

Logical nullish assignment `??=`

The logical nullish assignment x ??= y operator only assigns if x is null or undefined.

const a = { limt: 50 };

a.limit ??= 10;
console.log(a.limit); // output: 50

a.speed ??= 25;
console.log(a.speed); // output: 25

Short-circuit evaluation

A logical nullish assignment expression is short-circuit evaluated to the left-hand side expression if the left-hand side is neither null nor undefined.

In the above example, the reassignment of a.limit is not performed because its value is defined as 50, which is not null or undefined.

a.speed would initially evaluate to undefined, so the right-hand expression is evaluated, which is equivalent to a.speed = 25, and now a = {limit: 50, speed: 25}.

Nullish Coalescing Operator

Aaron Goldsmith — Fri, 08 Jan 2021 04:33:46 +0000

The nullish coalescing operator ?? is a logical operator that returns its right-hand side operand when its left-hand side operand is null or undefined, and otherwise returns its left-hand side operand.

const nullValue = null;

const val1 = nullValue ?? "default";

console.log(val1); // "default"

It is similar to the traditional 'or' logical operator ||, but the difference is that || will return the right-hand side operand with any falsy value, such as an empty string, 0 or NaN, not only null or undefined.

For example:

const val = null ?? 'default';
console.log(val);
// logs "default"

const val2 = 0 ?? 88;
console.log(val2);
// logs 0 because it is falsy

It is possible to use the nullish coalescing operator in combination with the traditional AND && and OR || operators, but the statements that include those logical operators must be enclosed in parentheses, or it will raise a syntax error. For example:

null || undefined ?? "value"; // SyntaxError
false || undefined ?? "value"; // SyntaxError

(null || undefined) ?? "value"; // returns "value"

A common pattern is to use the nullish coalescing operator with the optional chaining operator ?., which is useful to access a property of an object which may be null or undefined.

const obj = { prop1: "hello" };

console.log(obj.prop1?.toUpperCase() ?? "error"); // "HELLO"
console.log(obj.prop2?.toUpperCase() ?? "error"); // "error"

Optional Chaining Operator

Aaron Goldsmith — Thu, 07 Jan 2021 07:49:29 +0000

The optional chaining operator ?. is a new Javascript language feature introduced in ES2020. It provides a way to access nested object properties without having to expressly validate that each reference in the chain is valid.

The ?. operator functions similarly to the . chaining operator, except that instead of causing an error if a reference is null or undefined, the expression returns a value of undefined. When used with function calls, it returns undefined if the given function does not exist.

For example:

const obj = {
  prop1: {
    prop2: {
      prop3: "value"
    }
  }
};

If you attempted access a property in the above object that didn't exist: obj.prop2.prop3 that would throw the following error:

Uncaught TypeError: Cannot read property 'prop3' of undefined

Typically, you could try to handle the case of potentially missing property by combining logical operators and a ternary expression:

const myProp3 = obj.prop2 && obj.prop2.prop3 ? obj.prop2.prop3 : undefined;

This verbose syntax can be written instead using the optional chaining operator which will automatically assign the value of myProp3 to undefined:

const myProp3 = obj.prop2?.prop3; //undefined

For dealing with even more deeply nested objects, the optional chaining operator can be used multiple times in the same statement to safely accommodate the accessing of properties:

let temp = obj.first?.second?.third?.value;

It can also be used with function calls to make sure a function exists, and if it doesn't will return undefined:

let data = obj.someMethod?.(someArgument); //undefined

It can also be used with arrays:

const arr = [5,6,7,8];

arr?.[10] // undefined

With the Map data structure:

let myMap = new Map();
myMap.set("foo", {name: "baz", desc: "inga"});

let nameBar = myMap.get("bar")?.name;

And finally, with the nullish coalescing operator ??:

const myCar = {
  make: "Ford",
  details: { year: 1982 }
};
const customerCar = myCar?.make ?? "Unknown Car";
console.log(customerCity); // Unknown Car

DEV Community: Aaron Goldsmith

Twitch.tv API - Get live stream data from paginated results

Logical Assignment Operators

Logical AND assignment &&=

Short-circuit evaluation

Logical OR assignment ||=

Short-circuit evaluation

Logical nullish assignment ??=

Short-circuit evaluation

Nullish Coalescing Operator

Optional Chaining Operator

Logical AND assignment `&&=`

Logical OR assignment `||=`

Logical nullish assignment `??=`