DEV Community: Oleksii Trekhleb

I built a small app called CaliVibe to explore California counties and cities through data

Oleksii Trekhleb — Fri, 13 Mar 2026 20:34:09 +0000

Sometimes it is easier to understand a place through data rather than descriptions.

So I built a small interactive app that lets you explore California counties and cities using different metrics like:

population
crime rates
temperature trends
topographical landscape
and other geographic data

You can try it here:

👉 https://trekhleb.dev/cali-vibe/

The project is also open-sourced:

👉 https://github.com/trekhleb/cali-vibe

Demo (.gif)

What you can explore

The app lets you quickly switch between different datasets and see how regions compare.

For example you can explore:

Population distribution
Crime statistics
Temperature trends
Terrain elevation

Each dataset can reveal interesting patterns across California counties and cities.

Example views

Terrain view

Temperature trends

Crime rates per city/county

County population

County/city borders

Tech stack

The app is intentionally simple and fully client-side.

Frontend

React
TypeScript
Vite
Tailwind CSS

Visualization

MapLibre GL
Three.js

Data

JSON datasets loaded as static assets.

This means the app runs without a backend.

Data sources

Some of the data used in the project comes from:

CA DOJ OpenJustice
California Department of Finance
US Census Bureau
Open-Meteo climate datasets
Terrain tiles from AWS

The processed datasets are included in the repository.

Development experiment

Another interesting aspect of this project is that it was mostly AI-assisted development.

I vibe-coded for fun: ~98% of the code was written by Claude Code Opus with a few sprinkles from Gemini 3.1 Pro.

This made it possible to move from idea → working prototype very quickly.

Why I built this

I enjoy building small interactive tools that help explore data visually.

Sometimes a map + a few sliders can reveal patterns much faster than tables or reports.

This project was mostly a weekend experiment, but it turned into a fun way to explore California through data.

Source code

If you're curious how it works internally:

👉 https://github.com/trekhleb/cali-vibe

Feedback, ideas, and improvements are welcome.

Tech Jobs Radar

Oleksii Trekhleb — Fri, 10 Jan 2025 16:02:52 +0000

I was curious about the geographical distribution of top tech jobs—for example, how many FAANG jobs are in Europe and where they are primarily concentrated. To explore this, I created a draft (prototype) of Tech Jobs Radar: https://jobs-radar.com

The platform visualizes a snapshot of open top tech positions on a map (currently based on December 2024 data). To make comparisons easier, it aggregates jobs into clusters depending on the zoom level, allowing you to analyze data across continents, countries, states, or counties.

It also includes company and country filters to help you explore specific slices of the data—for instance, you can check open positions at Uber or identify the main engineering hubs for a particular company.

Additionally, you can search for jobs by title. Each listing includes a link to the detailed description on Indeed.

I’m sharing it here in case others are also curious about analyzing this kind of data.

An exploration of how X's home timeline API is designed

Oleksii Trekhleb — Thu, 12 Dec 2024 19:56:20 +0000

When it comes to designing the API of the system the software engineers often consider different options like REST vs RPC vs GraphQL (or other hybrid approaches) to determine the best fit for a specific task or project.

In this article, we explore how the X (Twitter) home timeline (x.com/home) API is designed and what approaches they use to solve the following challenges:

How to fetch the list of tweets
How to do a sorting and pagination
How to return the hierarchical/linked entities (tweets, users, media)
How to get tweet details
How to "like" a tweet

We will only explore these challenges on the API level, treating the backend implementation as a black box, since we don't have access to the backend code itself.

Showing the exact requests and responses here might be cumbersome and hard to follow since the deeply nested and repetitive objects are hard to read. To make it easier to see the request/response payload structure, I've made my attempt to "type out" the home timeline API in TypeScript. So when it comes to the request/response examples I'll use the request and response types instead of actual JSON objects. Also, remember that the types are simplified and many properties are omitted for brevity.

You may find all types in types/x.ts file or at the bottom of this article in the "Appendix: All types in one place" section.

Fetching the list of tweets

The endpoint and request/response structure

Fetching the list of tweets for the home timeline starts with the POST request to the following endpoint:

POST https://x.com/i/api/graphql/{query-id}/HomeTimeline

Here is a simplified request body type:

type TimelineRequest = {
  queryId: string; // 's6ERr1UxkxxBx4YundNsXw'
  variables: {
    count: number; // 20
    cursor?: string; // 'DAAACgGBGedb3Vx__9sKAAIZ5g4QENc99AcAAwAAIAIAAA'
    seenTweetIds: string[]; // ['1867041249938530657', '1867041249938530659']
  };
  features: Features;
};

type Features = {
  articles_preview_enabled: boolean;
  view_counts_everywhere_api_enabled: boolean;
  // ...
}

And here is a simplified response body type (we'll dive deeper into the response sub-types below):

type TimelineResponse = {
  data: {
    home: {
      home_timeline_urt: {
        instructions: (TimelineAddEntries | TimelineTerminateTimeline)[];
        responseObjects: {
          feedbackActions: TimelineAction[];
        };
      };
    };
  };
};

type TimelineAddEntries = {
  type: 'TimelineAddEntries';
  entries: (TimelineItem | TimelineCursor | TimelineModule)[];
};

type TimelineItem = {
  entryId: string; // 'tweet-1867041249938530657'
  sortIndex: string; // '1866561576636152411'
  content: {
    __typename: 'TimelineTimelineItem';
    itemContent: TimelineTweet;
    feedbackInfo: {
      feedbackKeys: ActionKey[]; // ['-1378668161']
    };
  };
};

type TimelineTweet = {
  __typename: 'TimelineTweet';
  tweet_results: {
    result: Tweet;
  };
};

type TimelineCursor = {
  entryId: string; // 'cursor-top-1867041249938530657'
  sortIndex: string; // '1866961576813152212'
  content: {
    __typename: 'TimelineTimelineCursor';
    value: string; // 'DACBCgABGedb4VyaJwuKbIIZ40cX3dYwGgaAAwAEAEEAA'
    cursorType: 'Top' | 'Bottom';
  };
};

type ActionKey = string;

It is interesting to note here, that "getting" the data is done via "POSTing", which is not common for the REST-like API but it is common for a GraphQL-like API. Also, the graphql part of the URL indicates that X is using the GraphQL flavor for their API.

I'm using the word "flavor" here because the request body itself doesn't look like a pure GraphQL query, where we may describe the required response structure, listing all the properties we want to fetch:

# An example of a pure GraphQL request structure that is *not* being used in the X API.
{
  tweets {
    id
    description
    created_at
    medias {
      kind
      url
      # ...
    }
    author {
      id
      name
      # ...
    }
    # ...
  }
}

The assumption here is that the home timeline API is not a pure GraphQL API, but is a mix of several approaches. Passing the parameters in a POST request like this seems closer to the "functional" RPC call. But at the same time, it seems like the GraphQL features might be used somewhere on the backend behind the HomeTimeline endpoint handler/controller. A mix like this might also be caused by a legacy code or some sort of ongoing migration. But again, these are just my speculations.

You may also notice that the same TimelineRequest.queryId is used in the API URL as well as in the API request body. This queryId is most probably generated on the backend, then it gets embedded in the main.js bundle, and then it is used when fetching the data from the backend. It is hard for me to understand how this queryId is used exactly since X's backend is a black box in our case. But, again, the speculation here might be that, it might be needed for some sort of performance optimization (re-using some pre-computed query results?), caching (Apollo related?), debugging (join logs by queryId?), or tracking/tracing purposes.

It is also interesting to note, that the TimelineResponse contains not a list of tweets, but rather a list of instructions, like "add a tweet to the timeline" (see the TimelineAddEntries type), or "terminate the timeline" (see the TimelineTerminateTimeline type).

The TimelineAddEntries instruction itself may also contain different types of entities:

Tweets — see the TimelineItem type
Cursors — see the TimelineCursor type
Conversations/comments/threads — see the TimelineModule type

type TimelineResponse = {
  data: {
    home: {
      home_timeline_urt: {
        instructions: (TimelineAddEntries | TimelineTerminateTimeline)[]; // <-- Here
        // ...
      };
    };
  };
};

type TimelineAddEntries = {
  type: 'TimelineAddEntries';
  entries: (TimelineItem | TimelineCursor | TimelineModule)[]; // <-- Here
};

This is interesting from the extendability point of view since it allows a wider variety of what can be rendered in the home timeline without tweaking the API too much.

Pagination

The TimelineRequest.variables.count property sets how many tweets we want to fetch at once (per page). The default is 20. However, more than 20 tweets can be returned in the TimelineAddEntries.entries array. For example, the array might contain 37 entries for the first page load, because it includes tweets (29), pinned tweets (1), promoted tweets (5), and pagination cursors (2). I'm not sure why there are 29 regular tweets with the requested count of 20 though.

The TimelineRequest.variables.cursor is responsible for the cursor-based pagination.

"Cursor pagination is most often used for real-time data due to the frequency new records are added and because when reading data you often see the latest results first. It eliminates the possibility of skipping items and displaying the same item more than once. In cursor-based pagination, a constant pointer (or cursor) is used to keep track of where in the data set the next items should be fetched from." See the Offset pagination vs Cursor pagination thread for the context.

When fetching the list of tweets for the first time the TimelineRequest.variables.cursor is empty, since we want to fetch the top tweets from the default (most probably pre-computed) list of personalized tweets.

However, in the response, along with the tweet data, the backend also returns the cursor entries. Here is the response type hierarchy: TimelineResponse → TimelineAddEntries → TimelineCursor:

type TimelineResponse = {
  data: {
    home: {
      home_timeline_urt: {
        instructions: (TimelineAddEntries | TimelineTerminateTimeline)[]; // <-- Here
        // ...
      };
    };
  };
};

type TimelineAddEntries = {
  type: 'TimelineAddEntries';
  entries: (TimelineItem | TimelineCursor | TimelineModule)[]; // <-- Here (tweets + cursors)
};

type TimelineCursor = {
  entryId: string;
  sortIndex: string;
  content: {
    __typename: 'TimelineTimelineCursor';
    value: string; // 'DACBCgABGedb4VyaJwuKbIIZ40cX3dYwGgaAAwAEAEEAA' <-- Here
    cursorType: 'Top' | 'Bottom';
  };
};

Every page contains the list of tweets along with "top" and "bottom" cursors:

After the page data is loaded, we can go from the current page in both directions and fetch either the "previous/older" tweets using the "bottom" cursor or the "next/newer" tweets using the "top" cursor. My assumption is that fetching the "next" tweets using the "top" cursor happens in two cases: when the new tweets were added while the user is still reading the current page, or when the user starts scrolling the feed upwards (and there are no cached entries or if the previous entries were deleted for the performance reasons).

The X's cursor itself might look like this: DAABCgABGemI6Mk__9sKAAIZ6MSYG9fQGwgAAwAAAAIAAA. In some API designs, the cursor may be a Base64 encoded string that contains the id of the last entry in the list, or the timestamp of the last seen entry. For example: eyJpZCI6ICIxMjM0NTY3ODkwIn0= --> {"id": "1234567890"}, and then, this data is used to query the database accordingly. In the case of X API, it looks like the cursor is being Base64 decoded into some custom binary sequence that might require some further decoding to get any meaning out of it (i.e. via the Protobuf message definitions). Since we don't know if it is a .proto encoding and also we don't know the .proto message definition we may just assume that the backend knows how to query the next batch of tweets based on the cursor string.

The TimelineResponse.variables.seenTweetIds parameter is used to inform the server about which tweets from the currently active page of the infinite scrolling the client has already seen. This most probably helps ensure that the server does not include duplicate tweets in subsequent pages of results.

Linked/hierarchical entities

One of the challenges to be solved in the APIs like home timeline (or Home Feed) is to figure out how to return the linked or hierarchical entities (i.e. tweet → user, tweet → media, media → author, etc):

Should we only return the list of tweets first and then fetch the dependent entities (like user details) in a bunch of separate queries on-demand?
Or should we return all the data at once, increasing the time and the size of the first load, but saving the time for all subsequent calls?
- Do we need to normalize the data in this case to reduce the payload size (i.e. when the same user is an author of many tweets and we want to avoid repeating the user data over and over again in each tweet entity)?
Or should it be a combination of the approaches above?

Let's see how X handles it.

Earlier in the TimelineTweet type the Tweet sub-type was used. Let's see how it looks:

export type TimelineResponse = {
  data: {
    home: {
      home_timeline_urt: {
        instructions: (TimelineAddEntries | TimelineTerminateTimeline)[]; // <-- Here
        // ...
      };
    };
  };
};

type TimelineAddEntries = {
  type: 'TimelineAddEntries';
  entries: (TimelineItem | TimelineCursor | TimelineModule)[]; // <-- Here
};

type TimelineItem = {
  entryId: string;
  sortIndex: string;
  content: {
    __typename: 'TimelineTimelineItem';
    itemContent: TimelineTweet; // <-- Here
    // ...
  };
};

type TimelineTweet = {
  __typename: 'TimelineTweet';
  tweet_results: {
    result: Tweet; // <-- Here
  };
};

// A Tweet entity
type Tweet = {
  __typename: 'Tweet';
  core: {
    user_results: {
      result: User; // <-- Here (a dependent User entity)
    };
  };
  legacy: {
    full_text: string;
    // ...
    entities: { // <-- Here (a dependent Media entities)
      media: Media[];
      hashtags: Hashtag[];
      urls: Url[];
      user_mentions: UserMention[];
    };
  };
};

// A User entity
type User = {
  __typename: 'User';
  id: string; // 'VXNlcjoxNDUxM4ADSG44MTA4NDc4OTc2'
  // ...
  legacy: {
    location: string; // 'San Francisco'
    name: string; //  'John Doe'
    // ...
  };
};

// A Media entity
type Media = {
  // ...
  source_user_id_str: string; // '1867041249938530657'  <-- Here (the dependant user is being mentioned by its ID)
  url: string; // 'https://t.co/X78dBgtrsNU'
  features: {
    large: { faces: FaceGeometry[] };
    medium: { faces: FaceGeometry[] };
    small: { faces: FaceGeometry[] };
    orig: { faces: FaceGeometry[] };
  };
  sizes: {
    large: MediaSize;
    medium: MediaSize;
    small: MediaSize;
    thumb: MediaSize;
  };
  video_info: VideoInfo[];
};

What's interesting here is that most of the dependent data like tweet → media and tweet → author is embedded into the response on the first call (no subsequent queries).

Also, the User and Media connections with Tweet entities are not normalized (if two tweets have the same author, their data will be repeated in each tweet object). But it seems like it should be ok, since in the scope of the home timeline for a specific user the tweets will be authored by many authors and repetitions are possible but sparse.

My assumption was that the UserTweets API (that we don't cover here), which is responsible for fetching the tweets of one particular user will handle it differently, but, apparently, it is not the case. The UserTweets returns the list of tweets of the same user and embeds the same user data over and over again for each tweet. It's interesting. Maybe the simplicity of the approach beats some data size overhead (maybe user data is considered pretty small in size). I'm not sure.

Another observation about the entities' relationship is that the Media entity also has a link to the User (the author). But it does it not via direct entity embedding as the Tweet entity does, but rather it links via the Media.source_user_id_str property.

The "comments" (which are also the "tweets" by their nature) for each "tweet" in the home timeline are not fetched at all. To see the tweet thread the user must click on the tweet to see its detailed view. The tweet thread will be fetched by calling the TweetDetail endpoint (more about it in the "Tweet detail page" section below).

Another entity that each Tweet has is FeedbackActions (i.e. "Recommend less often" or "See fewer"). The way the FeedbackActions are stored in the response object is different from the way the User and Media objects are stored. While the User and Media entities are part of the Tweet, the FeedbackActions are stored separately in TimelineItem.content.feedbackInfo.feedbackKeys array and are linked via the ActionKey. That was a slight surprise for me since it doesn't seem to be the case that any action is re-usable. It looks like one action is used for one particular tweet only. So it seems like the FeedbackActions could be embedded into each tweet in the same way as Media entities. But I might be missing some hidden complexity here (like the fact that each action can have children actions).

More details about the actions are in the "Tweet actions" section below.

Sorting

The sorting order of the timeline entries is defined by the backend via the sortIndex properties:

type TimelineCursor = {
  entryId: string;
  sortIndex: string; // '1866961576813152212' <-- Here
  content: {
    __typename: 'TimelineTimelineCursor';
    value: string;
    cursorType: 'Top' | 'Bottom';
  };
};

type TimelineItem = {
  entryId: string;
  sortIndex: string; // '1866561576636152411' <-- Here
  content: {
    __typename: 'TimelineTimelineItem';
    itemContent: TimelineTweet;
    feedbackInfo: {
      feedbackKeys: ActionKey[];
    };
  };
};

type TimelineModule = {
  entryId: string;
  sortIndex: string; // '73343543020642838441' <-- Here
  content: {
    __typename: 'TimelineTimelineModule';
    items: {
      entryId: string,
      item: TimelineTweet,
    }[],
    displayType: 'VerticalConversation',
  };
};

The sortIndex itself might look something like this '1867231621095096312'. It likely corresponds directly to or is derived from a Snowflake ID.

Actually most of the IDs you see in the response (tweet IDs) follow the "Snowflake ID" convention and look like '1867231621095096312'.

If this is used to sort entities like tweets, the system leverages the inherent chronological sorting of Snowflake IDs. Tweets or objects with a higher sortIndex value (a more recent timestamp) appear higher in the feed, while those with lower values (an older timestamp) appear lower in the feed.

Here’s the step-by-step decoding of the Snowflake ID (in our case the sortIndex) 1867231621095096312:

Extract the Timestamp:
- The timestamp is derived by right-shifting the Snowflake ID by 22 bits (to remove the lower 22 bits for data center, worker ID, and sequence): 1867231621095096312 → 445182709954
Add Twitter's Epoch:
- Adding Twitter's custom epoch (1288834974657) to this timestamp gives the UNIX timestamp in milliseconds: 445182709954 + 1288834974657 → 1734017684611ms
Convert to a human-readable date:
- Converting the UNIX timestamp to a UTC datetime gives: 1734017684611ms → 2024-12-12 15:34:44.611 (UTC)

So we can assume here that the tweets in the home timeline are sorted chronologically.

Tweet actions

Each tweet has an "Actions" menu.

The actions for each tweet are coming from the backend in a TimelineItem.content.feedbackInfo.feedbackKeys array and are linked with the tweets via the ActionKey:

type TimelineResponse = {
  data: {
    home: {
      home_timeline_urt: {
        instructions: (TimelineAddEntries | TimelineTerminateTimeline)[];
        responseObjects: {
          feedbackActions: TimelineAction[]; // <-- Here
        };
      };
    };
  };
};

type TimelineItem = {
  entryId: string;
  sortIndex: string;
  content: {
    __typename: 'TimelineTimelineItem';
    itemContent: TimelineTweet;
    feedbackInfo: {
      feedbackKeys: ActionKey[]; // ['-1378668161'] <-- Here
    };
  };
};

type TimelineAction = {
  key: ActionKey; // '-609233128'
  value: {
    feedbackType: 'NotRelevant' | 'DontLike' | 'SeeFewer'; // ...
    prompt: string; // 'This post isn’t relevant' | 'Not interested in this post' | ...
    confirmation: string; // 'Thanks. You’ll see fewer posts like this.'
    childKeys: ActionKey[]; // ['1192182653', '-1427553257'], i.e. NotInterested -> SeeFewer
    feedbackUrl: string; // '/2/timeline/feedback.json?feedback_type=NotRelevant&action_metadata=SRwW6oXZadPHiOczBBaAwPanEwE%3D'
    hasUndoAction: boolean;
    icon: string; // 'Frown'
  };
};

It is interesting here that this flat array of actions is actually a tree (or a graph? I didn't check), since each action may have child actions (see the TimelineAction.value.childKeys array). This makes sense, for example, when after the user clicks on the "Don't Like" action, the follow-up might be to show the "This post isn’t relevant" action, as a way of explaining why the user doesn't like the tweet.

Tweet detail page

Once the user would like to see the tweet detail page (i.e. to see the thread of comments/tweets), the user clicks on the tweet and the GET request to the following endpoint is performed:

GET https://x.com/i/api/graphql/{query-id}/TweetDetail?variables={"focalTweetId":"1867231621095096312","referrer":"home","controller_data":"DACABBSQ","rankingMode":"Relevance","includePromotedContent":true,"withCommunity":true}&features={"articles_preview_enabled":true}

I was curious here why the list of tweets is being fetched via the POST call, but each tweet detail is fetched via the GET call. Seems inconsistent. Especially keeping in mind that similar query parameters like query-id, features, and others this time are passed in the URL and not in the request body. The response format is also similar and is re-using the types from the list call. I'm not sure why is that. But again, I'm sure I might be might be missing some background complexity here.

Here are the simplified response body types:

type TweetDetailResponse = {
  data: {
    threaded_conversation_with_injections_v2: {
      instructions: (TimelineAddEntries | TimelineTerminateTimeline)[],
    },
  },
}

type TimelineAddEntries = {
  type: 'TimelineAddEntries';
  entries: (TimelineItem | TimelineCursor | TimelineModule)[];
};

type TimelineTerminateTimeline = {
  type: 'TimelineTerminateTimeline',
  direction: 'Top',
}

type TimelineModule = {
  entryId: string; // 'conversationthread-58668734545929871193'
  sortIndex: string; // '1867231621095096312'
  content: {
    __typename: 'TimelineTimelineModule';
    items: {
      entryId: string, // 'conversationthread-1866876425669871193-tweet-1866876038930951193'
      item: TimelineTweet,
    }[], // Comments to the tweets are also tweets
    displayType: 'VerticalConversation',
  };
};

The response is pretty similar (in its types) to the list response, so we won't for too long here.

One interesting nuance is that the "comments" (or conversations) of each tweet are actually other tweets (see the TimelineModule type). So the tweet thread looks very similar to the home timeline feed by showing the list of TimelineTweet entries. This looks elegant. A good example of a universal and re-usable approach to the API design.

Liking the tweet

When a user likes the tweet, the POST request to the following endpoint is being performed:

POST https://x.com/i/api/graphql/{query-id}/FavoriteTweet

Here is the request body types:

type FavoriteTweetRequest = {
  variables: {
    tweet_id: string; // '1867041249938530657'
  };
  queryId: string; // 'lI07N61twFgted2EgXILM7A'
};

Here is the response body types:

type FavoriteTweetResponse = {
  data: {
    favorite_tweet: 'Done',
  }
}

Looks straightforward and also resembles the RPC-like approach to the API design.

Conclusion

We have touched on some basic parts of the home timeline API design by looking at X's API example. I made some assumptions along the way to the best of my knowledge. I believe some things I might have interpreted incorrectly and I might have missed some complex nuances. But even with that in mind, I hope you got some useful insights from this high-level overview, something that you could apply in your next API Design session.

Initially, I had a plan to go through similar top-tech websites to get some insights from Facebook, Reddit, YouTube, and others and to collect battle-tested best practices and solutions. I'm not sure if I'll find the time to do that. Will see. But it could be an interesting exercise.

Appendix: All types in one place

For the reference, I'm adding all types in one go here. You may also find all types in types/x.ts file.

/**
 * This file contains the simplified types for X's (Twitter's) home timeline API.
 *
 * These types are created for exploratory purposes, to see the current implementation
 * of the X's API, to see how they fetch Home Feed, how they do a pagination and sorting,
 * and how they pass the hierarchical entities (posts, media, user info, etc).
 *
 * Many properties and types are omitted for simplicity.
 */

// POST https://x.com/i/api/graphql/{query-id}/HomeTimeline
export type TimelineRequest = {
  queryId: string; // 's6ERr1UxkxxBx4YundNsXw'
  variables: {
    count: number; // 20
    cursor?: string; // 'DAAACgGBGedb3Vx__9sKAAIZ5g4QENc99AcAAwAAIAIAAA'
    seenTweetIds: string[]; // ['1867041249938530657', '1867041249938530658']
  };
  features: Features;
};

// POST https://x.com/i/api/graphql/{query-id}/HomeTimeline
export type TimelineResponse = {
  data: {
    home: {
      home_timeline_urt: {
        instructions: (TimelineAddEntries | TimelineTerminateTimeline)[];
        responseObjects: {
          feedbackActions: TimelineAction[];
        };
      };
    };
  };
};

// POST https://x.com/i/api/graphql/{query-id}/FavoriteTweet
export type FavoriteTweetRequest = {
  variables: {
    tweet_id: string; // '1867041249938530657'
  };
  queryId: string; // 'lI07N6OtwFgted2EgXILM7A'
};

// POST https://x.com/i/api/graphql/{query-id}/FavoriteTweet
export type FavoriteTweetResponse = {
  data: {
    favorite_tweet: 'Done',
  }
}

// GET https://x.com/i/api/graphql/{query-id}/TweetDetail?variables={"focalTweetId":"1867041249938530657","referrer":"home","controller_data":"DACABBSQ","rankingMode":"Relevance","includePromotedContent":true,"withCommunity":true}&features={"articles_preview_enabled":true}
export type TweetDetailResponse = {
  data: {
    threaded_conversation_with_injections_v2: {
      instructions: (TimelineAddEntries | TimelineTerminateTimeline)[],
    },
  },
}

type Features = {
  articles_preview_enabled: boolean;
  view_counts_everywhere_api_enabled: boolean;
  // ...
}

type TimelineAction = {
  key: ActionKey; // '-609233128'
  value: {
    feedbackType: 'NotRelevant' | 'DontLike' | 'SeeFewer'; // ...
    prompt: string; // 'This post isn’t relevant' | 'Not interested in this post' | ...
    confirmation: string; // 'Thanks. You’ll see fewer posts like this.'
    childKeys: ActionKey[]; // ['1192182653', '-1427553257'], i.e. NotInterested -> SeeFewer
    feedbackUrl: string; // '/2/timeline/feedback.json?feedback_type=NotRelevant&action_metadata=SRwW6oXZadPHiOczBBaAwPanEwE%3D'
    hasUndoAction: boolean;
    icon: string; // 'Frown'
  };
};

type TimelineAddEntries = {
  type: 'TimelineAddEntries';
  entries: (TimelineItem | TimelineCursor | TimelineModule)[];
};

type TimelineTerminateTimeline = {
  type: 'TimelineTerminateTimeline',
  direction: 'Top',
}

type TimelineCursor = {
  entryId: string; // 'cursor-top-1867041249938530657'
  sortIndex: string; // '1867231621095096312'
  content: {
    __typename: 'TimelineTimelineCursor';
    value: string; // 'DACBCgABGedb4VyaJwuKbIIZ40cX3dYwGgaAAwAEAEEAA'
    cursorType: 'Top' | 'Bottom';
  };
};

type TimelineItem = {
  entryId: string; // 'tweet-1867041249938530657'
  sortIndex: string; // '1867231621095096312'
  content: {
    __typename: 'TimelineTimelineItem';
    itemContent: TimelineTweet;
    feedbackInfo: {
      feedbackKeys: ActionKey[]; // ['-1378668161']
    };
  };
};

type TimelineModule = {
  entryId: string; // 'conversationthread-1867041249938530657'
  sortIndex: string; // '1867231621095096312'
  content: {
    __typename: 'TimelineTimelineModule';
    items: {
      entryId: string, // 'conversationthread-1867041249938530657-tweet-1867041249938530657'
      item: TimelineTweet,
    }[], // Comments to the tweets are also tweets
    displayType: 'VerticalConversation',
  };
};

type TimelineTweet = {
  __typename: 'TimelineTweet';
  tweet_results: {
    result: Tweet;
  };
};

type Tweet = {
  __typename: 'Tweet';
  core: {
    user_results: {
      result: User;
    };
  };
  views: {
    count: string; // '13763'
  };
  legacy: {
    bookmark_count: number; // 358
    created_at: string; // 'Tue Dec 10 17:41:28 +0000 2024'
    conversation_id_str: string; // '1867041249938530657'
    display_text_range: number[]; // [0, 58]
    favorite_count: number; // 151
    full_text: string; //  "How I'd promote my startup, if I had 0 followers (Part 1)"
    lang: string; // 'en'
    quote_count: number;
    reply_count: number;
    retweet_count: number;
    user_id_str: string; // '1867041249938530657'
    id_str: string; // '1867041249938530657'
    entities: {
      media: Media[];
      hashtags: Hashtag[];
      urls: Url[];
      user_mentions: UserMention[];
    };
  };
};

type User = {
  __typename: 'User';
  id: string; // 'VXNlcjoxNDUxM4ADSG44MTA4NDc4OTc2'
  rest_id: string; // '1867041249938530657'
  is_blue_verified: boolean;
  profile_image_shape: 'Circle'; // ...
  legacy: {
    following: boolean;
    created_at: string; // 'Thu Oct 21 09:30:37 +0000 2021'
    description: string; // 'I help startup founders double their MRR with outside-the-box marketing cheat sheets'
    favourites_count: number; // 22195
    followers_count: number; // 25658
    friends_count: number;
    location: string; // 'San Francisco'
    media_count: number;
    name: string; //  'John Doe'
    profile_banner_url: string; // 'https://pbs.twimg.com/profile_banners/4863509452891265813/4863509'
    profile_image_url_https: string; // 'https://pbs.twimg.com/profile_images/4863509452891265813/4863509_normal.jpg'
    screen_name: string; // 'johndoe'
    url: string; // 'https://t.co/dgTEddFGDd'
    verified: boolean;
  };
};

type Media = {
  display_url: string; // 'pic.x.com/X7823zS3sNU'
  expanded_url: string; // 'https://x.com/johndoe/status/1867041249938530657/video/1'
  ext_alt_text: string; // 'Image of two bridges.'
  id_str: string; // '1867041249938530657'
  indices: number[]; // [93, 116]
  media_key: string; // '13_2866509231399826944'
  media_url_https: string; // 'https://pbs.twimg.com/profile_images/1867041249938530657/4863509_normal.jpg'
  source_status_id_str: string; // '1867041249938530657'
  source_user_id_str: string; // '1867041249938530657'
  type: string; // 'video'
  url: string; // 'https://t.co/X78dBgtrsNU'
  features: {
    large: { faces: FaceGeometry[] };
    medium: { faces: FaceGeometry[] };
    small: { faces: FaceGeometry[] };
    orig: { faces: FaceGeometry[] };
  };
  sizes: {
    large: MediaSize;
    medium: MediaSize;
    small: MediaSize;
    thumb: MediaSize;
  };
  video_info: VideoInfo[];
};

type UserMention = {
  id_str: string; // '98008038'
  name: string; // 'Yann LeCun'
  screen_name: string; // 'ylecun'
  indices: number[]; // [115, 122]
};

type Hashtag = {
  indices: number[]; // [257, 263]
  text: string;
};

type Url = {
  display_url: string; // 'google.com'
  expanded_url: string; // 'http://google.com'
  url: string; // 'https://t.co/nZh3aF0Aw6'
  indices: number[]; // [102, 125]
};

type VideoInfo = {
  aspect_ratio: number[]; // [427, 240]
  duration_millis: number; // 20000
  variants: {
    bitrate?: number; // 288000
    content_type?: string; // 'application/x-mpegURL' | 'video/mp4' | ...
    url: string; // 'https://video.twimg.com/amplify_video/18665094345456w6944/pl/-ItQau_LRWedR-W7.m3u8?tag=14'
  };
};

type FaceGeometry = { x: number; y: number; h: number; w: number };

type MediaSize = { h: number; w: number; resize: 'fit' | 'crop' };

type ActionKey = string;

Homemade GPT JS

Oleksii Trekhleb — Sun, 10 Nov 2024 17:41:44 +0000

For learning purposes, I made a minimal TensorFlow.js re-implementation of Karpathy's minGPT (Generative Pre-trained Transformer).

A full definition of this "homemade" GPT language model (all of it) can be found in this single model.ts file (less than 300 lines of code).

Since model.ts is written in TypeScript, you can use homemade GPT playground to train it, experiment with parameters, and generate its predictions directly in the browser using a GPU.

The model and the playground are written for learning purposes, to understand how GPT works and to use WebGPU for training.

To understand what's happening in the model.ts file please refer to Andrej Karpathy's well-explained, hands-on lecture "Let's build GPT: from scratch, in code, spelled out" (arguably one of the best explanations of GPT out there).

GPT Folder

Inside the ./gpt/src/ folder you'll find the following files:

model.ts - this is the main file of interest, as it contains the full (yet minimalistic) definition of the decoder GPT model, as described in the Attention Is All You Need paper.
model-easier.ts - this is the same GPT model as in the previous file but simplified for easier understanding. The main difference is that it processes all Heads inside CausalSelfAttention sequentially (instead of in parallel). As a result, the model is a bit slower but more readable.
config.ts - contains pre-configured sets of GPT model parameters: GPT-pico, GPT-nano, GPT-mini, GPT-2, etc.
dataset.ts - Nothing GPT-specific here. A helper wrapper on top of any txt-file-based character-level dataset. It loads an arbitrary txt file, treats each letter as a token, splits the characters into training and testing batches, and encodes/decodes letters to indices and vice versa.
trainer.ts - Nothing GPT-specific here as well. This file provides a simple training loop that could apply to any arbitrary neural network.

Web Playground

To experiment with model parameters, training, and text generation you may use the Homemade GPT JS playground.

You may also launch the playground locally if you want to modify and experiment with the code of the transformer model itself.

Install dependencies:

npm i

Launch web playground locally:

npm run playground-web

The playground will be accessible on http://localhost:3000/homemade-gpt-js

Run these commands from the root of the project. You need to have Node.js ≥ 20.0.0.

Node.js Playground

You may also experiment with the model in Node.js environment.

Install dependencies:

npm i

Launch Node.js playground:

npm run playground-node

The ./playground-node/src/index.ts file contains the basic example of training and text generation.

Run these commands from the root of the project. You need to have Node.js ≥ 20.0.0.

I hope this TS-based GPT example will be helpful for you, learners.

System Design Sketches

Oleksii Trekhleb — Mon, 15 Jul 2024 16:22:35 +0000

System design sketches for popular system design interview questions.

See the sketches in full resolution at okso.app/showcase/system-design

Social Network

Functional requirements:
------------------------
- Follow people
- See followers and falowees
- Create posts (text, image, url, @mentions)
- Like posts
- View user feed
- View home feed


Non-functional requirements:
----------------------------
- Add post <500ms
- View feed <500ms
- Post in feed with ~2 mins delay is ok
- Billions of users
- Unlimited followers


Core entities:
--------------
- User
- Follow
- Post


API (RPC-like):
---------------
- POST /getFollowers     {userId}
- POST /getFollowees     {userId}
- POST /followUser       {userId}

- POST /createPost       {text}
- POST /likePost         {postId}

- POST /getHomeFeed      {userId}
- POST /getUserFeed      {userId}

Design sketch:

Deep dives:

Social network: System design sketch (okso.app)
Facebook News Feed (hellointerview)
Instagram + Twitter + Facebook + Reddit (Jordan has no life)

Ride-Sharing App

Functional requirements:
------------------------
- Rider gets estimated fare (by Start and Destination locations)
- Rider requests a ride based on the estimate
- Driver accepts/denies request + Navigates to pickup/drop-off


Non-functional requirements:
----------------------------
- Low latency for rider-driver matching (<1min)
- Consistency of rider-driver 1:1 matching
  - For one ride, only one driver will get request at a time
- Highly available outside matching
- Handle high-throughput (peak hours, popular events)


Core entities:
--------------
- Rider
- Driver
- Ride
- Location


API:
---------------
- POST /getEstimate({from, to}) --> Partial<Ride>

- POST /requestRide({ride_id}) --> Ride

- POST /getRide({ride_id})

  - gets the latest ride entity to display the
    ride details for both rider and driver apps

- POST /updateLocation({lat, lon})

  - both driver and rider can update their locations
    which may be used to display their on the map
  - drivers might update their location i.e. once
    per 5 seconds (it is needed for matching)
  - however this can be optimised on the client side,
    where we might say that location update is
    proportional to driver speed: for still drivers
    we update location rarely, for moving - more often

- POST /acceptRide({ride_id, accept})
  - ride is being accepted by the driver

- POST /updateRide({ride_id, status})

Design sketch:

Deep dives:

Ride-sharing platform: System design sketch (okso.app)
Design Uber (hellointerview)

Messenger

Functional requirements:
------------------------
- Group chat (<100 users)
- Send message
- Receive message
- Persist messages (user messages are
  available on multiple devices)


Non-functional requirements:
----------------------------
- Low latency for message delivery (<500ms)
- No lost messages (delivery guarantee)


Core entities:
--------------
- Users
- Groups
- Messages


API:
---------------
- createGroup({users: [], name: ''}) -> {group_id}
- joinGroup({group_id}) -> "ACK"/ "ERR"
- quitGroup({group_id}) -> "ACK"/ "ERR"
- getMyGroups() -> {groups: []}
- getGroupMessages({group_id, max_message_id, min_message_id}) -> {messages: []}

WebSocket:
- connect()
- sendMessage({group_id, message, media: []}) -> "ACK"/ "ERR"
- onGroupUpdated({group_id, participants}) -> "ACK"
- onMessage({chat_id, user_id, message, attachments}) -> "ACK"

Design sketch:

Deep dives:

Messenger: System design sketch (okso.app)
Facebook Messenger + WhatsApp (Jordan has no life)
Messenger/WhatsApp Design Deep Dive (Jordan has no life)
Design Whatsapp or Messenger (hellointerview)

Video Streaming App

Functional requirements:
------------------------
- Upload video
- Watch video (read-heavy)
- Comment on video


Non-functional requirements:
----------------------------
- Reliable uploads (no corrupted/missing videos)
- Read-heavy (read/write ratio ~100:1)
- Availability > Consistency (1-2 min delay between upload and feed is ok)
- Low latency (before video starts playing)


Core entities:
--------------
- User
- Video Metadata
- Video Chunks
- Users
- Comments


API (RPC-like):
---------------
- POST /initiateUpload  {title, description, video, ...}}
  - server responds with S3 private upload URL
  - the actual video upload is going directly from client to S3

- GET /watch   {video_id}
  - returns video chunks metadata
    - S3 (or CDN) URLs to manifest/playlist files
  - video streaming happens directly from CDNs
  - download by chunks (buffering, ~1Mb), no need to load all

Design sketch:

Deep dives:

Video streaming platform: System design sketch (okso.app)
Design Youtube (NeetCode)
Netflix + YouTube (Jordan has no life)
System Design: YouTube (System Design Fight Club)

File Hosting App

Functional requirements:
------------------------
- Upload file
- Download file
- Sync files across devices


Non-functional requirements:
----------------------------
- Availability over consistency
  - Download the older version of
    the file instead of waiting
    for a new version to propagate
    is ok
- low latency for uploads and downloads
- support large files (50Gb)
  - resumable uploads
- high data integrity (accurate syncs)


Core entities:
--------------
- File (row bytes)
- File Metadata
- Users


API:
---------------
- POST /uploadFile({metadata}) --> {upload_id, presigned_url}

- POST /getFile({file_id}) --> {file{}, metadata}

- POST /getChanges({since}) --> file_metadata[]

Design sketch:

Deep dives:

File hosting app (okso.app)
Design Dropbox (hellointerview)

Web Crawler

Functional requirements:
------------------------
- Scrape the web starting from seed URLs
- Extract and store text data


Non-functional requirements:
----------------------------
- Fault-tolerant
- Politeness (respect robots.txt, don't DDoS)
- Scale to 1B pages
- Efficient (crawl within a week)


Core entities:
--------------
- URLs
- Text data
- Domain metadata


Interface:
----------
- input: seed URLs
- output: text data


Data Flow:
----------
1. Take seed URLs from a frontier and the IP from DNS
2. Fetch HTML
3. Extract text
4. Store text in the database
5. Extract URLs from the page and add then to the frontier
6. Repeat steps 1-5 until all URLs are crawled

Design sketch:

Deep dives:

Web-crawler app: System design sketch (okso.app)
Design a Web Crawler (hellointerview)
Design a Web Crawler (Jordan has no life)
System Design distributed web crawler (Tech Dummies Narendra)

Micrograd TS

Oleksii Trekhleb — Mon, 07 Aug 2023 07:50:16 +0000

I recently went through a very detailed and well-explained Python-based project/lesson by karpathy which is called micrograd. This is a tiny scalar-valued autograd engine and a neural net on top of it. This video explains how to build such a network from scratch.

The project above is, as expected, built on Python. For learning purposes, I wanted to see how such a network may be implemented in TypeScript and came up with a 🤖 micrograd-ts repository (and also with a demo of how the network may be trained).

Trying to build anything on your own very often gives you a much better understanding of a topic. So, this was a good exercise, especially taking into account that the whole code is just a ~200 lines of TS code with no external dependencies.

The micrograd-ts repository might be useful for those who want to get a basic understanding of how neural networks work, using a TypeScript environment for experimentation.

With that being said, let me give you some more information about the project.

Project structure

micrograd/ — this folder is the core/purpose of the repo
- engine.ts — the scalar Value class that supports basic math operations like add, sub, div, mul, pow, exp, tanh and has a backward() method that calculates a derivative of the expression, which is required for back-propagation flow.
- nn.ts — the Neuron, Layer, and MLP (multi-layer perceptron) classes that implement a neural network on top of the differentiable scalar Values.
demo/ - demo React application to experiment with the micrograd code
- src/demos/ - several playgrounds where you can experiment with the Neuron, Layer, and MLP classes.

Micrograd

See the 🎬 The spelled-out intro to neural networks and back-propagation: building micrograd YouTube video for the detailed explanation of how neural networks and backpropagation work. The video also explains in detail what the Neuron, Layer, MLP, and Value classes do.

Briefly, the Value class allows you to build a computation graph for some expression that consists of scalar values.

Here is an example of how the computation graph for the a * b + c expression looks like:

Based on the Value class we can build a Neuron expression X * W + b. Here we're simulating a dot-product of matrix X (input features) and matrix W (neuron weights):

Out of Neurons, we can build the MLP network class that consists of several Layers of Neurons. The computation graph in this case may look a bit complex to be displayed here, but a simplified version might look like this:

The main idea is that the computation graphs above "know" how to do automatic back propagation (in other words, how to calculate derivatives). This allows us to train the MLP network for several epochs and adjust the network weights in a way that reduces the ultimate loss:

Demo (online)

To see the online demo/playground, check the following link:

🔗 trekhleb.dev/micrograd-ts

Demo (local)

If you want to experiment with the code locally, follow the instructions below.

Setup

Clone the current repo locally.

Switch to the demo folder:

cd ./demo

Setup node v18 using nvm (optional):

nvm use

Install dependencies:

npm i

Launch demo app:

npm run dev

The demo app will be available at http://localhost:5173/micrograd-ts

Playgrounds

Go to the ./demo/src/demos/ to explore several playgrounds for the Neuron, Layer, and MLP classes.

I hope, playing around with the micrograd-ts code above and watching the video from Karpathy will be helpful at least for some of you, learners.

Observations

Oleksii Trekhleb — Tue, 27 Jun 2023 19:30:36 +0000

This is a collection of my subjective assumptions, questions, and interpretations about the world around us. Don’t take it seriously.

Check the Observations post for the detailed breakdown of observations and assumptions that has interactive dependency graph

SOLID Principles Sketches

Oleksii Trekhleb — Wed, 05 Oct 2022 00:45:15 +0000

I've recently launched the minimalistic drawing app okso.app that allows you to do interactive (nested) sketches. And, as a continuation of my previous S.O.L.I.D. Principles Around You article, I've organised it in interactive sketches that you may find here.

Check what I've got:

So, each SOLID principle has a dedicated sketch page that look like the following

Single Responsibility Principle

SRP Sketch

A class should have only a single responsibility. Only one potential change in the software’s specification should be able to affect the specification of the class.

Open/Closed Principle

OCP Sketch

Software entities should be open for EXTENSION, but closed for MODIFICATION. Allow behavior to be extended without modifying the source code.

Liskov Substitution Principle

LSP Sketch

Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program.

Interface Segregation Principle

ISP Sketch

Many client-specific interfaces are better than one general-purpose interface. No client should be forced to depend on methods it does not use.

Dependency Inversion Principle

DIP Sketch

One should depend upon abstractions, not concretions.

High-level modules should not depend on low-level modules. Both should depend on abstractions.
Abstractions should not depend on details. Details should depend on abstractions.

In the future I plan to add more visual explainers like this one to the okso.app showcase. Currently there is a Data Structures sketch is available that is done in a similar style as SOLID one. I hope having such visual explainers in one place will be convenient for you.

Data Structure Sketches

Oleksii Trekhleb — Wed, 31 Aug 2022 10:31:39 +0000

Recently I launched the minimalistic online drawing app okso.app. I wanted it to be a place where people could do fast, ad-hoc, napkin-based-like explanations of any concept as if you are sitting with your friend and trying to explain him/her something during lunch. Don't ask me why it is needed, I was just experimenting.

Btw, that's why the app name is "Ok! So...", because oftentimes these are the opening words after you pick up the pen and the piece of paper (or a whiteboard) and start drawing. I didn't see the study and the statistics of the opening phrases though, so I might be wrong here, here 😀

So, the first concept I've tried to explain with sketches was the Data Structures. Without further ado, here is the interactive ✍🏻 Data Structure Sketches showcase that you may play with.

Here is a YouTube version of the GIF just in case:

Of course, not all data structures are covered. And of course, this is not comprehensive material, but rather a cheatsheet that would create visual hints and associations for the following data structures:

Linked List
Doubly Linked List
Queue
Stack
Hash Table (with hash collision resolution)
Tree (including the Binary Search Tree)
Heap (including Mean Heap and Max Heap)
Trie
Graph

Each box on the sketch is clickable, so you may dig into the data structure you're interested. For example Heap → Max Heap, or Heap → Min Heap, or Heap → Array Representation.

The sketches are split into so-called Pages just to make it easier to grasp them, so the users stay focused on one concept at a time, they see the relationship between the concept, and thus, hopefully, they are not getting overwhelmed with seeing a lot of information at the same time on one drawing/page.

Each page has a link to the source-code examples that are implementing the data structure on JavaScript.

Here are some examples of the sketches.

Doubly Linked List

Tree

Hash Table

Trie

Graph

The full list you may find in the ✍🏻 Data Structure Sketches showcase.

I hope you find this showcase useful and I hope it will be a good visual cheatsheet-like complement to your data structure knowledge.

Ok! So... The drawing app to express, grasp, and organize your thoughts and ideas

Oleksii Trekhleb — Wed, 20 Jul 2022 15:21:37 +0000

TL;DR

The OkSo app is the drawing app to express, grasp, and organize your thoughts and ideas. Draw to explain. Draw to grasp.

Concept

Some of you are visual learners. You need to see pictures and visualizations to explain and grasp concepts. You keep a pencil and a piece of paper at hand. Sometimes it feels like connecting two circles with an arrow would speak more apparent than describing such circles' relationship with words. Visualizations, drawings, and sketches help you organize your thoughts and knowledge.

The process of visualization is not always easy. It requires some thinking and energy. That's why you might find yourself taking a deep breath, and saying "Ok! So..." to yourself or your vis-à-vis before diving deep into the concept. That's the reason for the Thinker with the "Ok! So..." in the logo.

The physical paper or the notepad is good but it has its limitations:

You may forget it at home.
The page space might not be enough to contain all the details.
Often you only have pens of one or two colors.
If you made a typo you can't just undo/redo it, the sketch is not really editable.

The OkSo app tries to overcome these limitations and gives users the possibility to "Draw to explain" and "Draw to grasp".

Features

The limitations mentioned above are being addressed as follows:

The OkSo app is available online and thus it is always with you. Currently, you may export all your drawings to the file and share them between devices manually (i.e. via Google Docs or Dropbox). All your drawings are stored in your browser. We don't send your data to the back-end. The possibility to automatically sync your drawings between devices is in our Roadmap.
It gives you an "infinite" canvas.
No color limits.
You may edit your drawings and undo/redo your edits.

Many online drawing apps give you these (and not only) possibilities. Take a look at tldraw or excalidraw, for example. These apps are awesome! Even more, the OkSo app is built on top of the @tldraw/tldraw NPM package itself.

However, what the OkSo app tries to do on top of that is to experiment with the way you organize your drawings. It uses the concept of nested pages. The page (to be more precise the link to a page) is a first-class citizen of your drawing along with other shapes like freehand curves, rectangles, ellipses, or texts. So you may embed/draw links to sub-pages inside the page. It means that you may organize your drawings in the nested tree structure.

For example, here is one of the variants you may organize your drawings/pages:

Root drawing
  ├─ Goals
  │     ├─ 2021
  │     ├─ 2022
  │     └─ ...
  ├─ Career
  └─ Learning
        ├─ Machine Learning
        └─ Algorithms breakdown
              ├─ Binary search
              ├─ Power Set
              └─ ...

Each item in this tree is a separate drawing/page itself. The content of each page might be anything you are using your notepad for. For example:

My 2022 goals
My career plans
The "Drawing App" system design
Binary search algorithm breakdown
Brainstorming the startup ideas
The AI learning path
Logistics for my next trip
You name it...

The fact that the link to the sub-page is just a regular shape allows you to use size- and color-coding, to group the links together, and to easily distinguish them visually.

Also, compared to the flat lists, you'll not be overwhelmed by a large number of pages, since they may be organized in a tree structure. You will only see the links to the sub-pages of a particular level. You can "travel" easily between the sub-pages back and forth, deeper and deeper until you find a page you need.

Another approach, that the OkSo app takes compared to the other drawing apps is to be as simple and as minimalist as possible (just like your physical notepad). This relates not only to the UI but to the list of tools and features in particular (therefore the rectangle, the ellipse, and the arrow are hidden from the tools panel by default). For example, instead of having a reach collection of primitives, what about just drawing the "DataBase Cylinder" icon from scratch? Yes, one of the real reasons for that is plain simple laziness and desire to code less :D However, I believe, this approach also feels more authentic and artistic. When you use a whiteboard during the technical interview or during the brainstorming you draw such primitives easily, aren't you? Plus, having such basic functionalities as copy-pasting, cloning, and resizing should allow you to deliver your thought fast and easily.

The OkSo app is a Progressive Web App (PWA). It means that you may install it on your device (add to your home screen) for faster access (see instructions).

Demo

Here is a demo of how the okso.app may be used to explain/grasp the idea behind some basic data structures:

Feedback

Note, that the app is in its early beta stage and some features from our Roadmap are still in progress and will be delivered soon.

Meanwhile:

feel free to subscribe to the product updates if you're interested
leave feedback (i.e. suggest a feature or report a bug)
and follow OkSo on Twitter

Top 33 JavaScript Projects on GitHub (November 2021)

Oleksii Trekhleb — Sat, 13 Nov 2021 15:23:08 +0000

2021 is coming to its end, and we may do another snapshot of 33 most starred open-sourced JavaScript repositories on GitHub as of November 13th, 2021.

Previous snapshots: 2018, 2020, 2021.

You may also query the GitHub to fetch the latest results.

#1 freeCodeCamp/freeCodeCamp

freeCodeCamp.org's open-source codebase and curriculum. Learn to code for free.
★ 335k (+18k)

#2 vuejs/vue

Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
★ 190k (+14k)

#3 facebook/react

A declarative, efficient, and flexible JavaScript library for building user interfaces.
★ 177k (+17k)

#4 twbs/bootstrap

The most popular HTML, CSS, and JavaScript framework for developing responsive, mobile first projects on the web.
★ 154k (+8k)

#5 trekhleb/javascript-algorithms

Algorithms and data structures implemented in JavaScript with explanations and links to further readings
★ 126k (+37k)

#6 airbnb/javascript

JavaScript Style Guide
★ 116k (+13k)

#7 facebook/react-native

A framework for building native applications using React
★ 99k (+7k)

#8 d3/d3

Bring data to life with SVG, Canvas and HTML.
★ 99k (+4k)

#9 facebook/create-react-app

Set up a modern web app by running one command.
★ 91k (+7k)

#10 axios/axios

Promise based HTTP client for the browser and node.js
★ 89k (+9k)

#11 30-seconds/30-seconds-of-code

Short JavaScript code snippets for all your development needs
★ 88k (+22k)

#12 nodejs/node

Node.js JavaScript runtime
★ 83k (+7k)

#13 vercel/next.js

The React Framework
★ 76k (+18k)

#14 mrdoob/three.js

JavaScript 3D Library.
★ 75k (+10k)

#15 mui-org/material-ui

MUI (formerly Material-UI) is the React UI library you always wanted. Follow your own design system, or start with Material Design.
★ 72k (+9k)

#16 goldbergyoni/nodebestpractices

The Node.js best practices list (October 2021)
★ 72k (+15k)

#17 awesome-selfhosted/awesome-selfhosted

A list of Free Software network services and web applications which can be hosted on your own servers
★ 67k (+16k)

#18 FortAwesome/Font-Awesome

The iconic SVG, font, and CSS toolkit
★ 66k (+1k)

#19 yangshun/tech-interview-handbook

Curated interview preparation materials for busy engineers
★ 60k (+60k)

#20 ryanmcdermott/clean-code-javascript

Clean Code concepts adapted for JavaScript
★ 59k (+59k)

#21 webpack/webpack

A bundler for javascript and friends. Packs many modules into a few bundled assets. Code Splitting allows for loading parts of the application on demand. Through "loaders", modules can be CommonJs, AMD, ES6 modules, CSS, Images, JSON, Coffeescript, LESS, ... and your custom stuff.
★ 59k (+3k)

#22 angular/angular.js

AngularJS - HTML enhanced for web apps!
★ 59k (+0k)

#23 hakimel/reveal.js

The HTML Presentation Framework
★ 57k (+2k)

#24 typicode/json-server

Get a full fake REST API with zero coding in less than 30 seconds (seriously)
★ 57k (+6k)

#25 atom/atom

The hackable text editor
★ 56k (+2k)

#26 jquery/jquery

jQuery JavaScript Library
★ 55k (+1k)

#27 chartjs/Chart.js

Simple HTML5 Charts using the canvas tag
★ 55k (+4k)

#28 expressjs/express

Fast, unopinionated, minimalist web framework for node.
★ 54k (+3k)

#29 adam-p/markdown-here

Google Chrome, Firefox, and Thunderbird extension that lets you write email in Markdown and render it before sending.
★ 53k (+4k)

#30 h5bp/html5-boilerplate

A professional front-end template for building fast, robust, and adaptable web apps or sites.
★ 51k (+51k)

#31 gatsbyjs/gatsby

Build blazing fast, modern apps and websites with React
★ 51k (+3k)

#32 lodash/lodash

A modern JavaScript utility library delivering modularity, performance, and extras.
★ 51k (+3k)

#33 resume/resume.github.com

Resumes generated using the GitHub informations
★ 50k (+3k)

Weighted Random algorithm in JavaScript

Oleksii Trekhleb — Fri, 22 Oct 2021 12:19:47 +0000

ℹ️ Examples are from the javascript-algorithms repository

What is "Weighted Random"

Let's say you have a list of items. Item could be anything. For example, we may have a list of fruits and vegetables that you like to eat: [ '🍌', '🍎', '🥕' ].

The list of weights represent the weight (or probability, or importance) of each item. Weights are numbers. For example, the weights like [3, 7, 1] would say that:

you would like to eat 🍎 apples more often (7 out of 3 + 7 + 1 = 11 times),
then you would like to eat bananas 🍌 less often (only 3 out of 11 times),
and the carrots 🥕 you really don't like (want to eat it only 1 out of 11 times).

If we speak in terms of probabilities than the weights list might be an array of floats that sum up to 1 (i.e. [0.1, 0.5, 0.2, 0.2]).

The Weighted Random in this case will be the function that will randomly return you the item from the list, and it will take each item's weight into account, so that items with the higher weight will be picked more often.

Example of the function interface:

const items =   [ '🍌', '🍎', '🥕' ];
const weights = [  3,    7,    1  ];

function weightedRandom(items, weights) {
  // implementation goes here ...
}

const nextSnackToEat = weightedRandom(items, weights); // Could be '🍎'

Applications of Weighted Random

In Genetic Algorithm the weighted random is used during the "Selection" phase, when we need to select the fittest/strongest individuums based on their fitness score for mating and for producing the next stronger generation. You may find an example in the Self-Parking Car in 500 Lines of Code article.
In Recurrent Neural Networks (RNN) when trying to decide what letter to choose next (to form the sentence) based on the next letter probability. You may find an example in the Recipe Generation using Recurrent Neural Network (RNN) Jupyter notebook.
In Nginx Load Balancing to send HTTP requests more often to the servers with the higher weights.
And more...

The Algorithm

The straightforward approach would be to:

Repeat each item in the list according to its weight.
Pick the random item from the list.

For example in our case with fruits and vegetables we could generate the following list of size 3 + 7 + 1 = 11:

const items =   [ '🍌', '🍎', '🥕' ];
const weights = [  3,    7,    1  ];

// Repeating the items based on weights.
const weightedItems = [
  '🍌', '🍌', '🍌',
  '🍎', '🍎', '🍎', '🍎', '🍎', '🍎', '🍎',
  '🥕',
];

// And now just pick the random item from weightedItems array.

However, as you may see, this approach may require a lot of memory, in case if the objects are heavy, and in case if we have a lot of them to repeat in weightedItems list.

The more efficient approach would be to:

Prepare the list of cumulative weights for each item (i.e. the cumulativeWeights list which will have the same number of elements as the original weights list). In our case it will look like this: cumulativeWeights = [3, 3 + 7, 3 + 7 + 1] = [3, 10, 11]
Generate the random number randomNumber from 0 to the highest cumulative weight value. In our case the random number will be in a range of [0..11]. Let's say that we have randomNumber = 8.
Go through the cumulativeWeights list from left to right and pick the first element which is higher or equal to the randomNumber. The index of such element we will use to pick the item from the items array.

The idea behind this approach is that the higher weights will "occupy" more numeric space. Therefore, there is a higher chance that the random number will fall into the "higher weight numeric bucket".

const weights =           [3, 7,  1 ];
const cumulativeWeights = [3, 10, 11];

// In a pseudo-representation we may think about the cumulativeWeights array like this.
const pseudoCumulativeWeights = [
  1, 2, 3,               // <-- [3] numbers
  4, 5, 6, 7, 8, 9, 10,  // <-- [7] numbers
  11,                    // <-- [1] number
];

Here is an example of how the weightedRandom function might be implemented:

/**
 * Picks the random item based on its weight.
 * The items with higher weight will be picked more often (with a higher probability).
 *
 * For example:
 * - items = ['banana', 'orange', 'apple']
 * - weights = [0, 0.2, 0.8]
 * - weightedRandom(items, weights) in 80% of cases will return 'apple', in 20% of cases will return
 * 'orange' and it will never return 'banana' (because probability of picking the banana is 0%)
 *
 * @param {any[]} items
 * @param {number[]} weights
 * @returns {{item: any, index: number}}
 */
export default function weightedRandom(items, weights) {
  if (items.length !== weights.length) {
    throw new Error('Items and weights must be of the same size');
  }

  if (!items.length) {
    throw new Error('Items must not be empty');
  }

  // Preparing the cumulative weights array.
  // For example:
  // - weights = [1, 4, 3]
  // - cumulativeWeights = [1, 5, 8]
  const cumulativeWeights = [];
  for (let i = 0; i < weights.length; i += 1) {
    cumulativeWeights[i] = weights[i] + (cumulativeWeights[i - 1] || 0);
  }

  // Getting the random number in a range of [0...sum(weights)]
  // For example:
  // - weights = [1, 4, 3]
  // - maxCumulativeWeight = 8
  // - range for the random number is [0...8]
  const maxCumulativeWeight = cumulativeWeights[cumulativeWeights.length - 1];
  const randomNumber = maxCumulativeWeight * Math.random();

  // Picking the random item based on its weight.
  // The items with higher weight will be picked more often.
  for (let itemIndex = 0; itemIndex < items.length; itemIndex += 1) {
    if (cumulativeWeights[itemIndex] >= randomNumber) {
      return {
        item: items[itemIndex],
        index: itemIndex,
      };
    }
  }
}

Implementation

Check the weightedRandom.js file for the implementation of the weightedRandom() function.
Check the weightedRandom.test.js file for the tests-cases.