[PART 18][Backend] Creating a Twitter clone with GraphQL, Typescript, and React ( hashtags, link's preview )

Hi everyone ;).

As a reminder, I'm doing this Tweeter challenge

Github repository ( Backend )

Github repository ( Frontend )

Db diagram

Hashtags

Adding a tweet is quite a lot of work actually :D. I wanted to quickly add a form in the frontend and start posting tweets. But I will try to add the same functionalities as the real Twitter. First of all, I will need to extract and keep track of the hashtags. This will allow me to have some data to play with when I'll need to show the trending ones. I need a many-to-many relationship for that. I will then create two tables (hashtags and hashtags_tweets). For the join table's name, I keep a convention that I have since I used Laravel ( alphabetical_order ).

src/db/migrations/create_hashtags_table

import * as Knex from 'knex'

export async function up(knex: Knex): Promise<void> {
  return knex.schema.createTable('hashtags', (t) => {
    t.bigIncrements('id')
    t.string('hashtag').unique()
    t.timestamps(false, true)
  })
}

export async function down(knex: Knex): Promise<void> {
  return knex.raw('DROP TABLE hashtags CASCADE')
}

src/db/migrations/create_hashtags_tweets_table

import * as Knex from 'knex'

export async function up(knex: Knex): Promise<void> {
  return knex.schema.createTable('hashtags_tweets', (t) => {
    t.bigIncrements('id')
    t.integer('hashtag_id').unsigned().notNullable()
    t.integer('tweet_id').unsigned().notNullable()
    t.timestamps(false, true)

    t.unique(['hashtag_id', 'tweet_id'])

    t.foreign('hashtag_id')
      .references('id')
      .inTable('hashtags')
      .onDelete('CASCADE')
    t.foreign('tweet_id').references('id').inTable('tweets').onDelete('CASCADE')
  })
}

export async function down(knex: Knex): Promise<void> {
  return knex.raw('DROP TABLE hashtags_tweets CASCADE')
}

Then I need to modify the AddTweetPayload:

src/dto/AddTweetPayload

import {
  ArrayUnique,
  IsDefined,
  IsIn,
  IsNotEmpty,
  IsUrl,
  isURL,
  Matches,
  MinLength,
  ValidateIf,
} from 'class-validator'
import { Field, InputType, Int } from 'type-graphql'
import { TweetTypeEnum } from '../entities/Tweet'

@InputType()
class AddTweetPayload {
  @Field()
  @IsNotEmpty()
  @MinLength(2)
  body: string

  @Field(() => [String], { nullable: true })
  @ArrayUnique()
  @Matches(/^#[\w]{2,20}$/, {
    each: true,
    message:
      'Each hashtag should start with a # and have a length betweet 2 and 20 characters',
  })
  hashtags?: string[]

  @Field({ nullable: true })
  @IsUrl()
  url?: string

  @Field(() => Int, { nullable: true })
  @ValidateIf((o) => o.type !== undefined)
  @IsDefined()
  parent_id?: number

  @Field(() => String, { nullable: true })
  @ValidateIf((o) => o.parent_id !== undefined)
  @IsDefined()
  @IsIn([TweetTypeEnum.COMMENT, TweetTypeEnum.RETWEET])
  type?: TweetTypeEnum

  @Field(() => String, { nullable: true })
  visibility?: string
}

export default AddTweetPayload

You can see two new properties ( hashtags and url ). I'll talk about the second one later. For the validation's rules, I just check that the hashtags are unique in the array and that they start with a # followed by alphanumeric characters.

Let's take a look at the TweetResolver

src/resolvers/TweetResolver.ts

if (hashtags && hashtags?.length > 0) {
        const hashTagToInsert = hashtags.map((h) => {
          return {
            hashtag: h,
          }
        })
        try {
          // Insert the hashtags
          const hashTagsIds = await db('hashtags')
            .insert(hashTagToInsert)
            .onConflict('hashtag')
            .merge()
            .returning('id')

          // Insert the relation betweet hashtag and the tweet
          const toInsert = hashTagsIds.map((id) => {
            return {
              hashtag_id: id,
              tweet_id: tweet.id,
            }
          })
          await db('hashtags_tweets').insert(toInsert)
        } catch (e) {
          console.log('e', e)
        }
      }

After my tweet is inserted, I add this bit of code. The onConflict().merge() correspond to an upsert. I don't make use of a transaction as I don't really care if the hashtags are not inserted. That's also why I don't throw any errors if something goes wrong while inserting the hashtags. Maybe I'm wrong but for now, that's ok for me :D

One issue I noticed while working on that part is that I can't insert multiline data in the graphQL editor from the apollo-server library. If you have an idea of how to do that, please tell me ;).

It works with postman tho:

I wrote some tests for the hashtags. I'll let you check that out in the Github repository

Link Preview

I didn't know how to handle that at first. Not from a technical point of view but from a UX point of view. I was sure that the preview was triggered on the frontend but on Twitter it's not the case. I think that on Facebook the preview is triggered when a link is detected. I chose to do it only when the tweet is inserted. But as it could take some time, I decided to use an EventEmitter to not do that in the resolver and block everything. It's certainly not super scalable but for this challenge, it should be enough. Also, I didn't want to add Redis or anything to manage a Queue. So I will stick to this super simple event system ;).

Let's start with the migrations:

src/db/migrations/create_previews_table

import * as Knex from 'knex'

export async function up(knex: Knex): Promise<void> {
  return knex.schema.createTable('previews', (t) => {
    t.bigIncrements('id')
    t.string('url').notNullable().unique()
    t.string('title').notNullable()
    t.string('description')
    t.string('image')
    t.timestamps(false, true)
  })
}

export async function down(knex: Knex): Promise<void> {
  return knex.raw('DROP TABLE previews CASCADE')
}

src/db/migrations/create_previews_tweets_table

import * as Knex from 'knex'

export async function up(knex: Knex): Promise<void> {
  return knex.schema.createTable('previews_tweets', (t) => {
    t.bigIncrements('id')
    t.integer('preview_id').notNullable()
    t.integer('tweet_id').notNullable()
    t.timestamps(false, true)

    t.unique(['preview_id', 'tweet_id'])

    t.foreign('preview_id')
      .references('id')
      .inTable('previews')
      .onDelete('CASCADE')
    t.foreign('tweet_id').references('id').inTable('tweets').onDelete('CASCADE')
  })
}

export async function down(knex: Knex): Promise<void> {
  return knex.raw('DROP TABLE previews_tweets CASCADE')
}

For the scrapping part, I chose puppeteer ( I don't know if it's because I'm French, but I think this is the worst name for a library :D. I never know if it's two "p", two "t" or even where are the two "e"... ;)).

Also as it's a pretty big library, I hope I will not have any issues when deploying to Heroku.

Here is the function for scrapping. I did a super simple thing to start.

src/utils/utils.ts

export const scrap = async (url: string) => {
  const browser = await puppeteer.launch({
    headless: true,
  })
  try {
    const page = await browser.newPage()
    console.log('url', url)
    await page.goto(url)
    const results = await page.evaluate(() => {      
      // @ts-ignore
      const title = document
        .querySelector("meta[property='og:title']")
        .getAttribute('content')
      // @ts-ignore
      const image = document
        .querySelector("meta[property='og:image']")
        .getAttribute('content')
      // @ts-ignore
      const description = document
        .querySelector("meta[property='og:description']")
        .getAttribute('content')
      // @ts-ignore
      const url = document
        .querySelector("meta[property='og:url']")
        .getAttribute('content')

      return {
        title,
        image,
        description,
        url,
      }
    })

    return results
  } catch (e) {
    console.log('e', e)
  } finally {
    browser.close()
  }
}

I've had some typescript errors and I didn't figure out how to fix them. That's why you can see some *// @ts-ignore *. Otherwise, it's a pretty basic example of puppeteer. I just look for og meta tags to get the title, description, image, and the url.

For the EventEmitter part:

src/events/scrapPreviewEmitter.ts

import { EventEmitter } from 'events'
import { scrap } from '../utils/utils'
import knex from '../db/connection'

const scrapPreviewEmitter = new EventEmitter()

scrapPreviewEmitter.on('scrap', async (url: string, tweet_id: number) => {
  try {
    const result = await scrap(url)
    const previewsIds = await knex('previews')
      .insert(result)
      .onConflict('url')
      .merge({
        title: result?.title,
        description: result?.description,
        image: result?.image,
        updated_at: knex.raw('NOW()'),
      })
      .returning('id')

    const toInsert = previewsIds.map((id) => {
      return {
        preview_id: id,
        tweet_id: tweet_id,
      }
    })

    await knex('previews_tweets').insert(toInsert)
  } catch (e) {
    console.log('e', e)
  }
})

export default scrapPreviewEmitter

Not convinced about my naming :D. It's also a many-to-many relationship so nothing new here. If you have any questions, feel free to let me a comment or contact me on Twitter ;). I'll be happy to help if I can.

I will then pass this emitter to the context.

src/server.ts

export const defaultContext = ({ req, res }: any) => {
  return {
    req,
    res,
    db,
    dataloaders,
    bus: scrapPreviewEmitter,
  }
}

To finish, I just need to send the event from my TweetResolver

src/resolvers/TweetResolver.ts

// Send the event to scrap the preview
if (url) {
    bus.emit('scrap', url, tweet.id)
}

And that's it!!! :D

I think we can finally create a form and try to send some tweets from the frontend. But we'll see that in the next part ;).

In the meantime, take care ;)