Hello and thank you for checking out my DEV x Redis Hackathon submission!
If you haven't had a chance, take a look at Part 1 of this series for additional context.
Overview of My Submission
For my submission, I made meatballs.live, an automated recommendation network and web app for discovering interesting conversations across social news, powered by the Redis Stack and Hacker News.
When it comes to social news, I find the best insights from deep within the comments. However, this is extremely time consuming and I'll sometimes regress to mindless scrolling.
I built meatballs.live to generate and archive daily digests (called collections) from social news platform APIs. These daily collections serve as recommendations to the most engaging comments and related stories.
Like a printed newspaper or photo strip, these collections are static and immutable. It's nice and calm to travel back to a time that's not constantly scrolling out of context and out of view.
But, not everything in social news happens at once. It is a temporal experience that is difficult to relate without long-term observation and relationship building.
(Did I mention the Live Comments stream that shows you Hacker News comments as they come in and also provides a place to chat with other meatballs.live readers?!)
Redis Stack and the Hacker News API were perfect for this challenge.
I had three primary design goals:
- Learn and utilize as many multi-model capabilities of the Redis Stack as possible
- Leverage the Redis Stack to consume, process, sift and generate insights from big data
- Combine 1 and 2 to create a fast and compelling user experience for desktop and mobile
To meet the challenge, meatballs.live is three connected, open-source projects:
- APP / UI
- APIs and a snappy web app made with TypeScript and Next.js that only consumes cache and socket-based, real-time data (this project)
- JOBS SERVER
- node server called meatballs.live-jobs (casper) responsible for making scheduled, weighted calls to the APP's ingest service APIs
- STREAM SERVER
- node server called meatballs.live-stream (kodama) responsible for processing Redis pub/sub events via web sockets
Architecture
The JOBS SERVER (casper) calls the APP's services APIs to ingest, process and route source data from Hacker News (Official | Algolia) to the primary cloud Redis database.
Ingest data is saved to JSON, graph, time series and cache (see details in the following sections).
The APP's front end UI connects to the STREAM SERVER (kodama) via web sockets (Socket.io). The STREAM SERVER subscribes to Redis channels for presence, new comments and frontpage stats and emits events to connected APP clients. The STREAM SERVER also publishes to the presence channel as clients connect and disconnect.
On new story activity, the JOBS SERVER publishes an update to the Frontpage stats channel, which is sent as an event from the STREAM SERVER to connected APP clients. On new comments from an APP user, the APP front end calls the protected publish-comments API to publish to the new comments channels.
APP users must be signed in to send new comments. For authentication, the APP uses a combination of NextAuth, GitHub and Upstash. Yes, even auth on the backend is persisted to a Redis database!
After at least 24 hours of ingest data, the APP's services API for generating collections can be called. The generate processor uses a combination of time series, graph and JSON queries and search to create JSON->string caches for the day (9 stories) and each individual story collection.
Here's a live example:
- Collections page: https://www.meatballs.live/c/2022/8/27/
- Collection page: https://www.meatballs.live/c/2022/8/27/ask-hn-how-can-i-come-to-peace-with-the-years-i-wasted-on-pointless-things-9o2r6s3o/
When accessing a collections or collection page, the APP leverages getServerSideProps to retrieve and parse the respective cache data, which is then sent to the user.
Hosted by Redis Cloud and Vercel, story collections and collection page renders are between 1-5 milliseconds!
For full details (data models, command examples, local install guide), I recommend taking a look at the lengthy project README.
Here's a sneak peak of the graph model(s)!
Submission Category
Wacky Wildcards
Language Used
TS/Node.js
Link to Code
meatballs.live is 3 open-source (MIT) projects. The README includes a local installation guide to configure and connect these projects.
meatballs.live APP / UI
ritsuke
/
meatballs.live
remix your social news experience with Redis Stack + Hacker News
meatballs.live
meatballs is an automated recommendation network and web app for discovering interesting conversations across social news.
Live deployment: https://www.meatballs.live
The network currently supports Hacker News, with more integrations planned.
By ingesting, connecting and analyzing large amounts of structured, temporal data, meatballs (and associated services) is capable of weighting and generating daily collections of top stories, with a bias for the comment section.
Overview video
Here's a short video that explains the project and how it uses Redis:
How it works
meatballs was created for the DEV.to x Redis Hackathon, with three primary design goals:
- Learn and utilize as many multi-model capabilities of the Redis Stack as possible
- Leverage the Redis Stack to consume, process, sift and generate insights from big data
- Combine 1 and 2 to create a fast and compelling user experience for desktop and mobile
To meet the challenge, meatballs is three open-source projects:
- *…
meatballs.live JOBS SERVER (casper)
meatballs.live-jobs (JOBS SERVER; casper)
How to run it locally
Prerequisites
Assuming that you've arrived here from the meatballs.live (APP) local installation guide...
Local installation
- Clone this repo
- Run
yarnin the project folder to install dependencies - Copy
.env.sampleto.envand fill in your unique values, following the steps below:
INGEST_API_KEY
-
Generate an API key and copy and paste to your
.envfile; this can be anything, but should remain private
Next, copy and replace the remaining with the following (feel free to tweak):
SOURCE_HN_NEW_STORIES_CRON_TIME=*/5 * * * * *
SOURCE_HN_NEW_STORIES_PAUSED=false
SOURCE_HN_NEW_STORIES_LIMIT=10
SOURCE_HN_STORY_ACTIVITY_0_CRON_TIME=3-59/5 * * * * *
SOURCE_HN_STORY_ACTIVITY_0_GATE_SCORE=
SOURCE_HN_STORY_ACTIVITY_0_GATE_COMMENT_TOTAL=
SOURCE_HN_STORY_ACTIVITY_1_CRON_TIME=5-59/29 * * * * *
SOURCE_HN_STORY_ACTIVITY_1_GATE_SCORE=
SOURCE_HN_STORY_ACTIVITY_1_GATE_COMMENT_TOTAL=
SOURCE_HN_STORY_ACTIVITY_2_CRON_TIME=7 */2 * * * *
SOURCE_HN_STORY_ACTIVITY_2_GATE_SCORE=
SOURCE_HN_STORY_ACTIVITY_2_GATE_COMMENT_TOTAL=
SOURCE_HN_STORY_ACTIVITY_3_CRON_TIME=23 */7 * * * *
SOURCE_HN_STORY_ACTIVITY_3_GATE_SCORE=
SOURCE_HN_STORY_ACTIVITY_3_GATE_COMMENT_TOTAL=
SOURCE_HN_STORY_ACTIVITY_4_CRON_TIME=29 */13 * * * *
SOURCE_HN_STORY_ACTIVITY_4_GATE_SCORE=
SOURCE_HN_STORY_ACTIVITY_4_GATE_COMMENT_TOTAL=
SOURCE_HN_STORY_ACTIVITY_5_CRON_TIME=31 23 */12 * * *
SOURCE_HN_STORY_ACTIVITY_5_GATE_SCORE=
SOURCE_HN_STORY_ACTIVITY_5_GATE_COMMENT_TOTAL=
SOURCE_HN_STORY_ACTIVITY_6_CRON_TIME=37 37 0meatballs.live STREAM SERVER (kodama)
meatballs.live-stream (STREAM SERVER; kodama)
How to run it locally
Prerequisites
Assuming that you've arrived here from the meatballs.live (APP) local installation guide...
Local installation
- Clone this repo
- Run
yarnin the project folder to install dependencies - Copy
.env.sampleto.envand fill in your unique values, following the steps below:
PORT
- Enter the port you would like the STREAM SERVER to use during development
REDIS_DB_URL
- This should mirror the value used by the meatballs.live (APP) development environment
Continue meatballs.live (APP) installation
Run yarn dev to start the development server, though it will be quiet until the APP and JOBS SERVER are running.
Copy the PORT value from your .env file and continue with the APP's local installation guide.
Deployment
Though meatballs uses Railway to automatically deploy the STREAM SERVER (kodama), you can run the server on any host that is node.js-capable.
Additional Resources / Info
meatballs.live wouldn't be possible without the essential efforts of the open-source community.
Dependencies:
- APP / UI package.json
- JOBS SERVER package.json
- STREAM SERVER package.json
Hosting:
- This project uses 2 Redis databases:
- primary and multi-model; hosted by Redis Cloud (JSON, GRAPH, TIME SERIES, PUB/SUB, CACHE)
- auth; hosted by Upstash
- The APP / UI is hosted by Vercel
- JOBS SERVER and STREAM SERVER servers are hosted by Railway
- Check out Redis OM, client libraries for working with Redis as a multi-model database.
- Use RedisInsight to visualize your data in Redis.
- Sign up for a free Redis database.
As with Part 1, I'd like to share some music I've been listening to for inspiration.
Thanks for reading!
Top comments (0)