Basically I am designing a system, which can have thousand of cron jobs. I am creating a UI where one can create cron job, and specify the interval in it. What system design you will suggest making it scalable?
More Details: My Question is in terms of the backend infrastructure. So I am going to use the package called npmjs.com/package/html-differ to compare two HTML. There will be a user interface where the user will define the webpage URL and interval time where webpage changes will be checked. Let's assume we have 10k URL to check with different time intervals. I wanted to have a recommendation about the database or queuing system like rabbitmq i should use to make it scalable and fault tolerance.
Top comments (6)
What have you thought of so far?
Are you asking for advice on what technology stack to use, or just a specific part of it (e.g. DB)?
What do you have experience in?
I am a full-stack engineer, My Question is in terms of the backend infrastructure. So I am going to use the package called npmjs.com/package/html-differ to compare two HTML. There will be a user interface where the user will define the webpage URL and interval time where webpage changes will be checked. Let's assume we have 10k URL to check with different time intervals. I wanted to have a recommendation about the database or queuing system like rabbitmq i should use to make it scalable and fault tolerance.
You're asking an architectural question, which isn't easily answered without full context of what you're trying to make, what technologies/languages you want you use, or how much your budget is.
I'd also suggest text based communication isn't the right place for such an architectural design conversation.
I am creating an open-source project
Thanks for being so responsive and helping on this
Let me have a think of some questions that will help me understand your needs more, and maybe we can figure something out together 👍
Awesome, I havenot did any commit on this project, but if it do, it will be here github.com/tushariscoolster/Telesto