Nice work here and as you said it's not as easy as it looks.
You could use Golang which is superb for its concurrency, each go routine only takes up 2 kilobytes!
I am also taking on this type of project in my spare time and at the planning stages at the moment.
I am using Golang for the processing of queues and actually sending out HTTP requests / pings etc and Laravel for the rest.
Question Time
I am stuck at one thing, the part of the process where the system fetches checks to be processed on the queue etc, what if you have 5 million checks to be processed?
You cannot fetch all checks at once, you would run out of memory.
You could batch the checks, ie. fetch 200 checks at a time from the database, with this approach let's say it takes 0.1s to fetch 200 checks and push them to the queue.
That would be 5,000,000 (checks) / 200 (batch) * 0.1 (seconds) = 2500 (seconds)
2500 / 60 = 41 minutes...
How is this scale-able and have you hit such an issue yet?
Thanks
Ben
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Hi Viktor,
Nice work here and as you said it's not as easy as it looks.
You could use Golang which is superb for its concurrency, each go routine only takes up 2 kilobytes!
I am also taking on this type of project in my spare time and at the planning stages at the moment.
I am using Golang for the processing of queues and actually sending out HTTP requests / pings etc and Laravel for the rest.
Question Time
I am stuck at one thing, the part of the process where the system fetches checks to be processed on the queue etc, what if you have 5 million checks to be processed?
You cannot fetch all checks at once, you would run out of memory.
You could batch the checks, ie. fetch 200 checks at a time from the database, with this approach let's say it takes 0.1s to fetch 200 checks and push them to the queue.
That would be 5,000,000 (checks) / 200 (batch) * 0.1 (seconds) = 2500 (seconds)
2500 / 60 = 41 minutes...
How is this scale-able and have you hit such an issue yet?
Thanks
Ben