This part of code is within a bigger logic working with queues that bufferise the data. Indeed the i/o between the file and the program isn't the fastest but it did the job for what I needed at that time.
Maybe there is something to do with:
for scanner.Scan() {
if r.MatchString(scanner.Text()) {
fmt.Println(scanner.Text())
}
}
and send each line to a own thread with go func(). The fmt.Println is slow, so in my code this does something else and you don't really need it as such. If you want to output the result, I'll print the results at the end of the loop.
I ended up speeding this up by reading the entire file at once instead of line by line. I think the issue was my program was constantly going back and forth with the disk, instead of streaming all of the information it needs at once.
Fortunately for me, the files I’m scanning are small, so it was ok to keep them in RAM for my processing.
Hi! Thanks for your comment.
This part of code is within a bigger logic working with queues that bufferise the data. Indeed the i/o between the file and the program isn't the fastest but it did the job for what I needed at that time.
Maybe there is something to do with:
and send each line to a own thread with
go func(). Thefmt.Printlnis slow, so in my code this does something else and you don't really need it as such. If you want to output the result, I'll print the results at the end of the loop.I ended up speeding this up by reading the entire file at once instead of line by line. I think the issue was my program was constantly going back and forth with the disk, instead of streaming all of the information it needs at once.
Fortunately for me, the files I’m scanning are small, so it was ok to keep them in RAM for my processing.
Indeed, you would want to minimize the i/o between your program and the file as those are costly in memory. Good that you found a solution! :)