First, I love this concept. Second ... you should use BigQuery!!! I think I read somewhere you're already on GCP. BigQuery was built for this. Storage pricing is basically the same as S3 and in some cases automatically a lot less. Use standard SQL for querying. You can actually update records if you like. You can query the streaming buffer for free (probably mostly what your users will need). You could even build it so that people could provide their own GCP credentials and store traces for however long they like (in this case they'd get charged for storage and queries). Partitioned tables can have a TTL so old data is auto pruned. Plus, you get Google Data Studio for free. I built Logflare on top of BigQuery initially for all these reasons and I've had zero regrets so far. The downside is that queries are pretty much never sub-second but they are very rarely above 5 seconds. If you'd like to play with this exact setup check out Logflare.
That's super interesting, thanks! I'll have to think about that as I start to work on it. Subsecond is important, but I could put a cache in between so that things won't feel all that slow. And querying capability would actually be super useful.
Thanks!
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
First, I love this concept. Second ... you should use BigQuery!!! I think I read somewhere you're already on GCP. BigQuery was built for this. Storage pricing is basically the same as S3 and in some cases automatically a lot less. Use standard SQL for querying. You can actually update records if you like. You can query the streaming buffer for free (probably mostly what your users will need). You could even build it so that people could provide their own GCP credentials and store traces for however long they like (in this case they'd get charged for storage and queries). Partitioned tables can have a TTL so old data is auto pruned. Plus, you get Google Data Studio for free. I built Logflare on top of BigQuery initially for all these reasons and I've had zero regrets so far. The downside is that queries are pretty much never sub-second but they are very rarely above 5 seconds. If you'd like to play with this exact setup check out Logflare.
That's super interesting, thanks! I'll have to think about that as I start to work on it. Subsecond is important, but I could put a cache in between so that things won't feel all that slow. And querying capability would actually be super useful.
Thanks!