DEV Community

Discussion on: Evolving Dark's tracing system

richeyryan profile image
Richey Ryan • Edited

More of a question for my own sake but why wouldn't you store this kind of data in some no sql data store separate from your primary user data store?

It would allow for updates and you could still facilitate whatever eviction policy suited you.

Also would using a time-based eviction method help? So if you rename a handler and it ultimately becomes defunct you then evict it 24 hours (or whatever makes sense later) whereas an active handler that lives on will still accumulate logs. It may mean that a user has to build up some logs again if they left a handler over the weekend but ultimately it favours things that are in active use. Obviously very active debugging sessions might create an awful lot of logs so you might need to compensate with a last X approach too

paulbiggar profile image
Paul Biggar Author

Yeah, when I said S3 I really meant "some no sql data store separate from the primary user data store". Some folks have suggested Kafka and BigQuery, which have some very nice properties around this.