DEV Community

loading...

Discussion on: Handle backpressure between Kafka and a database with Vert.x

Collapse
kriska profile image
Kristina Gocheva

Great article! You've given really easy to understand example. However, I was wondering something I didn't see you mention. In the first case, when you don't use the polling principle upon failing you see which record exactly has failed. Is it true that with the polling you can only get the batch of records in which a failing record appeared and not the single record itself? And another question, could you explain what would happen if for example the 3 of 10 records in the batch fails? Does this fail that the whole batch of 10 records?

Collapse
ndrbrt_23 profile image
ndr_brt Author

with the polling you can get the single record by setting the config max.poll.records to 1, but it's not good for performances.
With polling you get the a batch of messages, but you can process it one at a time, so, it's possible to know which is failing, the advice is to avoid commit when at least one message fails.
So yes, with this approach one or more fail = all batch fails.

Forem Open with the Forem app