To ensure that duplicate messages are not processed and no messages are skipped, ksqlDB’s persistent queries must be configured to provide exactly-once processing guarantees. This is achieved by setting the processing.guarantee property to exactly_once. This configuration ensures that messages are neither lost nor processed more than once in the event of failures or restarts.
As per the official Confluent documentation:
“ksqlDB persistent queries can be configured for either at-least-once or exactly-once processing semantics by setting the processing.guarantee configuration property.”
“When processing.guarantee is set to exactly_once, ksqlDB ensures that messages are processed only once, even in failure scenarios. This eliminates duplicates and guarantees strong message delivery semantics.”
Other options:
Option B and C (ksql.streams.auto.offset.reset) only control the starting offset for new consumers, and do not enforce exactly-once semantics.
Option D (ksql.fail.on.production.error) controls whether errors during message production should fail the query, and does not affect exactly-once processing.
Page Reference:
Confluent Kafka: The Definitive Guide, 1st Edition, Chapter 11 (Stream Processing with ksqlDB), p. 365.
Chosen Answer:
This is a voting comment (?). You can switch to a simple comment. It is better to Upvote an existing comment if you don't have anything to add.
Submit