🎅 🎄 Twelve Days of SMT 🎄 🎅

Kafka Connect’s Single Message Transform functionality is really useful for building data pipelines that modify the data passing through. For more complex transformations (such as those requiring state, or joins, etc) check out Kafka Streams or ksqlDB.

Each of the tutorials below has the full code available for you to try using the provided Docker Compose, and a complete 🎥 recording to go with it.

Day 1 - InsertField - add message timestamp as a field - a sink
Day 2 - ValueToKey and ExtractField - set the message key - a field from the value
Day 3 - Flatten - turn a nested structure into a flat one
Day 4 - RegexRouter - change the topic name based on a pattern match and replacement
Day 5 - MaskField - mask the value of fields with a fixed replacement string
Day 6 - InsertField - same SMT as Day 1, this time showing adding to the payload the topic name, Kafka message partition and offset, as well as hardcoded values
Day 7 - TimestampRouter - change the topic name based on the timestamp of the Kafka message
Day 8 - TimestampConverter - Convert timestamp fields between unix epoch, string, Timestamp, Date, and Time
Day 9 - Cast - Convert string, numeric, and boolean field types
Day 10 - ReplaceField - Drop or rename fields
Day 11 - Filter and predicates - Drop messages based on conditions
Day 12 - Community Transformations (ChangeTopicCase, ExtractTimestamp, TimestampNowField)

🎥 Check out the recordings for all of these in this YouTube playlist