This Processors polls Apache Kafka
for data. When a message is received from Kafka, this Processor emits a FlowFile
where the content of the FlowFile is the value of the Kafka message. If the
message has a key associated with it, an attribute named kafka.key
will be added to the FlowFile, with the value being the UTF-8 Encoded value
of the Message's Key.
Kafka supports the notion of a Consumer Group when pulling messages in order to provide scalability while still offering a publish-subscribe interface. Each Consumer Group must have a unique identifier. The Consumer Group identifier that is used by NiFi is the UUID of the Processor. This means that all of the nodes within a cluster will use the same Consumer Group Identifier so that they do not receive duplicate data but multiple GetKafka Processors can be used to pull from multiple Topics, as each Processor will receive a different Processor UUID and therefore a different Consumer Group Identifier.
Modifies Attributes:
| Attribute Name | Description |
|---|---|
| kafka.topic | The name of the Kafka Topic from which the message was received |
| kafka.key | The key of the Kafka message, if it exists and batch size is 1. If the message does not have a key, or if the batch size is greater than 1, this attribute will not be added. |
| kafka.partition | The partition of the Kafka Topic from which the message was received. This attribute is added only if the batch size is 1. |
| kafka.offset | The offset of the message within the Kafka partition. This attribute is added only if the batch size is 1. |