Fetches messages from Apache Kafka
Kafka, Apache, Get, Ingest, Ingress, Topic, PubSub
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
| Name | Default Value | Valid Values | Description |
|---|---|---|---|
| ZooKeeper Connection String | The Connection String to use in order to connect to ZooKeeper. This is often a comma-separated list of <host>:<port> combinations. For example, host1:2181,host2:2181,host3:2188 | ||
| Topic Name | The Kafka Topic to pull messages from | ||
| Zookeeper Commit Frequency | 60 secs | Specifies how often to communicate with ZooKeeper to indicate which messages have been pulled. A longer time period will result in better overall performance but can result in more data duplication if a NiFi node is lost | |
| Batch Size | 1 | Specifies the maximum number of messages to combine into a single FlowFile. These messages will be concatenated together with the <Message Demarcator> string placed between the content of each message. If the messages from Kafka should not be concatenated together, leave this value at 1. | |
| Message Demarcator | \n | Specifies the characters to use in order to demarcate multiple messages from Kafka. If the <Batch Size> property is set to 1, this value is ignored. Otherwise, for each two subsequent messages in the batch, this value will be placed in between them. | |
| Client Name | NiFi- | Client Name to use when communicating with Kafka | |
| Kafka Communications Timeout | 30 secs | The amount of time to wait for a response from Kafka before determining that there is a communications error | |
| ZooKeeper Communications Timeout | 30 secs | The amount of time to wait for a response from ZooKeeper before determining that there is a communications error |
| Name | Description |
|---|---|
| success | All FlowFiles that are created are routed to this relationship |