Fetches messages from Apache Kafka
Tags: Kafka, Apache, Get, Ingest, Ingress, Topic, PubSub
Properties:
In the list below, the names of required properties appear in bold. Anyother properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language (or simply EL), and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
| Name | Description | Default Value | Valid Values | EL | Sensitive |
|---|---|---|---|---|---|
| ZooKeeper Connection String | The Connection String to use in order to connect to ZooKeeper. This is often a comma-separated list of <host>:<port> combinations. For example, host1:2181,host2:2181,host3:2188 | No | No | ||
| Topic Name | The Kafka Topic to pull messages from | No | No | ||
| Zookeeper Commit Frequency | Specifies how often to communicate with ZooKeeper to indicate which messages have been pulled. A longer time period will result in better overall performance but can result in more data duplication if a NiFi node is lost | 60 secs | No | No | |
| Batch Size | Specifies the maximum number of messages to combine into a single FlowFile. These messages will be concatenated together with the <Message Demarcator> string placed between the content of each message. If the messages from Kafka should not be concatenated together, leave this value at 1. | 1 | No | No | |
| Message Demarcator | Specifies the characters to use in order to demarcate multiple messages from Kafka. If the <Batch Size> property is set to 1, this value is ignored. Otherwise, for each two subsequent messages in the batch, this value will be placed in between them. | \n | No | No | |
| Client Name | Client Name to use when communicating with Kafka | NiFi-null | No | No | |
| Kafka Communications Timeout | The amount of time to wait for a response from Kafka before determining that there is a communications error | 30 secs | No | No | |
| ZooKeeper Communications Timeout | The amount of time to wait for a response from ZooKeeper before determining that there is a communications error | 30 secs | No | No |
Relationships:
| Name | Description |
|---|---|
| success | All FlowFiles that are created are routed to this relationship |