Interact Triggered Message Integration with Apache Kafka

Unica Interact Triggered Message capability is a popular module that is used by most of our customers. When a Triggered Message rule is activated, the corresponding Marketing message is sent to Dispatcher, which is sent to an outbound gateway.

Similarly, in the case of an Inbound Gateway, the messages arrive on the receiver queue of the activity Orchestrator. Starting Unica 12.0, Interact has introduced ‘Kafka’ as an option that can be used as a message queue for Dispatchers and Receivers in Interact gateways. Apache Kafka is introduced as a replacement for IBM MQ, which acts as a communication channel between Interact and third-party systems.

Let’s see in detail about the Kafka configurations in Interact.

Kafka Dispatcher

For configuring an outbound gateway, you need to first create a dispatcher. This dispatcher can be of type: InMemoryQueue, JMSQueue, Custom, or Kafka.

When the type is Kafka, the following parameters are mandatory.

providerUrl: A list of host/port pairs to be used for establishing the initial connection to the Kafka cluster. This list must be in the form of host1:port1,host2:port2,…..
E.g., The value can be “server1:9092,” where ‘server1’ is the name of the Kafka server.

topic: This is the topic name used in the Kafka server. A topic is a category or feed name to which messages are stored and published. All Kafka messages are organized into topics.
Topic name must contain an ASCII alphanumeric, ‘.’, ‘_’ and ‘-‘ characters. Due to the limitations in topic names, you can either use topics with a period (‘.’) or underscore (‘_’). The maximum length of a topic name can be 255 characters.

zookeeper.connect: Specifies the zookeeper connection string in the form of hostname: port, where hostname and port are the host and port of a ZooKeeper server.
E.g., The value can be “server1:2181” where ‘server1’ is the name of the zookeeper server.

authentication: Users can use Kafka by enabling different authentication mechanisms.
By default, the Kafka server does not support any authentication mechanism. Users can start the Kafka server considering that the authentication mechanism is disabled. In this case, users can set the “authentication” parameter with the value “None.”
Supported values are none, Plain, SSL, SASL_SSL

1. None: If this value is set for ‘authentication,’ then no additional parameters are needed.

2. Plain: If this value is set for ‘authentication,’ then the below parameters are needed.

You can either use asmUserforMQAuth/authDS or username/password parameters for authentication. If authDS is used, then you need to create the data source (authDS) in the User section in the platform configuration.

3. SSL: If this value is set for ‘authentication,’ then the below parameters are needed.

You need to create the data sources (SSLKeystoreDS, SSLKeyDS, and SSLTruststoreDS) in the User section in the platform configuration. Provide the username and password for these data sources as per your environment.

4. SASL_SSL: If this value is set for ‘authentication,’ then the below parameters are needed.

You need to create the data sources (authDS, SSLKeystoreDS, SSLKeyDS, and SSLTruststoreDS) in the User section in platform configuration. You can either use asmUserforMQAuth/authDS or username/password parameters for authentication.

In addition to the above parameters, Interact also supports additional parameters mentioned below for Dispatchers, which are completely optional.

• acks: The acks config controls the criteria under which requests are considered complete. The “all” setting results in blocking the full commit of the record.

• retries: If the request fails, the producer can retry. Since the specified retries are set as 0, retry is not possible. Enabling retries can lead to duplicates. 

• batch.size: The default batch size is in bytes when multiple records are batched and sent to a partition. 

• linger.ms: The producer waits till the given delay time to allow other records to be sent so that the sent records can be batched together.

• buffer.memory: The total bytes of memory that the producer can use to buffer records, which are waiting to be sent to the server.

The following table provides the Default and Sample values for the above parameters.

Kafka Receiver

For configuring an inbound gateway, you need to first create a receiver. This receiver can be of type: Custom or Kafka. When the type is Kafka, the following parameters are mandatory.

• providerUrl: A list of host/port pairs to be used for establishing the initial connection to the Kafka cluster. This list must be in the form of host1:port1,host2:port2,…

E.g., The value can be “server1:9092,” where ‘server1’ is the name of the Kafka server.

• topic: This is the topic name used in the Kafka server.  The rest of the details are similar to those mentioned in the ‘Dispatcher’ section above.

Please keep the dispatcher and receiver topics separate.

• group.id: Specifies the name of the consumer group to which a Kafka consumer belongs.

• zookeeper.connect: Specifies the zookeeper connection string in the form of hostname:port, where hostname and port are the host and port of a ZooKeeper server. 

E.g., The value can be “server1:2181” where ‘server1’ is the name of the zookeeper server.

• authentication: Users can use Kafka by enabling different authentication mechanisms.

                     Supported values – none, Plain, SSL, SASL_SSL.

Details for these authentication types are similar to those mentioned above in the case of dispatchers.

In addition to the above parameters, Interact also supports some more parameters mentioned below for Receivers, which are completely optional.

• • enable.auto.commit means that offsets are committed automatically with a frequency controlled by the config “auto.commit.interval.ms”. 
  • auto.commit.interval.ms: The value of auto.commit.interval.ms must not exceed 1000 as the poll interval is set to 1000. The value of auto.commit.interval.ms must not exceed the value of poll interval.

The following table provides the Default and Sample values for the above parameters.

Also, there are optional parameters for supporting variations in Thread Management, Zookeeper, and topic creation. Once the Dispatcher of a Receiver is configured as mentioned above, then the same can be used in an outbound or inbound gateway and subsequent channel.

To learn more about Interact triggered messages, you can reach out to us and we will be happy to help.