////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
:description: The Kafka Producer transform allows you to publish messages in near-real-time across worker nodes where multiple, subscribed members have access.

= image:transforms/icons/KafkaProducerOutput.svg[Kafka Producer transform Icon, role="image-doc-icon"] Kafka Producer

[%noheader,cols="3a,1a", role="table-no-borders" ]
|===
|
== Description

The Kafka Producer transform allows you to publish messages in near-real-time across worker nodes where multiple, subscribed members have access.

A Kafka Producer transform publishes a stream of records to one Kafka topic.

|
== Supported Engines
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
!Spark! image:question_mark.svg[Maybe Supported, 24]
!Flink! image:question_mark.svg[Maybe Supported, 24]
!Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===

== Options

[options="header"]
|===
|Option|Description

|Transform name
|the name for this transform

|Bootstrap servers
|comma separated list of bootstrap servers in a Kafka cluster

|Client ID
|The unique Client identifier, used to identify and set up a durable connection path to the server to make requests and to distinguish between different clients.

|Topic
|The category to which records are published.

|Key Field
|In Kafka, all messages can be keyed, allowing for messages to be distributed to partitions based on their keys in a default routing scheme.
If no key is present, messages are randomly distributed to partitions.

|Message Field
|The individual record contained in a topic.

|===

=== Options

Use this tab to configure the property formats of the Kafka consumer broker sources.
A few of the most common property formats have been included for your convenience.
You can enter any desired Kafka property.
For further information on these input names, see the Apache Kafka documentation site: https://kafka.apache.org/documentation/.

The options that are included by default are:

[options="header"]
|===
|Option|Value
|auto.offset.reset|latest
|ssl.key.password|
|ssl.keystore.location|
|ssl.keystore.password|
|ssl.truststore.location|
|ssl.truststore.password|
|===

=== Avro and Schema registry

Here are some options you need to send Avro Record values to a Kafka server.
The schema of Avro values are not sent to Kafka but to a schema registry.  As such you need to have one available.
Here are some options you need to set to make this work on a Confluent Cloud Kafka instance.  There are various parts of the software stack that need authentication, hence the bit of redundancy.   We recommend that you put these options in variables in your environment configuration file.

[options="header"]
|===
|Option|Example

|schema.registry.url
|https://abcd-12345x.europe-west3.gcp.confluent.cloud

|value.converter.schema.registry.url
|https://abcd-12345x.europe-west3.gcp.confluent.cloud

|auto.register.schemas
|true

|security.protocol
|SASL_SSL

|sasl.jaas.config
|org.apache.kafka.common.security.plain.PlainLoginModule required username="CLUSTER_API_KEY" password="CLUSTER_API_SECRET";

|username
|CLUSTER_API_KEY

|password
|CLUSTER_API_SECRET

|sasl.mechanism
|PLAIN

|client.dns.lookup
|use_all_dns_ips

|acks
|ALL

|basic.auth.credentials.source
|USER_INFO

|basic.auth.user.info
|CLUSTER_API_KEY:CLUSTER_API_SECRET

|schema.registry.basic.auth.user.info
|SCHEMA_REGISTRY_API_KEY:SCHEMA_REGISTRY_API_SECRET

|===
