////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
:description: The De-normalizer transform allows you de-normalize data by looking up key-value pairs, with the option to convert data types in the process.

= image:transforms/icons/denormaliser.svg[Row Denormaliser transform Icon, role="image-doc-icon"] Row Denormaliser

[%noheader,cols="3a,1a", role="table-no-borders" ]
|===
|
== Description

The De-normalizer transform allows you de-normalize data by looking up key-value pairs, with the option to convert data types in the process.

Note: make sure to check the notes on this transform in the xref:pipeline/beam/getting-started-with-beam.adoc#_unsupported_transforms[Getting started with Beam] documentation.

|
== Supported Engines
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
!Spark! image:question_mark.svg[Maybe Supported, 24]
!Flink! image:question_mark.svg[Maybe Supported, 24]
!Dataflow! image:question_mark.svg[Maybe Supported, 24]
!===
|===

== Options

|===
|Transform name|Name of the transform.
This name has to be unique in a single pipeline.
|Key field|The field that defined the key of the output row.
|Group fields|Specify the fields that make up the grouping here.
|Target fields|Select the fields to de-normalize by specifying the String value for the key field (see above).
Options are provided to convert data types.
Strings are most common as key-value pairs so you must often convert to Integer, Number or Date.
If you get key-value pair collisions (key is not unique for the group specified) specify the aggregation method to use.
|===

== Metadata Injection Support

You can use the Metadata Injection supported fields with ETL Metadata Injection transform to pass metadata to your pipeline at runtime.
All fields can be injected, the values used for the aggregation field are the following

|===
|key|value
|TYPE_AGGR_NONE| No Aggregation is done
|TYPE_AGGR_SUM| Sum all values
|TYPE_AGGR_AVERAGE| Calculate the average
|TYPE_AGGR_MIN| Take the minimal value of the group
|TYPE_AGGR_MAX| Take the maximum value of the group
|TYPE_AGGR_COUNT_ALL| Count rows
|TYPE_AGGR_CONCAT_COMMA| Aggragate values separated by comma
|===

== Example

=== Input data
The input data must be ordered by the grouping keys (**RecordID** in this example), use a xref:pipeline/transforms/sort.adoc[Sort rows] transform if needed:
[options="header"]
|===
|RecordID|key|value
|345-12-0000|FirstName|Mitchel
|345-12-0000|LastName|Runolfsdottir
|345-12-0000|City|Jerryside
|976-67-7113|FirstName|Elden
|976-67-7113|LastName|Welch
|976-67-7113|City|Lake Jamaal
|824-21-0000|FirstName|Rory
|824-21-0000|LastName|Ledner
|824-21-0000|City|Scottieview
|===

=== Denormalized data
Set **The key field** = "key" and add **RecordID** in **The fields that make up the grouping**.
Compile the **Target fields** table as follows:
[options="header"]
|===
|Target fieldname|Value fieldname|Key value|Type
|FirstName|value|FirstName|String
|LastName|value|LastName|String
|City|value|City|String
|===
The result is:
[options="header"]
|===
|RecordID|FirstName|LastName|City
|345-12-0000|Mitchel|Runolfsdottir|Jerryside
|976-67-7113|Elden|Welch|Lake Jamaal
|824-21-0000|Rory|Ledner|Scottieview
|===
