////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
:description: The MongoDB Input transform enables you to retrieve documents or records from a collection within MongoDB.

= image:transforms/icons/mongodb-input.svg[MongoDB Input transform Icon, role="image-doc-icon"] MongoDB Input

[%noheader,cols="3a,1a", role="table-no-borders" ]
|===
|
== Description

The MongoDB Input transform enables you to retrieve http://docs.mongodb.org/manual/reference/glossary/[documents] or http://docs.mongodb.org/manual/reference/glossary/[records] from a collection within MongoDB.

For additional information about MongoDB, see the MongoDB http://www.mongodb.org/[documentation].

|
== Supported Engines
[%noheader,cols="2,1a",frame=none, role="table-supported-engines"]
!===
!Hop Engine! image:check_mark.svg[Supported, 24]
!Spark! image:check_mark.svg[Supported, 24]
!Flink! image:check_mark.svg[Supported, 24]
!Dataflow! image:check_mark.svg[Supported, 24]
!===
|===

== Options

=== General

Transform name : Specify the unique name of the MongoDB Input transform in the pipeline.
Preview button: Display the rows generated by this transform.
Enter the maximum number of records that you want to preview, then click OK.
The preview data appears in the Examine preview data window.

=== Input options tab

The Input options tab enables you to specify which connection and collection you want to retrieve information from.

Enter the following information in the Input options fields:

|===
|Option|Definition

|MongoDB Connection
|the xref:metadata-types/mongodb-connection.adoc[ MongoDB connection] to use for this MongoDB Input transform.

|Collection
|Name of the collection to retrieve data from.
Click Get collections to populate the drop-down menu with a list of collections within the database.
|===

=== Query tab

The Query tab enables you to refine read requests.
This tab operates in two different query modes:

* *Query expression* mode (default)
* *Aggregation pipeline specification* mode.

The *Query is aggregation pipeline* option toggles between these two modes.
The Query expression uses MongoDB’s JSON-like query language with query operators to perform https://docs.mongodb.com/manual/reference/operator/query/[query operations].
The *Aggregation pipeline specification* field uses MongoDB’s http://docs.mongodb.org/manual/applications/aggregation/[aggregation framework] to transform and combine documents in a collection.
An aggregation pipeline connects several https://docs.mongodb.com/manual/core/aggregation-pipeline/#pipeline-expressions[pipeline expressions] together, with the output of the previous expression becoming the input for the next.

Enter the following information in the Query fields:

|===
|Fields/Option |Definition

|Query expression (JSON)
|Enter a query expression in this field to limit the output.

|Aggregation pipeline specification (JSON)
|Select the *Query is aggregation pipeline* option to display the *Aggregation pipeline specification (JSON)* field.
Then enter a pipeline expression to perform aggregations or selections.
The method name, including the collection name of the database you selected in the Input Options tab, appears after the label for this field.

|Query is aggregation pipeline
|Select this option to use the aggregation pipeline framework.

|Execute for each row
|Select this option to perform the query on each row of data.

|Fields expression (JSON)
|Enter an argument to control the projection (fields to return) from a query.
If empty, all fields are returned.
This field is only available for query expressions.

|===

=== Fields tab

Use the Fields tab to define properties for exported fields.
The Fields tab operates in two different modes:

1. including all fields in a single JSON field
2. including selected fields in the output.

If you store the output in a single JSON field, you can parse this JSON using the JSON Input transform, or by using a User Defined Java Class transform.

*Note:* All fields in the Fields tab except the Name of JSON output field are inactive when the Output single JSON field is selected.
When the Output single JSON field is not selected, the Name of JSON output field is inactive.

General options:

* *The Get fields button*: Click it to generate a sample set of documents.
You can edit the list of field names, paths, and data type for each field in the sample.
* *Output single JSON field*: Specify that the query results in a single JSON field with the String data type (default).
* *Name of JSON output field*: Specify the field name of containing the JSON output from the server.


Enter the following information in the table if you want to output distinct fields:

|===
|Option| Definition

|#
|The order of this entry in the list.

|Name
|The name of the field based on the value in the Path field.
The name that appears here maps the name of the field as it appears in the pipeline with the field that appears in the MongoDB database.
You can edit the name.

|Path
a|Indicates the JSON path of the field in MongoDB.

If the path shown is an array, you can specify a specific element of the array by passing it the key value in the bracketed part of the array.

For example:

* `$.emails[0]` indicates that you want the result to display the first value in the array.
* To display all array values, use the asterisk as the key, like this `$.email[*]`.
* If the array contains records, and not just strings, you can specify that you want to display the record like this: `$.emails[].sender`.

|Type
|Indicates the data type.

|Indexed values
|Specify a comma-separated list of legal values for String fields.
When you specify values in this field, the Hop indexed data type is applied to the data.
If no values are specified, the String data type is applied.
Usually, you will only need to modify this field if you are using Weka metadata for nominal fields.

|Sample: array min: max index
|Indicates minimum and maximum values for the index in the sampled documents.

|Sample: #occur/#docs
|Indicates how often the field occurs and the number of documents processed.

|Sample: disparate types
|Indicates if different data types populate the same field in the sampled documents.
When several documents are sampled and the same field contain different data types, the Sample: disparate types field is populated with a Y and the Type field displays the String data type.
The Hop type for the field is set to the String data type, for different output value types.

|===

== Examples

The following sections contain examples of query expressions and aggregate pipelines.

=== Query expression

MongoDB allows you to select and filter documents in a collection using specific fields and values.
The http://docs.mongodb.org/manual/reference/mongodb-extended-json/[MongoDB Extended JSON] documentation details how to use queries.
Apache Hop supports only the features discussed on this page.

The following table displays some examples of the syntax and structure of the queries you can use to request data from MongoDB:

|===
|Query expression |Description

|```{ name : "MongoDB" }```
|Queries all values where the name field has a value equal to MongoDB.

|```{ name : { '$regex' : "m.*", '$options' : "i" } }```
|Uses a regular expression to find name fields starting with m, case insensitive.

|```{ name : { '$gt' : "M" } }```
|Searches all strings greater than M.

|```{ name : { '$lte' : "T" } }```
|Searches all strings less than or equal to T.

|```{ name : { '$in' : [ "MongoDB", "MySQL" ] } }```
|Finds all names that are either MongoDB or MySQL (Reference).

|```{ name : { '$nin' : [ "MongoDB", "MySQL" ] } }```
|Finds all names that are not either MongoDB or MySQL, or where the field is not set .

|```{ created_at : { $gte : { $date : "2014-12-31T00:00:00.000Z" } } }```
|Finds all created_at documents that are greater than or equal to the specified UTC date.

|```{ $where : "this.count == 1" }```
|Uses JavaScript to evaluate a condition.

|```{ $query: {}, $orderby: { age : -1 } }```
|Returns all documents in the collection named collection sorted by the age field in descending order.

|===

=== Aggregate pipeline

MongoDB allows you to select and filter documents using the http://docs.mongodb.org/manual/tutorial/aggregation-examples/[aggregation] pipeline framework.
The Aggregation page in the MongoDB documentation provides additional examples of function calls.

The following table displays some examples of the query syntax and structure you can use to request data from MongoDB:

|===
|Query expression |Description

|```{ $match : {state : "FL", city : "ORLANDO" } }, {$sort : {pop : -1 } }```
|Returns all fields from all documents where the state field has a value of FL and the city field has a value of ORLANDO.
The returned documents will be sorted by the pop field in descending order.

|```{ $group : { _id: "$state"} }, { $sort : { _id : 1 } }```
|Returns one field named _id containing the distinct values for state in ascending order.
This is similar to the SQL statement SELECT DISTINCT state AS _id FROM collection ORDER BY state ASC.

|```{ $match : {state : "FL" } }, { $group: {_id: "$city" , pop: { $sum: "$pop" } } }, { $sort: { pop: -1 } }, { $project: {_id : 0, city : "$_id" } }```
|Returns all documents where the state field has a value of FL, aggregates all values of pop for each city, sorts by population descending, and returns one field named city.

|```{ $unwind : "$result" }```
|Peels off the elements of an array individually, and returns one document for each element of the array.

|===
