////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
////
:description: First, we need to initialize the Hop API. This means we load configuration details, search for and load plugins and so on.
:toc:

= The Hop SDK

== Initializing

First, we need to initialize the Hop API.
This means we load configuration details, search for and load plugins and so on.

You can do this with:

[source,java]
----
HopEnvironment.init();
----

== Plugin folders

If you need to load plugins from different folders you can set system property `HOP_PLUGIN_BASE_FOLDERS`.
The default value is `plugins`, the relative path to the installation folder plugins folder.
You can add your own plugin folders with commas (`,`) separating the values.

== Hop Metadata Providers

Shared metadata in Hop is handled by a HopMetadataProvider.
When using the Hop GUI you store it in a project in the ```metadata/``` folder.
In that case you can point to such a folder with class ```JsonMetadataProvider```.
Note that you can serialize such a metadata collection into a single JSON string using ```SerializableMetadataProvider```.
This can be used to send metadata to remote servers and locations.

When you have a provider you can ask it to give you a serializer with which you can add and retrieve all sorts of metadata objects.

== Variables

If you want to work with variables in your pipelines and workflows it makes sense to create a top level IVariables object.
The easiest way to do this is with:

[source,java]
----
IVariables variables = Variables.getADefaultVariableSpace();
----

This method also takes into account variables which are configured in hop-config.json (if that file can be found);

== Pipelines

=== Loading pipeline metadata from a file

You can get the pipeline metadata from a `.hpl` XML file using:

[source,java]
----
PipelineMeta pipelineMeta = new PipelineMeta(
  "path-to-your-filename.hpl",   // The filename
  metadataProvider,             // See above
  true,                        // set internal variables
  variables                   // see above
);
----

=== Loading pipeline metadata from an input stream

[source,java]
----
PipelineMeta pipelineMeta = new PipelineMeta(
  inputStream,         // The stream to load from
  metadataProvider,   // See above
  true,              // set internal variables
  variables         // see above
);
----

=== Construct pipeline metadata with the Hop API

Obviously you can start with an empty pipeline and add the transforms, hops you like:

[source,java]
----
PipelineMeta pipelineMeta = new PipelineMeta();

// Generate 1M empty rows
//
RowGeneratorMeta rowGeneratorMeta = new RowGeneratorMeta();
rowGeneratorMeta.setRowLimit("1000000");

TransformMeta rowGenerator = new TransformMeta("1M", rowGeneratorMeta);
rowGenerator.setLocation(50, 50);
pipelineMeta.addTransform(rowGenerator);

// Just a dummy placeholder for testing
//
DummyMeta dummyMeta = new DummyMeta();
TransformMeta dummy = new TransformMeta("Output", dummyMeta);
dummy.setLocation(250, 50);
pipelineMeta.addTransform(dummy);

// Add a hop between both
//
PipelineHopMeta generatorDummyHop = new PipelineHopMeta(rowGenerator, dummy);
pipelineMeta.addPipelineHop(generatorDummyHop);

----

=== Pipeline execution

The way a pipeline is executed depends on the run configuration you specify.
To make it easy to get the engine we created a factory for you:

[source,java]
----
IPipelineEngine pipelineEngine = PipelineEngineFactory.createPipelineEngine(
  variables,           // see above
  "local",            // The name of the run configuration defined in the metadata
  metadataProvider,  // The metadata provider to resolve the run configuration details
  pipelineMeta      // The pipeline metadata
);

// We can now simply execute this engine...
//
pipelineEngine.execute();

// This execution runs in the background but we can wait for it to finish:
//
pipelineEngine.waitUnitlFinished();

// When it's done we can evalute the results:
//
Result result = pipelineEngine.getResult();

----

=== Injecting data into a pipeline

You can only inject data into a `LocalPipelineEngine`.
Do so using the `addRowProducer`.
Call this method after your run `prepareExecution()` so that the row producer can be attached to the correct transform copy.
After starting the execution of the pipeline you can then use the `RowProducer` to put rows into the pipeline using `putRow()`.
Make sure to call `setFinished()` when you're done feeding rows into the pipeline.

=== Retrieving rows from a pipeline

This again is only supported on the local pipeline engine `LocalPipelineEngine`.
After `prepareExecution()` you can add row listeners to the various transforms:

[source,java]
----
ITransform transform = localPipeline.getTransform("transform-name", 0);
transform.addRowListener(new RowAdapter() {
  void rowWrittenEvent( IRowMeta rowMeta, Object[] row ) throws HopTransformException {
    // A row was written during execution
  }
});
----

== Workflows

=== Loading workflow metadata from a file

You can get the workflow metadata from a `.hwf` XML file using:

[source,java]
----
WorkflowMeta workflowMeta = new WorkflowMeta(
  variables,                     // see above
  "path-to-your-filename.hwf",  // The filename
  metadataProvider             // See above
);
----

=== Loading workflow metadata from an input stream

[source,java]
----
WorkflowMeta workflowMeta = new WorkflowMeta(
  inputStream,                   // the inputstream to read the metadata from
  metadataProvider,             // See above
  variables                    // see above
);
----

=== Construct workflow metadata with the Hop API

You typically start with an empty workflow and then add the actions and hops you want:

[source,java]
----
WorkflowMeta workflowMeta = new WorkflowMeta();

// Add the Start action
//
ActionStart actionStart = new ActionStart("Start");
ActionMeta startMeta = new ActionMeta(actionStart);
startMeta.setLocation(50, 50);
workflowMeta.addAction(startMeta);

// Just a dummy placeholder for testing
//
ActionDummy actionDummy = new ActionDummy("Dummy");
ActionMeta dummyMeta = new ActionMeta(dummyMeta);
dummyMeta.setLocation(250, 50);
workflowMeta.addAction(dummyMeta);

// Add a hop between both
//
WorkflowHopMeta startDummyHop = new WorkflowHopMeta(startMeta, dummyMeta);
workflowMeta.addWorkflowHop(generatorDummyHop);

----

=== Workflow execution

Workflow engines are also plugins.
Which plugin is used to execute your workflow metadata is specified in a xref:manual::workflow/workflow-run-configurations/workflow-run-configurations.adoc[Workflow Run Configuration].

To make it easy to get the engine we created a factory for you:

[source,java]
----
IWorkflowEngine workflowEngine = WorkflowEngineFactory.createWorkflowEngine(
  variables,           // see above
  "local",            // The name of the run configuration defined in the metadata
  metadataProvider,  // The metadata provider to resolve the run configuration details
  workflowMeta,     // The workflow metadata
  parentLogging    // The parent logging object
);

// We can now execute this engine...
// This execution does not run in the background.
// When you get the result, the execution has completed.
//
Result result = workflowEngine.startExecution();

----

## Logging

### Logging Registry

Everything that executes something worth our time is registering its own Log Channel in the hop `LoggingRegistry`.
Every log channel gets its own unique ID with which we can see where a log line came from.
You can access the Logging Registry using `LoggingRegistry.getInstance()`.
It contains the execution hierarchy of Hop work.
For example if you have the log channel ID of a parent you can see all its children with `getLogChannelChildren()` which will give you all the IDs of the child log channels.
What we get is in effect the execution lineage.

## Log lines

Whenever a log channel logs something using `logBasic()` or other logging variants, that text along with some basic information is kept in the Hop Log Store.
You can get that one with `HopLogStore.getInstance()`.
The logging lines are kept as logging events or class `HopLoggingEvent` in a logging buffer `LoggingBuffer` which you can access using `HopLogStore.getInstance().getAppender()`.

If you want to grab the logging output of a pipeline, transform, workflow, action, ... you need to start with the ID of the log channel associated with that runtime object.
Usually you can do `getLogChannel()` and then get the ID or the shortcut: `getLogChannelId()`.

If you want to get detailed information about every logging event you can ask for a list with:

[source,java]
----
int lastNr = HopLogStore.getLastBufferLineNr();
List<HopLoggingEvent> events = getLogBufferFromTo( logChannelId, false, 0, lastNr);
----

The details allow you to see which line was an error, what the timestamp was, to which executable it belonged and so on.

If you just want to see the flattened logging text you can ask the appender for the information:

[source,java]
----
StringBuffer loggingText = HopLogStore.getAppender().getBuffer(logChannelId);
----

