User Defined Java Class

Description

The User Defined Java Class transform runs Java code you type in the dialog as the implementation of one transform.

It is not a place to build a full plugin. A real plugin is the right tool when the logic is reused, needs its own dialog, or needs more than a class body. This transform is for a small piece of logic that has to run as fast as a normal transform.

The code is compiled when the pipeline starts by the Janino libraries. You do not write a .java file, a package line, an import line, or a class declaration. Each tab is the body of one class: fields and methods only. The tab name is the class name, so it has to be a valid Java identifier. A new transform starts with one tab named Processor.

Janino already imports these packages, so classes from them can be used by their simple name:

org.apache.hop.pipeline.transforms.userdefinedjavaclass
org.apache.hop.pipeline.transform
org.apache.hop.core.row
org.apache.hop.core
org.apache.hop.core.exception
org.apache.hop.pipeline
org.apache.hop.pipeline.engine
org.apache.hop.workflow
org.apache.hop.workflow.action
org.apache.hop.core.plugins
org.apache.hop.core.variables
java.util

Anything else needs a fully qualified name. Put extra jars in plugins/transforms/janino/lib so this transform can load them. See Installation and configuration.

One tab is the transform class. Right-click that tab and choose Set as transform Class. Hop compiles it as a subclass of its own transform base and adds the constructor, so do not write a constructor yourself. That class is the one Hop calls while the pipeline runs. Other tabs are helper classes. They are compiled first, in alphabetical order of the tab name, and the transform class can use them with new. A helper can use another helper only when that other helper’s name sorts first. Choosing OK or Test class with no transform class asks whether the first tab should become one.

Options

Option Description

Option	Description
Transform name	Name of the transform. This name has to be unique in the pipeline.
Java target version	Java language level Janino uses to compile every class tab. Choose a value from 6 to 21. A pipeline that does not store the option, or stores a value outside that range, is compiled as Java 6. The level does not unlock newer language features. Janino accepts generics, the diamond operator, for-each loops, `switch` on strings, and text blocks at every level. It rejects lambdas, method references, `var`, switch expressions, records, pattern matching in `instanceof`, and multi-type `catch` at every level, including 21. Use anonymous classes, explicit types, and plain `switch` statements instead.
Class code	The class body of the selected tab. See The class.
Classes and code fragments	Tree of class tabs, code snippets, and the input, info, and output fields. Double-click or drag an entry to insert it at the cursor. See Classes and code snippets.
Fields	Output fields this transform adds. Each row is one field: Fieldname: name of the output field. Type: Hop data type. See Data types. Length and Precision: see Formatting numbers and dates. Clear the result fields? drops every incoming field from the output layout. Leave it unchecked to keep the incoming fields and append the rows in this table. `createOutputRow()` follows the same choice: it copies the input row when the box is unchecked, and allocates an empty output row when the box is checked.
Parameters	Named values the class reads with `getParameter(tag)`, so field names and other constants are not hard-coded. See Parameters. Tag: the name passed to `getParameter`. Value: the text returned for that tag. Variables in the value are resolved. Description: comment. Not visible to the class.
Info transforms	Extra input hops, separate from the main input that `getRow()` reads. See Info transforms and target transforms. Tag: the name passed to `findInfoRowSet`. Transform: an upstream transform already connected to this one. Description: label shown for that info hop.
Target transforms	Specific output hops the class writes to with `putRowTo`, instead of the ordinary output of `putRow`. See Info transforms and target transforms. Tag: the name passed to `findTargetRowSet`. Transform: a downstream transform already connected to this one. Description: label shown for that target hop.
Test class	Compiles the classes and previews them on generated rows. See Test class.

Transform name

Name of the transform. This name has to be unique in the pipeline.

Java target version

Java language level Janino uses to compile every class tab. Choose a value from 6 to 21. A pipeline that does not store the option, or stores a value outside that range, is compiled as Java 6. The level does not unlock newer language features. Janino accepts generics, the diamond operator, for-each loops, switch on strings, and text blocks at every level. It rejects lambdas, method references, var, switch expressions, records, pattern matching in instanceof, and multi-type catch at every level, including 21. Use anonymous classes, explicit types, and plain switch statements instead.

Class code

The class body of the selected tab. See The class.

Classes and code fragments

Tree of class tabs, code snippets, and the input, info, and output fields. Double-click or drag an entry to insert it at the cursor. See Classes and code snippets.

Fields

Output fields this transform adds. Each row is one field:

Fieldname: name of the output field.
Type: Hop data type. See Data types.
Length and Precision: see Formatting numbers and dates.

Clear the result fields? drops every incoming field from the output layout. Leave it unchecked to keep the incoming fields and append the rows in this table. createOutputRow() follows the same choice: it copies the input row when the box is unchecked, and allocates an empty output row when the box is checked.

Parameters

Named values the class reads with getParameter(tag), so field names and other constants are not hard-coded. See Parameters.

Tag: the name passed to getParameter.
Value: the text returned for that tag. Variables in the value are resolved.
Description: comment. Not visible to the class.

Info transforms

Extra input hops, separate from the main input that getRow() reads. See Info transforms and target transforms.

Tag: the name passed to findInfoRowSet.
Transform: an upstream transform already connected to this one.
Description: label shown for that info hop.

Target transforms

Specific output hops the class writes to with putRowTo, instead of the ordinary output of putRow. See Info transforms and target transforms.

Tag: the name passed to findTargetRowSet.
Transform: a downstream transform already connected to this one.
Description: label shown for that target hop.

Test class

Compiles the classes and previews them on generated rows. See Test class.

The class

Which methods are called

Hop drives the transform class the same way it drives any other transform. Three methods matter:

init()

Called once while the pipeline is preparing to start. Return true when initialization worked, or false to make the pipeline abort. The default implementation initializes the transform. Override it to open files or connections, and call parent.initImpl() (or super.init()) so that still happens.

processRow()

The method that does the work. You have to implement it; the class does not compile without it. Once execution has started, Hop calls it in a tight loop.

Return true to be called again.
When there is nothing left to do, call setOutputDone() and return false.
getRow() blocks until the next row from the main input arrives, and returns null when that input is finished. That null is the usual place to call setOutputDone() and return false.

An exception thrown out of processRow() stops the pipeline: the error is logged, the error count is set, and the transform marks its output done. Catch a bad row yourself and send it to the error hop when the pipeline should keep going. See Error handling.

dispose()

Called once when this transform stops running, after its last processRow() call. Close what init() opened. Call parent.disposeImpl() (or super.dispose()).

initBeforeStart() exists as well and is called once after init(), just before the threads start. Most classes do not need it.

The dialog inserts starter code for processRow(), init(), and dispose() from the Code Snippets tree (Implement processRow, Implement init, Implement dispose).

Example

This class reads two input fields and writes a third. On the Parameters tab, add tags FIRSTNAME_FIELD, LASTNAME_FIELD, and NAME_FIELD with values firstname, lastname, and name. On the Fields tab, add one output field named name, type String. The transform class tab can stay named Processor.

java

String firstNameField;
String lastNameField;
String nameField;

public boolean processRow() throws HopException {
  // Resolve the parameter tags once. The values are the field names.
  //
  if (first) {
    firstNameField = getParameter("FIRSTNAME_FIELD");
    lastNameField = getParameter("LASTNAME_FIELD");
    nameField = getParameter("NAME_FIELD");
    first = false;
  }

  // Read one row from the main input hop. null means that input is finished.
  //
  Object[] r = getRow();
  if (r == null) {
    setOutputDone();
    return false;
  }

  String firstName = get(Fields.In, firstNameField).getString(r);
  String lastName = get(Fields.In, lastNameField).getString(r);

  // Copy the input row and make room for the fields added on the Fields tab.
  // When "Clear the result fields?" is checked this is a new empty row instead,
  // so the input values above must be read from r, not from outputRow.
  //
  Object[] outputRow = createOutputRow(r, data.outputRowMeta.size());
  get(Fields.Out, nameField).setValue(outputRow, firstName + " " + lastName);

  // Send the row to the ordinary output hops.
  //
  putRow(data.outputRowMeta, outputRow);
  return true;
}

Classes and code snippets

The left-hand tree has three kinds of entry.

Classes: One entry per class tab. Double-click a name to show that tab. Right-click a tab for Add new, Add copy, Set as transform Class, and Remove class type. Right-click a class in the tree to rename or delete it.
Code Snippets: Fragments for the methods this class can call, grouped into Common use, Row manipulation, transform logging, transform status, transform/Row listeners, and Uncommon use. Double-click a snippet to insert it, or right-click it and choose Show Sample to open the sample in a read-only tab.
Input fields, Info fields, and Output fields: The fields reaching this transform, the fields on its info hops, and the fields it produces. Double-click a field to insert a get(Fields.In, …) or get(Fields.Out, …) call, or open the field and insert the getter for its data type or a setValue call.

Reading and writing fields

A row is an Object[]. The matching IRowMeta describes the field names, types, and order.

getRow() returns the next main-input row and, on the first call, refreshes data.inputRowMeta. data.outputRowMeta is that layout plus the fields from the Fields tab, or only those fields when Clear the result fields? is checked. getInputRowMeta() returns the same input layout, but only once getRow() has returned a row; before that it is null.

The usual way to read and write a field is get(Fields.In, name), get(Fields.Out, name), or get(Fields.Info, name). Each returns a helper that remembers the field index, so the name is not searched again on every row. Pass the input row to an In getter and the output row to an Out setter.

Method Java type Hop type

Method	Java type	Hop type
`getString`	`String`	String
`getLong`	`Long`	Integer
`getDouble`	`Double`	Number
`getBigDecimal`	`BigDecimal`	BigNumber
`getBoolean`	`Boolean`	Boolean
`getDate`	`java.util.Date`	Date
`getTimestamp`	`java.sql.Timestamp`	Timestamp
`getBinary`	`byte[]`	Binary
`getInetAddress`	`java.net.InetAddress`	Internet Address
`getObject`	`Object`	any, including Serializable

getString

String

String

getLong

Long

Integer

getDouble

Double

Number

getBigDecimal

BigDecimal

BigNumber

getBoolean

Boolean

Boolean

getDate

java.util.Date

Date

getTimestamp

java.sql.Timestamp

Timestamp

getBinary

byte[]

Binary

getInetAddress

java.net.InetAddress

Internet Address

getObject

Object

any, including Serializable

setValue(row, value) writes into that row at the field’s index.

The same lookup by index, which is what the helper caches for you, looks like this:

java

Object[] r = getRow();
if (r == null) {
  setOutputDone();
  return false;
}
// The input layout is only known after getRow() has returned a row.
//
IRowMeta inputRowMeta = getInputRowMeta();
int yearIndex = inputRowMeta.indexOfValue(getParameter("YEAR"));
if (yearIndex < 0) {
  throw new HopException("Year field not found in the input row, check parameter 'YEAR'!");
}
Long year = inputRowMeta.getInteger(r, yearIndex);

get(Fields.In, "year").getLong(r) is the same read. get(Fields.Info, name) uses the field layout of the first info hop Hop finds on this transform, not every info hop. For any other info hop, take the layout from that row set, as in Info transforms and target transforms.

Info transforms and target transforms

Why "info" and not "source" or "input"

An info transform is not another name for the main input, and renaming the tab to Source transforms or Input transforms would hide that.

Hop already calls this kind of hop an info stream. Stream Lookup uses the same idea: one hop is the stream of rows to process, and another hop only supplies data that helps process those rows. In this transform the split is:

Hops that are not listed on Info transforms are the main input. getRow() reads them, and also reads an info hop that has not been drained yet (see below).
Hops that are listed there are info hops. The canvas draws an info icon on the hop. Read them with findInfoRowSet(tag) and getRowFrom(rowSet).

"Input" is already the main hop. "Source" would not say which of the two upstream hops it is.

A target transform is the same idea on the way out. putRow(data.outputRowMeta, row) writes to the ordinary output hops, copied or distributed according to the transform’s hop settings. A hop listed on Target transforms is a specific destination. The canvas marks it with a target icon, and the class selects it with findTargetRowSet(tag) and putRowTo.

Both lists can have several rows

Each row is one transform, and both tables accept as many rows as you need. Give every row its own tag. The class then picks the hop by that tag:

java

public boolean processRow() throws HopException {
  if (first) {
    first = false;

    // Drain every info hop before getRow(). Layouts differ, and getRow()
    // would otherwise mix these rows into the main input.
    //
    IRowSet lookup = findInfoRowSet("lookup");
    Object[] infoRow;
    while ((infoRow = getRowFrom(lookup)) != null) {
      String key = lookup.getRowMeta().getString(infoRow, "key", null);
      // keep what the main rows need from this info hop
    }
  }

  Object[] r = getRow();
  if (r == null) {
    setOutputDone();
    return false;
  }
  boolean accepted = Boolean.TRUE.equals(get(Fields.In, "flag").getBoolean(r));
  r = createOutputRow(r, data.outputRowMeta.size());

  IRowSet target = findTargetRowSet(accepted ? "accepted" : "rejected");
  putRowTo(data.outputRowMeta, r, target);
  return true;
}

getRowFrom removes an info row set from the main input once that hop is exhausted, which is why the loop above has to run first. findInfoRowSet and findTargetRowSet throw when the tag is missing or the hop is not connected.

The transform named in an info or target row has to run as a single copy. findInfoRowSet also accepts an info transform that is partitioned in the same way as this transform. A main input and an info hop that share an upstream transform can stall a local pipeline; see Avoiding deadlocks.

What the tag is

The tag is a name you choose. It is the only identifier the class should use.

The Transform column is the transform’s name on the canvas. The Tag column is a stable alias for that hop, so renaming the transform means editing the table instead of the code. findInfoRowSet("lookup") resolves the tag lookup to the transform name stored on that row, then finds the row set coming from that transform. findTargetRowSet("accepted") does the same for an output row set. The description is only a label, shown on the hop; the class never reads it.

A tag on the Parameters tab is a different map. It does not name a transform. See Parameters.

Why list a transform that is already on the canvas

The hop and the table do different jobs. Drawing the hop creates the pipe. The table tells Hop what kind of pipe it is, and the name the class uses for it.

The Transform drop-down only offers transforms that are already connected: upstream transforms on the info tab, downstream transforms on the target tab. After you pick one, Hop treats that existing hop as an info hop or a target hop. Leave the table empty and every incoming hop stays main input (getRow()) and every outgoing hop stays an ordinary output (putRow()). The table does not replace the hops, and the hops do not record the tag.

Target hops stay in the output list. A row passed to putRow is therefore copied or distributed to them as well as to any ordinary output. Call putRowTo when that row should go to one target and not to every output.

Parameters

getParameter("TAG") returns the Value of the parameter with that tag, with variables resolved, or null when the tag is not in the table. A value of ${YEAR_FIELD} is resolved in the pipeline before the class sees it.

getVariable("NAME") and getVariable("NAME", "default") read pipeline variables. setVariable("NAME", "value") sets one. Those are not the parameter table.

Test class

Test class does not run the pipeline on the canvas, and it does not open a dialog that asks you to type rows.

It checks that a tab is marked as the transform class.
It compiles every class tab the way a real run would. A compile error is shown and the test stops. The message names the forbidden text when a code exclusion matched.
It looks up this transform in the pipeline the dialog was opened from. If the transform is not there, the test stops and reports that the fields from the previous transforms could not be read.
It builds a throwaway pipeline named after this transform with - PREVIEW appended:
- A Generate Rows transform named TEST DATA feeds the main input. It produces 10 rows. The fields are the main input fields of this transform, not the fields of the info transforms. The sample value depends on the type: test value test value for a String, zero for an Integer, Number, or BigNumber, Y or true for a Boolean, the current date and time for a Date, and the bytes of ABCDEFGHIJ for Binary.
- Each info transform is replaced by a Generate Rows transform of the same name, using that transform’s fields, hopped into the class.
- Each target transform is replaced by a Dummy transform of the same name, hopped out of the class. Nothing is written to the real downstream transforms.
It runs that pipeline and previews up to 10 rows written by this class with putRow. Rows sent only with putRowTo go to the dummy target and are not part of that preview. When the run reports errors, the test shows the log as well.

Error handling

The transform supports the standard error hop. Right-click it on the canvas and choose Error Handling, then pick the transform that receives rejected rows and the fields that will carry the error count, description, field name, and code. That hop is not an info transform and not a target transform. putError fails when this has not been set.

java

try {
  Long amount = get(Fields.In, "amount").getLong(r);
  get(Fields.Out, "doubled").setValue(outputRow, amount * 2);
  putRow(data.outputRowMeta, outputRow);
} catch (Exception ex) {
  // nr errors, description, field name, error code
  putError(data.outputRowMeta, outputRow, 1, ex.getMessage(), "amount", "UDJC001");
}

Logging

Logging is explicit. logBasic, logMinimal, logDetailed, logDebug, logRowlevel, and logError write to this transform’s log channel. checkFeedback(getLinesOutput()) is true every time the feedback line interval is reached, which keeps a long run from logging every row:

java

putRow(data.outputRowMeta, outputRow);

if (checkFeedback(getLinesOutput())) {
  logBasic("Wrote " + getLinesOutput() + " rows");
}

getLinesInput, getLinesRead, getLinesWritten, getLinesUpdated, getLinesSkipped, getLinesRejected, and getErrors return the counters. setErrors(1) and stopAll() fail the pipeline from inside the class. addResultFile(resultFile) adds a file to the pipeline result.

Blocking specific code

Code is checked, before it is compiled, against a list of forbidden substrings. A match is rejected with the text that matched.

Hop reads codeExclusions.xml in the Janino plugin folder (plugins/transforms/janino). The file installed there is empty, so everything is allowed. Add an exclusion element for each piece of text to block:

xml

<exclusions>
    <exclusion>System.</exclusion>
    <exclusion>HopVfs.</exclusion>
</exclusions>

The check is a substring search of the class source, not a Java parser. System. also matches that text inside a comment or a string.