XML and JSON serialization

Table of Contents

The Basics
Legacy and renamed keys
Lists
Maps
Sub-elements
Serializing interfaces

The Basics

To serialize fields to XML in the Meta transform classes or the Action action classes, or indeed to JSON for project metadata elements, you can use the @HopMetadataProperty annotation. The serialization to XML or JSON will happen with the name of the field by default. For example:

The default

java

@HopMetadataProperty
String filename;

This will serialize to:

XML

<filename>field-content</filename>

JSON

{
  "filename" : "field-content"
}

The key property

If you want to use a different XML element name you can specify this with the key property of the annotation:

java

@HopMetadataProperty(key="file-name")
String filename;

This will serialize to:

XML

<file-name>field-content</file-name>

JSON

{
  "file-name" : "field-content"
}

Metadata references

For fields of types that implement IHopMetadata, you can opt to store the name like this:

java

@HopMetadataProperty(
        key="connection",
        storeWithName=true,
        hopMetadataPropertyType=HopMetadataPropertyType.RDBMS_CONNECTION)
DatabaseMeta databaseMeta;

This will serialize to:

XML

<connection>element-name</connection>

JSON

{
  "connection" : "element-name"
}

Enum fields

Enumerations are stored by default using their name. Since it’s used so often we also allow you to implement IEnumHasCode or even IEnumHasCodeAndDescription. For enums that implement either you can opt to store the value by a chosen code. This is useful for backward compatibility with older XML formats and to give you the flexibility needed.

Example from DimensionLookupMeta.java:

java

 @HopMetadataProperty(
        key = "creation_method",
        storeWithCode = true)
    private TechnicalKeyCreationMethod creationMethod;

This will serialize to:

XML

<creation_method>autoinc</creation_method>
<creation_method>sequence</creation_method>
<creation_method>tablemax</creation_method>
<creation_method>uuid</creation_method>
<creation_method>field</creation_method>

One of these code values will be used depending on the enum chosen.

JSON

{
  "creation_method" : "autoinc"
}

int codes

If often happens in the codebase that field and trim types are stored as integers, typically something like IValueMeta.TYPE_STRING (2) or IValueMeta.TRIM_TYPE_LEFT. Since this isn’t very meaningful, we offer the ability to specify an IIntCodeConverter to convert 2 into String.

Example from CsvInputField.java:

java

  @HopMetadataProperty(
      key = "type",
      intCodeConverter = ValueMetaBase.ValueTypeCodeConverter.class)
  private int type;

This will serialize to:

XML

<type>String</type>

JSON

{
  "type" : "String"
}

In the same way there is a standard class for trim type: ValueMetaBase.TrimTypeCodeConverter.

Passwords & secrets

If you have sensitive information that you want to obfuscate or encrypt, you can use the password annotation property.

java

  @HopMetadataProperty(password=true)
  private String password;

This will serialize to:

XML

<password>Encrypted: SomeObfuscationGoingOn</password>
<secret>AES2: SomeEncryptionGoingOn</secret>

JSON

{
  "password" : "Encrypted: SomeObfuscationGoingOn",
  "secret": "AES2: SomeEncryptionGoingOn"
}

String encoder

Sometimes you want to encode Strings to Base64 or other schemes to avoid having unsupported characters or even binary content in your serialized files. You can use the stringEncoder property of the annotation for this.

From TextFileInputMeta.java:

java

  /** The string to filter on */
  @HopMetadataProperty(
      key = "filter_string",
      stringEncoder = Base64StringEncoder.class)
  private String filterString;

This will serialize Apache Hop to:

XML

<filter_string>QXBhY2hlIEhvcA==</filter_string>

JSON

{
 "filter_string" : "QXBhY2hlIEhvcA=="
}

Legacy and renamed keys

When a property is renamed or a field changes type, existing serialized files still carry the old key. To keep reading them you can mark a method with @OnUnknownDeserializationProperty. The serializers invoke it once for every property encountered during deserialization that does not map to any @HopMetadataProperty field.

This is the recommended way to translate a legacy or renamed key into a current field. It works for both XML and JSON serialization.

From MetaInjectMeta.java, where the legacy boolean allow_empty_stream_on_execution was replaced by the ConditionStreamBehavior enum:

java

@OnUnknownDeserializationProperty
public void onUnknownDeserializationProperty(UnknownProperty property) {
  // Only translate the legacy boolean when the current condition_stream_behavior enum was not
  // part of the file. When both are present the current enum wins, so a file that mixes the
  // two can still represent ALWAYS.
  if ("allow_empty_stream_on_execution".equals(property.getName())
      && !conditionStreamBehaviorSet) {
    conditionStreamBehavior = ConditionStreamBehavior.fromLegacyBoolean(property.getValue());
  }
}

The serializers set annotated fields before firing this callback, and the conditionStreamBehaviorSet flag (raised whenever the current enum value is set, including during deserialization) therefore tells the callback whether a file that contains both the old boolean and the new enum really carries the new value. The legacy boolean is written only when the current tag was absent.

The callback method must be public and take exactly one parameter of type UnknownProperty, which exposes getName() (the key as found in the serialized data) and getValue() (its raw textual value). You may declare several such methods; they are all invoked for each unknown property.

Discovery is based on Class.getMethods(), so private and protected methods are ignored, and a public override in a subtype replaces the inherited method (only the most-derived declaration is invoked, the overridden parent method is not called). Any annotated method that does not take exactly one UnknownProperty parameter raises an IllegalArgumentException.

If you prefer, you can still read the raw serialized node directly in a convertLegacyXml(Node) override for XML-only conversions.

Lists

If you want to serialize List of objects, you can typically wrap them in a group. An example from CsvInputMeta.java:

java

  @HopMetadataProperty(
      key = "field",
      groupKey = "fields")
  private List<CsvInputField> inputFields;

Will serialize the content of the CsvInputField objects inside <fields><field> wrappers:

XML

<fields>
  <field>
    <name>fieldName</name>
    <type>String</type>
    ...
  </field>
  ...
</fields>

JSON

{
  "fields": [
    {
      "name": "fieldName",
      "type": "String",
      ...
    },
    ...
  ]
}

Maps

You can serialize Map fields to XML as well.

It’s not possible to inject metadata into a Map data type so don’t use these constructs for transform serialization.

java

  @HopMetadataProperty(
      key = "mapping",
      groupKey = "mappings",
      inline = true,
      mapKeyClass = Key.class,
      mapValueClass = Value.class)
  private Map<Key, Value> mappings;

Serializes to:

XML

<mappings>
  <mapping>
    <key>
      <keyProperty1>key1</keyProperty1>
      <keyProperty2>key2</keyProperty2>
    </key>
    <value>
      <valueProperty1>value1</keyProperty1>
      <valueProperty2>value2</keyProperty2>
      <valueProperty3>value3</keyProperty3>
    </value>
  </mapping>
  ...
</mappings>

JSON

JSON serialization of Map objects is not supported yet.

Sub-elements

All elements you want to put in a different sub-element you can put in a different field and give it another key.

Serializing interfaces

It’s possible to serialize objects with only their interface. You need to specify a @HopMetadataObject annotation on the interface in question. The following example is taken from IValueMeta.java where data types are plugins. We want to read the type String and know to get object ValueMetaString by specifying an Object Factory like this:

java

@HopMetadataObject(objectFactory = IValueMeta.ValueMetaHopMetadataObjectFactory.class)
public interface IValueMeta {}

The way that this is done is with 2 simple methods. One for creating a new object with an id, and one for getting an ID from an object interface.

From class ValueMetaHopMetadataObjectFactory:

java

    @Override
    public Object createObject(String id, Object parentObject) throws HopException {
      int type = ValueMetaFactory.getIdForValueMeta(id);
      return ValueMetaFactory.getValueMetaName(type);
    }

    @Override
    public String getObjectId(Object object) throws HopException {
      return ((IValueMeta) object).getTypeDesc();
    }