Welcome in 2021, happy new year!

While the world has seen its share of problems in 2020, it’s been a roller coaster for Hop!

Our community continued to work tirelessly in December, even through the holiday period.

Let’s have a look at what happened in this last burst of productivity for 2020. With this pace of development, 2021 will be a great year for Hop!

A special shout out to Nicolas Adment, who was on an absolute roll in December!

Apache Hop Incubation - 0.50 release preparation

We had two voting rounds for 0.50 release candidates: 0.50-rc1 and 0.50-rc2.

We were told in advance that getting used to "the Apache Way" would take a while. Because of some lingering legacy problems (code, dependency licensing etc), we didn’t reach the required total of 3 positive votes to release 0.50-rc2 as Apache Hop (Incubating) 0.50.

Some of the issues mentioned were already resolved in the code after the 0.50 release branch, and development moved on in preparation for the 0.60 release. We therefore won’t be building a 0.50-rc3 and will move on to 0.60-rc1.

Check the mailing list archives for more information:

UI Updates

Hop Gui continues to evolve. A couple of the changes we made in December:

Single Click UI Actions

Since eliminating right click and double actions improves developer productivity and helps us to move towards a unified user interface on all devices and platforms, we increased focus on single-click actions in the user interface.

New Icons

Nicolas worked hard to update the icons in Hop Gui. A couple of areas where Hop Gui was given some icon love:

  • new workflow and pipeline icons

  • new toolbar and menu icons

  • new and/or updated icons for a number of actions and transforms

The result is a Hop Gui that looks and feels lighter and cleaner:

Icon Updates

New Metadata Perspective

A lot of metadata operations (relational, database connections, runtime configurations etc) had their own pop-up configuration dialogs.

These various pop-ups were grouped in one central metadata configuration perpective.

Metadata Perspective

Export to SVG

A neat little feature was introduced to Hop Gui: export to SVG.

Export To SVG

Although not spectacular in itself, this little menu item gives you a full svg (scalable) export of your workflow or pipeline graph. For Hop, this is a first step towards self documenting workflows and pipelines.

New Plugins

Two new transform plugins are now available in Hop:

Integration tests

Since nothing works as well as eating your own dog food, Hop now has an integration testing framework that is included in our daily build.

The purpose of the integration testing framework is as simple as useful: code tests can only test a number of cases. Hop was designed and built to be easy. Easy to use, easy to configure, and now easy to test.

Tests are built as workflows and pipelines. The success or failure of a test workflow is determined through the abort or success action exit code.

There is no doubt about the usefulness of this framework: since its introduction (mid-December), the Hop developers have found and fixed a number of bugs that may have been in the code for over a decade!

Integration Testing

Partitioned Stream Lookup

The Stream Lookup transform now supports partitioning.

Partitioned Stream Lookup

This significantly helps to improve lookups on large volumes of data on large machines.

Hop Berserker

Docker, Kubernetes updates

The separate Dockerfile and configuration were moved to the main Hop repository.

A Hop Docker image is now one docker pull away through Docker Hub.

Work on Kubernetes continues in the separate repository.

Jandex

All plugins in Hop now use Jandex.

Jandex uses annotations to find and load all plugins at startup. This reduces the startup times for Hop Gui, Hop Run and Hop Server.

Documentation

Lots of existing pages have been extended and/or updated. A selection of the new documentation we have available:

As always, ping us on mattermost to discuss documentation, and create tickets if you find any missing or incorrect information in the docs. Documentation is code, we take bugs in the documentation just as serious as bugs in the code.

Community

The Hop community continues to grow:

  • chat: 122 registered members (up from 108) join

  • LinkedIn: 321 followers (up from 309) follow

  • Twitter: 250 followers (up from 233) follow

  • YouTube: 50 subscribers (up from 40) subscribe

  • Facebook: 20 followers (no change) like

The following people have been added as committers:

  • Rodrigo Haces (late arrival, not included in November update)

  • Jason Chu (late arrival, not included in November update)

  • Yannick Mols

Check out the complete list of committers and contributors.

Without community contribution, Hop is just a coding club! Please feel free to join, participate in the discussion, test, file bug tickets on the software or documentation, …​ Contributing is a lot more than writing code.

Check out our contribution guides to find out more.

Various

Hop Web

Hop Web has seen some impressive activity from Hiromu over the last months.

The code is in the final stages of being merged into Hop. Once that’s done, you’ll be able to choose to run Hop Gui from a fat (desktop) or web client. Both will be started from the same code base.

Needless to say this will open an entire new world of possible use cases for Hop.

Hop Web

JIRA Tickets

The full list of issues that had activity over the last month is:

Resolved (66)

Issue Summary Components Created Updated

HOP-2143

Not found from Search option

Documentation

2020-11-06

2020-12-16

HOP-2246

The HopGui metadata mangers are not refreshed

GUI

2020-12-02

2020-12-16

HOP-2248

The Pipeline action dialog doesn’t remember its size

GUI

2020-12-03

2020-12-04

HOP-2249

Workflows: when not pasting XML see if other content can be recognized

GUI

2020-12-03

2020-12-04

HOP-2251

creating a new environment with a new name is not possible

GUI

2020-12-03

2020-12-04

HOP-2255

Trying to create a fat jar in a project home folder fails

Beam

2020-12-04

2020-12-04

HOP-2256

Create a GUI events handler

GUI

2020-12-04

2020-12-04

HOP-2258

Separate ActionSpecial into 2 actions START and DUMMY

Actions

2020-12-04

2020-12-13

HOP-2260

Metadata perspective: code cleanup

GUI

2020-12-04

2020-12-15

HOP-2261

Partition schema editor: doesn’t pick up number of partitions

GUI

2020-12-04

2020-12-05

HOP-2266

Beam generating unbounded synthetic data too fast

Beam

2020-12-05

2020-12-05

HOP-2267

Beam complaining about slf4j

Beam

2020-12-05

2020-12-16

HOP-2268

Avoid state of metadata by not implementing IVariables

API

2020-12-06

2020-12-16

HOP-2270

Add Github README with badges

2020-12-06

2020-12-06

HOP-2273

Remove duplicate Hop jar files

Build

2020-12-06

2020-12-16

HOP-2274

Declare all plugins with only annotations without penalizing startup time

2020-12-06

2020-12-11

HOP-2275

Error opening files

GUI

2020-12-07

2020-12-15

HOP-2277

Options dialog has i18n issues

GUI

2020-12-07

2020-12-08

HOP-2279

Add log channel to the JavaScript action

Actions

2020-12-08

2020-12-11

HOP-2280

Pipeline and Workflow action dialog: change name after selecting file

Actions, GUI

2020-12-08

2020-12-11

HOP-2282

De-select objects on single canvas click

GUI

2020-12-09

2020-12-18

HOP-2283

The @HopMetadata annotation must be consistent with the other plugins

API

2020-12-09

2020-12-14

HOP-2285

Remove unused annotation EnginePlugin

2020-12-10

2020-12-11

HOP-2286

Images not found for Dummy and Start actions

2020-12-11

2020-12-11

HOP-2288

Define an order of perspective

GUI

2020-12-11

2020-12-12

HOP-2289

Output of Sort Rows is suspect

Transforms

2020-12-11

2020-12-11

HOP-2290

Set Variables transform throws an error when missing workflow

Transforms

2020-12-11

2020-12-11

HOP-2292

Internal Hop variables not set on pipelines and workflows

Pipelines, Workflows

2020-12-11

2020-12-16

HOP-2293

Various issues while executing Beam pipelines

Pipelines

2020-12-11

2020-12-12

HOP-2294

Move the plugin image to its folder

2020-12-12

2020-12-16

HOP-2295

Improve the sort order of the variables shown with CTRL-SPACE

GUI

2020-12-12

2020-12-12

HOP-2296

HopVfsFileDialog is unusable under windows

2020-12-12

2020-12-14

HOP-2298

Port Token Replacement Plugin from Chris Deptula

Transforms

2020-12-13

2020-12-22

HOP-2299

Search: when finding metadata objects you can’t open them

GUI

2020-12-14

2020-12-16

HOP-2301

Add icon to each type of Hop file

GUI

2020-12-14

2020-12-14

HOP-2302

Stream Lookup: allow it to be run partitioned

Transforms

2020-12-14

2020-12-16

HOP-2303

remove duplicate 'download' from download link

Website

2020-12-15

2020-12-15

HOP-2304

Widget disposed in parameter tabs on Executor transforms

GUI

2020-12-15

2020-12-16

HOP-2306

Add support for TLS 1.2

2020-12-15

2020-12-15

HOP-2310

Exasol dependency error

2020-12-16

2020-12-19

HOP-2312

Add Sequence delivers duplicate ids when running in copies

Transforms

2020-12-16

2020-12-17

HOP-2313

Running integration tests locally is dangerous

Integration Testing

2020-12-17

2020-12-17

HOP-2316

Create an export to SVG menu options in Hop GUI

GUI

2020-12-18

2020-12-18

HOP-2317

Execute row SQL script broken

2020-12-18

2020-12-18

HOP-2318

Add database to test container

Integration Testing

2020-12-18

2020-12-27

HOP-2319

top bar menu disappears when using any item in documentation menu

Website

2020-12-18

2020-12-19

HOP-2322

With double-click support disabled, allow editing with a single click

GUI

2020-12-19

2020-12-21

HOP-2323

Options dialog: mangled option

GUI

2020-12-19

2020-12-21

HOP-2325

Add the Apache License header to all .properties files

2020-12-20

2020-12-21

HOP-2326

Add the Apache License header to all shell scripts

CLI

2020-12-20

2020-12-21

HOP-2327

Add the Apache License header to all XML files

Build

2020-12-20

2020-12-21

HOP-2331

Add right padding in Project dialog

2020-12-21

2020-12-23

HOP-2332

Add right padding in Environment dialog

2020-12-21

2020-12-23

HOP-2333

Environment variables not properly managed

2020-12-21

2020-12-22

HOP-2334

Review Token Replacemen plugin icon

2020-12-21

2020-12-29

HOP-2337

Selection mode entered inadvertently

GUI

2020-12-21

2020-12-21

HOP-2338

After selecting a transform in the action dialog with CTRL you get a NPE

GUI

2020-12-21

2020-12-21

HOP-2340

No longer able to create a hop in a workflow

GUI

2020-12-21

2020-12-23

HOP-2346

Remove mulesoft redshift jdbc dependency

Build

2020-12-23

2020-12-23

HOP-2347

Repair image for ValueMeta

2020-12-23

2020-12-25

HOP-2350

Rat should print filenames to console

Build

2020-12-26

2020-12-26

HOP-2353

Add a function to the GuiResource class to get image from IValueMeta

GUI

2020-12-26

2020-12-28

HOP-2356

Memory Group By - Count - outputs counts with padded zeros.

2020-12-27

2021-01-02

HOP-2360

Excel Input transform with XLSX Streaming don’t work

Transforms

2020-12-28

2020-12-29

HOP-2361

Hop About Box opens too small.

GUI

2020-12-29

2020-12-30

HOP-2363

Ctrl-+ doesn’t change the zoom level

2020-12-29

2021-01-02

In Progress (15)

Issue Summary Components Created Updated

HOP-2244

Reverse sort the projects drop-down list by use date

GUI

2020-11-30

2020-12-04

HOP-2281

Clean GuiResource images

2020-12-08

2020-12-29

HOP-2284

Update kettle to hop plugin porting documentation

API

2020-12-10

2020-12-15

HOP-2287

Move hop-docker into ASF HOP project and make docker image available in dockerhub via pipeline

Build, Containers, Infrastructure, Pipelines

2020-12-11

2020-12-29

HOP-2307

Add support to Clickhouse database

Database

2020-12-16

2021-01-03

HOP-2308

Enable pritty print on JSON output

Transforms

2020-12-16

2021-01-03

HOP-2309

Enable pritty print on XML output

Transforms

2020-12-16

2021-01-03

HOP-2320

NPE in User Defined Java Class code crashes Hop GUI

GUI

2020-12-18

2020-12-18

HOP-2324

add release process information to contribution page

Website

2020-12-20

2020-12-21

HOP-2330

Review of italian translation files

Translations

2020-12-21

2020-12-22

HOP-2341

Improve integration testing documentation

Documentation, Integration Testing

2020-12-21

2020-12-21

HOP-2342

Single click mode: moving a transform or action can cause edit

GUI

2020-12-21

2020-12-21

HOP-2355

cleanup integration tests

Integration Testing

2020-12-27

2020-12-27

HOP-2358

Remove jxl dependency from hop-transform-excelinput

Transforms

2020-12-28

2020-12-31

HOP-2359

Transform ExcelWriter upgrade POI 4.1.2 and remove unnecessary dependencies

Transforms

2020-12-28

2020-12-29

Open (40)

Issue Summary Components Created Updated

HOP-2118

macOS dark mode results in illegible UI text

GUI

2020-10-14

2020-12-15

HOP-2247

Need a way to clear a setting in Meta Data Injection Transform

Transforms

2020-12-02

2020-12-02

HOP-2250

Pipelines: when pasting allow other content then XML to be recognized

GUI

2020-12-03

2020-12-03

HOP-2252

Create page with branding/sample files

Documentation, Website

2020-12-03

2020-12-03

HOP-2253

Add a bit of left padding on the Action Dialog

GUI

2020-12-03

2020-12-03

HOP-2254

create usage documentation + examples for the various runtime configurations

Documentation, GUI

2020-12-04

2020-12-04

HOP-2257

Update Jetty Dependencies

API, Hop Server

2020-12-04

2020-12-04

HOP-2259

Add a warning to indicate the consequences of renaming metadata objects

GUI

2020-12-04

2020-12-16

HOP-2262

Native core plugins a registered twice

2020-12-04

2020-12-05

HOP-2263

Add support for a HOP_PLUGIN_FOLDERS variable

API

2020-12-04

2020-12-04

HOP-2264

Join rows (cartesian products) - Fields should be editable

GUI

2020-12-04

2020-12-04

HOP-2265

Allow results to be evaluated in an action

Actions

2020-12-05

2020-12-05

HOP-2269

Allow metadata to be read from other projects

Metadata

2020-12-06

2020-12-16

HOP-2271

Integration Tests: Add ability to pass parameters to hop-run.sh

Integration Testing

2020-12-06

2020-12-06

HOP-2276

Add option to stop pipeline or workflow after some time

Pipelines, Workflows

2020-12-07

2020-12-07