Git Input

The Git Input transform reads commits, issues, pull requests, issue comments and issue events from a Git hosting provider, or reads commit history straight from a clone on disk.

Supported providers are GitHub (cloud and Enterprise), GitLab, Bitbucket, Forgejo and Gitea.

Every provider is normalized to the same output row, so a pipeline written against GitHub keeps working when it is pointed at a GitLab or Bitbucket Git connection.

The row layout is chosen per Resource type, not shared by all of them. A single layout would have to be the union of everything any type can report, so a commit row would carry empty label and assignee columns and an issue row would carry empty branch and merge columns.

Within a type the layout is identical across providers, so a pipeline written against GitHub keeps working when it is pointed at GitLab or Bitbucket. A field a given provider does not report arrives as an empty value, not as a missing column.

Changing Resource type changes the fields the transform emits, so re-open the dialog after changing it to let the downstream transforms pick the new layout up.

Set Source to LOCAL and point Local repository path at the folder that contains .git. No connection or token is needed.

Commit rows carry both Git idents in full: author and author_email from the author ident, committer and committer_email from the committer ident, and is_merge from the parent count. author_login stays empty, because a clone on disk has no provider accounts to match idents against. All of it is repeated in raw_json.

That is enough for a contributor list: group on author and author_email, count the rows, and filter on is_merge being false to leave merge commits out of the totals.

COMMIT_FILES emits one row per changed file per commit, with the file path in title and the change type (added, modified, deleted, renamed, copied) in state. Rename and copy detection is on, so moving a file produces a single renamed row carrying its previous path in body rather than a deleted row plus an added row. Merge commits are compared against their first parent.

Comments on issues and on pull requests are separate resource types, so ISSUE_COMMENTS never returns pull request comments and PR_COMMENTS never returns issue comments. That matches ISSUES and PULL_REQUESTS, which are also separate.

On GitHub both come from one repository-wide feed and are told apart by the comment’s own URL, so reading both types costs two passes over that feed. On a repository where most discussion happens in pull requests, ISSUE_COMMENTS can legitimately return very few rows.

The GitHub comment feed returns comments without their parent issue, so title and state are empty for GitHub ISSUE_COMMENTS rows. That feed also ignores State; use Since to limit it by comment time. GitLab and Bitbucket walk issues one at a time and do fill both fields.

The row cap applies to the rows that come out, not to the number of issues scanned.

If a provider request fails part way through pagination, after the built-in retries, the transform fails rather than passing on the rows it had already collected. An empty page cannot be told apart from the end of the data, so continuing would report a successful run over a silently incomplete result set.

A provider refusing to paginate further because of a documented limit of its own is not a failure. GitHub caps page-based pagination at 1,000 items on some endpoints; the transform stops there, keeps the rows it read and logs why. The same applies when the configured row cap is reached.

Requests use the JVM default proxy selector, so the usual http.proxyHost, https.proxyHost and http.nonProxyHosts system properties apply. That matters for a self-hosted GitLab, Gitea or Forgejo behind a corporate proxy. Custom certificate authorities are picked up from the usual javax.net.ssl.trustStore properties.

The samples project contains transforms/git-input-github-pull-requests.hpl, which reads recent pull requests from the apache/hop repository over the anonymous GitHub API.