Upgrade DataHub from 0.13.3 to 1.6.0 (T402408)

Btullisrequested to merge
T402408-upgrade-datahub-1.6.0 into main

DataHub 1.x restructures the images, so this is more than a version bump:

  • Build on JDK 21 and run on JRE 25 (1.x uses a Java 21 build toolchain).
  • GMS and datahub-upgrade are now Spring Boot executable archives; drop the jetty-runner/jetty-jmx/jetty-util downloads and jetty.xml/jetty-jmx.xml, which upstream no longer ships.
  • Frontend moves to Play 3: build the stage task and ship build/stage/main plus the mfe.*.yaml configs, replacing the old dist zip.
  • Replace jwilder/dockerize with upstream's wait_for_deps.sh helper sourced by the service start.sh scripts; drop the golang dockerize build stages.
  • Bump the OpenTelemetry agent to 2.27.0 and jmx_prometheus_javaagent to 1.0.1.
  • Drop the mysql-setup, elasticsearch-setup and kafka-setup images: upstream removed them in 1.x and their work is now done by the datahub-system-update job (run via the datahub-upgrade image). The Helm chart must run that job in place of the old setup jobs.
  • Bump the blubber buildkit syntax to v1.8.0.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Closes T402408

Merge request reports