To support scenarios where the `runs.as` or `lives.as` user/group
already exists in the base image, pair the calls to `useradd` and
`groupadd` with `getent` conditionals.

Bug: T268819
Change-Id: I3aef722ac57b38695c7411e64b41589fad16e95b

The `apt.proxies` field is now merged by appending second set of
proxies to the original.

Example:

    apt:
      proxies:
        - url: http://proxy.example:8080
    variants:
      foo:
        apt:
          proxies:
            - url: http://proxy.example:8081
              source: http://security.debian.org

In the above example, the effective `apt.proxies` configuration for the
`foo` variant would include both the base `proxy.example:8080` proxy and
the `proxy.example:8081` proxy specific to the `security.debian.org`
source.

Bug: T277109
Change-Id: I64bad1eb1e6fb6712a5de178be3aa273936a2df5

Defines a new `apt.proxies` field that allows users to specify
http/https proxy URLs that apply to one or all APT sources. APT
configuration for the proxies will be written prior to package
installation and then removed.

Both a shorthand and longhand configuration are supported.

Example:

    apt:
      packages: [foo]
      proxies: [http://proxy.example:8080]

Is a shorthand equivalent of:

    apt:
      packages: [foo]
      proxies:
        - url: http://proxy.example:8080

In which case, the following APT config is written before installing
packages, proxying all http sources via `http://proxy.example:8080`.

    Acquire::http::Proxy "http://proxy.example:8080";

Example:

    apt:
      packages: [foo]
      proxies:
        - url: https://proxy.example:8081
          source: https://security.debian.org

In which case, the following APT config is written before installing
packages, proxying only https requests to security.debian.org via
`https://proxy.example:8081`.

    Acquire::https::Proxy::security.debian.org "https://proxy.example:8081";

Other changes include a refactoring of the `AptConfig.Packages` type and
unmarshalling function. A formal type `AptPackages` is now defined,
derived from `map[string][]string`, and the `UnmarshalJSON` member
function moved there. This was done to avoid bloating of the general
`AptConfig` unmarshal function with implementation for both fields
(`packages` and `proxies`).

The new `AptConfig.Proxy` type defines its own unmarshal function to
support both shorthand and longhand configurations.

Bug: T277109
Change-Id: I5b82efcc441e48890cfab62747519d8986d7e8ac

To support image compositions that rely on copying files from one or
more external images, allow `ArtifactsConfig.From` to be any valid image
reference.

Generalize `baseimage` validator as `imageref` and create a new
`artifactfrom` validator alias for `imageref|variantref`. Use the new
validator for `From`.

Additionally, omits non-variant `From` references from the dependency
graph.

Change-Id: I3b92d815b62856fe5a8df836b37cce0043a74ffa

A `python` config block already triggered installing pip via
easy_install which is pretty specific to python2 era usage. It then used
pip to install/update pypi packages for setuptools, wheel, and tox. This
change adds the pip package to that set of pypi install/update packages.
This is useful to ensure that the resulting image is capable of
installing manylinux2014 formatted wheels (pip >= 19.3). Debian Buster's
python3-pip package is pip 18.1 which is too old to look for
manylinux2014 wheels when installing packages such as cryptography which
have stopped publishing wheels with the older manylinux1 and
manylinux2010 platform tags.

Existing pinning of pip to a version less than 21.0 is preserved for
those poor, unloved python2 projects that linger in an ever colder and
darker world hostile to their kind.

Bug: T274435
Change-Id: I81f0af69dbd4f9202bc9736faf653813c81fc0ad

Pip 21 has dropped support for Python 2 but is still strangely
selected for installation when running setuptools with a python2 binary.
Let's pin pip when a `version` is specified that starts with "python2".

Note this will require python 2 users to explicitly specify `version:
python2` even if their default `python` binary in the base image is for
python 2.

Bug: T273793
Change-Id: Id7d4315e345657ce7ac2efdd0b065d693e47b2f5

Kubernetes non-root verification only supports numeric UIDs. With
usernames (string) used, it can't verify that the user is not root and
therefore will reject running the image if RunAsNonRoot is enforced.

Bug: T228967
Change-Id: Iab59a28291c3712a135aa6aa21e4b338efe3ec43

In order to allow packages from backports to be downloaded, the ability
to target a release when running apt-get install is required.

apt.packages now takes the previously defined list of string or a map
of targets and packages to install. When using a map, in order to
install the packages for the base image, the 'default' key should be
used:

packages:
  default: ["libfoo", "libbar"]
  buster-backports: ["libbaaz"]

Bug: T272759
Change-Id: I435c47794ffbc0264e0a440bfbd23e11570645d1

There is not reason to stick to stretch. We already have a buster image
with golang 1.13-3, let's switch to it.

Bug: T271901
Change-Id: I231940c92354ddbaa3e486f5f2a93cadce3c489a

Copying of required files is now a generic operation implemented by
`config.RequirementsConfig`. Other builder types should re-use this
implementation.

To support this change, some `RequirementsConfig` implementation around
source and destination paths was moved to functions of
`ArtifactsConfig`.

Build macro functions `SortFilesByDir` and `SyncFiles` are no longer
needed and were removed.

Change-Id: Ieb0cd2a0b1e4f11b05d19c63bea9bdf9d578e3ea
Follows-up: If813829bdace6851bdba56abcdfcab1cd967df03

composer allows packages to run code after downloading them. In nearly every
case this is never needed and just opens up room for arbitrary code
execution.

Bug: T270207
Change-Id: Ic7db66f6e00d82ae70078eab333736f107ca5a48

Flip the order of `path.Clean()` and `path.Dir()` in the NewFromShort()
constructor. The prior order was stripping directory indicators from the
source path prematurely.

Bug: T263597
Change-Id: I0d76b4632660a8c32fb1c8a7a90b287546271348

Destination is now an optional field, defaulting to the source path
when omitted.

This makes for less redundancy when defining cross-variant builder
requirements and copies in cases where the application directories
are the same between variants.

Change-Id: I44cc3fdff260670b07b6cb3d82d4d7a49ff221f7

Allow `builder.requirements` to use a long form similar to `copies` when
the user needs to copy files from one variant to another as part of the
builder stage. This allows multi-stage builds to generate some artifact
in stage "A" and then copy that artifact into stage "B" in the
PreInstall phase before running `builder.command`.

Usage example:
```
variants:
  A:
    apt:
      packages:
        - wget
    builder:
      command:
        - /bin/bash
        - -c
        - >-
          wget https://example.org/some_cool_utility.tgz


          && tar xzvf some_cool_utility.tgz
          && rm some_cool_utility.tgz
  B:
    builder:
      requirements:
        - src
        - from: A
          source: some_cool_utility
          destination: .
      command:
        - some_cool_utility --do-something-cool src/
```

Bug: T263597
Co-authored-by: Dan Duvall <dduvall@wikimedia.org>
Change-Id: If813829bdace6851bdba56abcdfcab1cd967df03

The `lint` make target contained a logic error which made make evaluate
the test condition rather than bash. This in turn has prevented all
gofmt warnings from being seen by developers. The fix was escaping the
`$` character in the Makefile by doubling it.

Following this fix, the existing linter errors are also fixed by this
commit.

Change-Id: Ieb0c91cbdd0b13602fba9b6f81c97d22f36a7fd7

Fixes various issues with the openapi 3.0 blubberoid specification,
includes the variant schema using `additionalProperties`'s object
notation, and implements a test that performs validation.

Change-Id: I203ff340a9753a1541512ab466ac845e0e7e4f64

Poetry (<https://python-poetry.org/>) is a packaging and dependency
management tool for Python. Poetry creates a fully versioned dependency
tree for your project. The poetry.lock file tracks the exact versions of
each library and its dependencies for repeatable installs.

Usage example:
```
variants:
  build:
    python:
      version: python3
      poetry:
        version: ==1.0.10
      requirements: [pyproject.toml, poetry.lock]
  test:
    includes: [build]
    python:
      poetry:
        devel: true
    copies: [local]
  production:
    includes: [build]
    copies: [local]
```

python.poetry.version must be a simplified Python package version
specification. A new "pypkgver" validator is provided to ensure that the
value is usable for a `pip install poetry...` command. Typically it
would be reasonable to pin the version of Poetry in use to an exact
version as show in the sample above, but it may be desirable to use
other more complex constraints such as ">=1.0,!=1.0.3,<2.0".

python.poetry.devel is a boolean flag indicating whether or not to
install development dependencies in the Poetry managed venv. It defaults
to false.

When python.poetry.version is non-empty various changes will be made to
the generated Dockerfile.
; PhasePrivileged
: A version of Poetry matching the python.poetry.version constraint
  will be installed using pip.
: POETRY_VIRTUALENVS_PATH will be set in the environment.
; PhasePreInstall
: PIP_WHEEL_DIR and PIP_FIND_LINKS will not be set in the environment.
: `poetry install --no-root --no-dev` will be used to create and
  populate a venv within POETRY_VIRTUALENVS_PATH.
: If python.poetry.devel is true, the `--no-dev` flag will be omitted.
; PhasePostInstall
: PYTHONPATH and PATH will not be set in the environment.

Change-Id: I33b356ff90983f8b8d5b76003851db139d97fe2e

Since the chart name is different than the service name, the chart name
is needed in the promote step.

Change-Id: If145f17ca05c917c62eaca82b73d7ebdcc2362f7

Adds a promote step to the pipeline config so that the blubber
deployment chart will be updated with each newly published image.

Bug: T255835
Change-Id: I978792af67528009f7b964e9a55fcd594d44c72f

Dduvall authored Aug 07, 2020 and Dduvall committed Sep 09, 2020

Allow for variants that have no base image (scratch images), omitting
compilation of all features but copies and entrypoint.

One use case for this feature is to build images of minimal size. For
example, the following produces a production blubberoid image of only
11M compared to the current 66M image based on WMF's stretch.

    version: v4
    variants:
      build:
        base: docker-registry.wikimedia.org/golang:1.11.5-1
        apt: {packages: [gcc, git, make]}
      prep:
        includes: [build]
        runs:
          environment:
            CGO_ENABLED: '0'
        builder:
          command: [make, blubberoid]
          requirements: [.]
      production:
        copies:
          - from: prep
            source: /srv/app/blubberoid
            destination: /srv/app/blubberoid
        entrypoint: [/srv/app/blubberoid]

Another possible use case would be to allow for the intermediate
publishing of individual MW extensions and skins that have gone through
a build phase yet require further integration downstream into fully
deployable images that include the core platform, configuration, l10n
database, etc.  Basing such images on scratch would drastically lower
storage costs.

Bug: T260830
Change-Id: I403206981f55b59246886dfcf2bbff9c316b285b

Add ability to run 'composer install' in order to support php
microservices.

Bug: T261783
Change-Id: I1a81378aac83db31c66fc013fc37bb1d75e41c3f

Removes the chart url from the chart definition and adds a name
property. Chart registry location is now stored in PipelineLib.

Bug: T261346
Depends-on: Ifeb5caec6bf120b86ae16afd0315d98ffbc63dad
Change-Id: Ied1c2403a67581c45edde4b9527a41876ae33290

Broken by https://gerrit.wikimedia.org/r/c/blubber/+/613701

Test added.

Bug: T259069
Change-Id: I5d25f3b30d2a29f98ac2fa085fece1dbde60d4a4