1. 06 May, 2022 5 commits
  2. 20 Apr, 2021 1 commit
    • Dduvall's avatar
      user: Check for existing user/group before creating · 459234d2
      Dduvall authored
      To support scenarios where the `runs.as` or `lives.as` user/group
      already exists in the base image, pair the calls to `useradd` and
      `groupadd` with `getent` conditionals.
      Bug: T268819
      Change-Id: I3aef722ac57b38695c7411e64b41589fad16e95b
  3. 18 Mar, 2021 1 commit
  4. 12 Mar, 2021 1 commit
    • Dduvall's avatar
      apt: Support configuration of http/https proxies · 7bb48691
      Dduvall authored
      Defines a new `apt.proxies` field that allows users to specify
      http/https proxy URLs that apply to one or all APT sources. APT
      configuration for the proxies will be written prior to package
      installation and then removed.
      Both a shorthand and longhand configuration are supported.
            packages: [foo]
            proxies: [http://proxy.example:8080]
      Is a shorthand equivalent of:
            packages: [foo]
              - url: http://proxy.example:8080
      In which case, the following APT config is written before installing
      packages, proxying all http sources via `http://proxy.example:8080`.
          Acquire::http::Proxy "http://proxy.example:8080";
            packages: [foo]
              - url: https://proxy.example:8081
                source: https://security.debian.org
      In which case, the following APT config is written before installing
      packages, proxying only https requests to security.debian.org via
          Acquire::https::Proxy::security.debian.org "https://proxy.example:8081";
      Other changes include a refactoring of the `AptConfig.Packages` type and
      unmarshalling function. A formal type `AptPackages` is now defined,
      derived from `map[string][]string`, and the `UnmarshalJSON` member
      function moved there. This was done to avoid bloating of the general
      `AptConfig` unmarshal function with implementation for both fields
      (`packages` and `proxies`).
      The new `AptConfig.Proxy` type defines its own unmarshal function to
      support both shorthand and longhand configurations.
      Bug: T277109
      Change-Id: I5b82efcc441e48890cfab62747519d8986d7e8ac
  5. 23 Feb, 2021 1 commit
    • Dduvall's avatar
      copies: Allow copying directly from other images · db3a5397
      Dduvall authored
      To support image compositions that rely on copying files from one or
      more external images, allow `ArtifactsConfig.From` to be any valid image
      Generalize `baseimage` validator as `imageref` and create a new
      `artifactfrom` validator alias for `imageref|variantref`. Use the new
      validator for `From`.
      Additionally, omits non-variant `From` references from the dependency
      Change-Id: I3b92d815b62856fe5a8df836b37cce0043a74ffa
  6. 12 Feb, 2021 1 commit
    • BryanDavis's avatar
      python: upgrade pip before installing requirements · 007009ea
      BryanDavis authored
      A `python` config block already triggered installing pip via
      easy_install which is pretty specific to python2 era usage. It then used
      pip to install/update pypi packages for setuptools, wheel, and tox. This
      change adds the pip package to that set of pypi install/update packages.
      This is useful to ensure that the resulting image is capable of
      installing manylinux2014 formatted wheels (pip >= 19.3). Debian Buster's
      python3-pip package is pip 18.1 which is too old to look for
      manylinux2014 wheels when installing packages such as cryptography which
      have stopped publishing wheels with the older manylinux1 and
      manylinux2010 platform tags.
      Existing pinning of pip to a version less than 21.0 is preserved for
      those poor, unloved python2 projects that linger in an ever colder and
      darker world hostile to their kind.
      Bug: T274435
      Change-Id: I81f0af69dbd4f9202bc9736faf653813c81fc0ad
  7. 04 Feb, 2021 1 commit
    • Dduvall's avatar
      python: Pin pip package to <21 for Python 2 · d91393a4
      Dduvall authored
      Pip 21 has dropped support for Python 2 but is still strangely
      selected for installation when running setuptools with a python2 binary.
      Let's pin pip when a `version` is specified that starts with "python2".
      Note this will require python 2 users to explicitly specify `version:
      python2` even if their default `python` binary in the base image is for
      python 2.
      Bug: T273793
      Change-Id: Id7d4315e345657ce7ac2efdd0b065d693e47b2f5
  8. 03 Feb, 2021 1 commit
  9. 02 Feb, 2021 2 commits
  10. 01 Feb, 2021 1 commit
    • Jeena Huneidi's avatar
      apt.go: Add ability to target releases · eb038f2c
      Jeena Huneidi authored
      In order to allow packages from backports to be downloaded, the ability
      to target a release when running apt-get install is required.
      apt.packages now takes the previously defined list of string or a map
      of targets and packages to install. When using a map, in order to
      install the packages for the base image, the 'default' key should be
        default: ["libfoo", "libbar"]
        buster-backports: ["libbaaz"]
      Bug: T272759
      Change-Id: I435c47794ffbc0264e0a440bfbd23e11570645d1
  11. 26 Jan, 2021 1 commit
  12. 15 Jan, 2021 1 commit
  13. 18 Dec, 2020 1 commit
    • Dduvall's avatar
      Refactor other builder types to use RequirementsConfig · 18c9968e
      Dduvall authored
      Copying of required files is now a generic operation implemented by
      `config.RequirementsConfig`. Other builder types should re-use this
      To support this change, some `RequirementsConfig` implementation around
      source and destination paths was moved to functions of
      Build macro functions `SortFilesByDir` and `SyncFiles` are no longer
      needed and were removed.
      Change-Id: Ieb0cd2a0b1e4f11b05d19c63bea9bdf9d578e3ea
      Follows-up: If813829bdace6851bdba56abcdfcab1cd967df03
  14. 16 Dec, 2020 1 commit
  15. 15 Dec, 2020 1 commit
  16. 14 Dec, 2020 1 commit
    • BryanDavis's avatar
      requirements: Fix regression in short form handling · f063588e
      BryanDavis authored
      Flip the order of `path.Clean()` and `path.Dir()` in the NewFromShort()
      constructor. The prior order was stripping directory indicators from the
      source path prematurely.
      Bug: T263597
      Change-Id: I0d76b4632660a8c32fb1c8a7a90b287546271348
  17. 11 Dec, 2020 1 commit
    • Dduvall's avatar
      Make artifact destination optional · 058c2d54
      Dduvall authored
      Destination is now an optional field, defaulting to the source path
      when omitted.
      This makes for less redundancy when defining cross-variant builder
      requirements and copies in cases where the application directories
      are the same between variants.
      Change-Id: I44cc3fdff260670b07b6cb3d82d4d7a49ff221f7
  18. 10 Dec, 2020 2 commits
    • BryanDavis's avatar
      builder: support cross variant copying for builder.requirements · a9a61206
      BryanDavis authored
      Allow `builder.requirements` to use a long form similar to `copies` when
      the user needs to copy files from one variant to another as part of the
      builder stage. This allows multi-stage builds to generate some artifact
      in stage "A" and then copy that artifact into stage "B" in the
      PreInstall phase before running `builder.command`.
      Usage example:
              - wget
              - /bin/bash
              - -c
              - >-
                wget https://example.org/some_cool_utility.tgz
                && tar xzvf some_cool_utility.tgz
                && rm some_cool_utility.tgz
              - src
              - from: A
                source: some_cool_utility
                destination: .
              - some_cool_utility --do-something-cool src/
      Bug: T263597
      Co-authored-by: Dduvall's avatarDan Duvall <dduvall@wikimedia.org>
      Change-Id: If813829bdace6851bdba56abcdfcab1cd967df03
    • BryanDavis's avatar
      Fix Makefile syntax for running linter & legacy errors · e81d1adf
      BryanDavis authored
      The `lint` make target contained a logic error which made make evaluate
      the test condition rather than bash. This in turn has prevented all
      gofmt warnings from being seen by developers. The fix was escaping the
      `$` character in the Makefile by doubling it.
      Following this fix, the existing linter errors are also fixed by this
      Change-Id: Ieb0c91cbdd0b13602fba9b6f81c97d22f36a7fd7
  19. 16 Oct, 2020 1 commit
    • Dduvall's avatar
      Fix openapi spec and include variant schema · f34686ba
      Dduvall authored
      Fixes various issues with the openapi 3.0 blubberoid specification,
      includes the variant schema using `additionalProperties`'s object
      notation, and implements a test that performs validation.
      Change-Id: I203ff340a9753a1541512ab466ac845e0e7e4f64
  20. 25 Sep, 2020 1 commit
    • BryanDavis's avatar
      python.go: Add support for Poetry package manager · 5718d4d6
      BryanDavis authored
      Poetry (<https://python-poetry.org/>) is a packaging and dependency
      management tool for Python. Poetry creates a fully versioned dependency
      tree for your project. The poetry.lock file tracks the exact versions of
      each library and its dependencies for repeatable installs.
      Usage example:
            version: python3
              version: ==1.0.10
            requirements: [pyproject.toml, poetry.lock]
          includes: [build]
              devel: true
          copies: [local]
          includes: [build]
          copies: [local]
      python.poetry.version must be a simplified Python package version
      specification. A new "pypkgver" validator is provided to ensure that the
      value is usable for a `pip install poetry...` command. Typically it
      would be reasonable to pin the version of Poetry in use to an exact
      version as show in the sample above, but it may be desirable to use
      other more complex constraints such as ">=1.0,!=1.0.3,<2.0".
      python.poetry.devel is a boolean flag indicating whether or not to
      install development dependencies in the Poetry managed venv. It defaults
      to false.
      When python.poetry.version is non-empty various changes will be made to
      the generated Dockerfile.
      ; PhasePrivileged
      : A version of Poetry matching the python.poetry.version constraint
        will be installed using pip.
      : POETRY_VIRTUALENVS_PATH will be set in the environment.
      ; PhasePreInstall
      : PIP_WHEEL_DIR and PIP_FIND_LINKS will not be set in the environment.
      : `poetry install --no-root --no-dev` will be used to create and
        populate a venv within POETRY_VIRTUALENVS_PATH.
      : If python.poetry.devel is true, the `--no-dev` flag will be omitted.
      ; PhasePostInstall
      : PYTHONPATH and PATH will not be set in the environment.
      Change-Id: I33b356ff90983f8b8d5b76003851db139d97fe2e
  21. 22 Sep, 2020 1 commit
  22. 16 Sep, 2020 1 commit
    • Jeena Huneidi's avatar
      .pipeline/config.yaml: add promote step · 1fbe4e19
      Jeena Huneidi authored
      Adds a promote step to the pipeline config so that the blubber
      deployment chart will be updated with each newly published image.
      Bug: T255835
      Change-Id: I978792af67528009f7b964e9a55fcd594d44c72f
  23. 09 Sep, 2020 1 commit
    • Dduvall's avatar
      Support scratch images · b31af34e
      Dduvall authored and Dduvall's avatar Dduvall committed
      Allow for variants that have no base image (scratch images), omitting
      compilation of all features but copies and entrypoint.
      One use case for this feature is to build images of minimal size. For
      example, the following produces a production blubberoid image of only
      11M compared to the current 66M image based on WMF's stretch.
          version: v4
              base: docker-registry.wikimedia.org/golang:1.11.5-1
              apt: {packages: [gcc, git, make]}
              includes: [build]
                  CGO_ENABLED: '0'
                command: [make, blubberoid]
                requirements: [.]
                - from: prep
                  source: /srv/app/blubberoid
                  destination: /srv/app/blubberoid
              entrypoint: [/srv/app/blubberoid]
      Another possible use case would be to allow for the intermediate
      publishing of individual MW extensions and skins that have gone through
      a build phase yet require further integration downstream into fully
      deployable images that include the core platform, configuration, l10n
      database, etc.  Basing such images on scratch would drastically lower
      storage costs.
      Bug: T260830
      Change-Id: I403206981f55b59246886dfcf2bbff9c316b285b
  24. 02 Sep, 2020 1 commit
    • Jeena Huneidi's avatar
      Support PHP microservices · 8b88cb6a
      Jeena Huneidi authored
      Add ability to run 'composer install' in order to support php
      Bug: T261783
      Change-Id: I1a81378aac83db31c66fc013fc37bb1d75e41c3f
  25. 01 Sep, 2020 1 commit
    • Jeena Huneidi's avatar
      .pipeline/config.yaml: Update chart object · fb636f37
      Jeena Huneidi authored
      Removes the chart url from the chart definition and adds a name
      property. Chart registry location is now stored in PipelineLib.
      Bug: T261346
      Depends-on: Ifeb5caec6bf120b86ae16afd0315d98ffbc63dad
      Change-Id: Ied1c2403a67581c45edde4b9527a41876ae33290
  26. 28 Jul, 2020 1 commit
  27. 24 Jul, 2020 1 commit
    • Ahmon Dancy's avatar
      Fix handling of indirect 'copies' references · 5dcc587f
      Ahmon Dancy authored
      The following blubber configuration used to result in a broken
      Dockerfile which would build the 'two' stage but not the implicit
      'one' stage.  This is now fixed.
      Change-Id: I3f5d777c783de3ca113dafe5e0c728edf253db63
      version: v4
          base: docker-registry.wikimedia.org/buster-nodejs10-slim
            command: [touch, variant-one-artifact]
          base: docker-registry.wikimedia.org/releng/java8
          copies: [one]
            command: [touch, variant-two-artifact]
          base: docker-registry.wikimedia.org/releng/ci-buster
          copies: [two]
        A simple dependency graph implementation which can be used for both
        'includes' and 'copies'.
      Bug: T254629
      Change-Id: I2b11b01e27d8255bd35269d1c9f3f14c32b4e2e2
  28. 15 Jul, 2020 1 commit
  29. 18 Jun, 2020 1 commit
  30. 17 Jun, 2020 1 commit
  31. 08 Jun, 2020 1 commit
    • Jeena Huneidi's avatar
      Use ExpandIncludesAndCopies in tests · 33c79c19
      Jeena Huneidi authored
      Replaces most instances of ExpandVariant in tests with
      ExpandIncludesAndCopies, since ExpandVariant is not called outside
      of ExpandIncludesAndCopies.
      Adds a test for ExpandIncludesAndCopies.
      Change-Id: I1bdce84c1bc85c341cdb92f48d3653e75011b32b
  32. 06 Jun, 2020 1 commit
    • dcslagel's avatar
      Blubber/Blubberoid: mv expansion before verifying · 4bb38f63
      dcslagel authored
      - Separate variant expansion for 'includes' and 'copies' from
      - Move variant expansion to a separate function and preceding the
        policy check call in both blubber and blubberoid
      - Add tests for post expansion policy check
      - Add test for new GetVariant function
      Bug: T248927
      Change-Id: Id8aaccc09f81e03d205cdfdef5f99f7472c5fa12
  33. 01 May, 2020 2 commits
    • Jeena Huneidi's avatar
      PythonConfig: Change UseSystemFlag to Flag · 1429ca9e
      Jeena Huneidi authored
      Change UseSystemFlag from bool to Flag so that a false can be
      recorded when merging variants.
      Change-Id: I3e379021572cc6fcbbedb8bcfcf734c952ca0a75
    • Jeena Huneidi's avatar
      node.go: Add npm ci to build instructions · 0032f532
      Jeena Huneidi authored
      Add use-npm-ci as an option to run "npm ci" instead
      of "npm install."
      For compatibility with npm ci, change --production to
      --only=production, which was introduced in npm 6.
      Bug: T250764
      Change-Id: Ice2989675d704c9031156952332a553858a67f3f