🚧 This instance is under construction; expect occasional downtime. Runners available in /repos. Questions? Ask in #wikimedia-gitlab on libera.chat, or under GitLab on Phabricator.

  1. 20 Apr, 2021 1 commit
    • Dduvall's avatar
      user: Check for existing user/group before creating · 459234d2
      Dduvall authored
      To support scenarios where the `runs.as` or `lives.as` user/group
      already exists in the base image, pair the calls to `useradd` and
      `groupadd` with `getent` conditionals.
      
      Bug: T268819
      Change-Id: I3aef722ac57b38695c7411e64b41589fad16e95b
      459234d2
  2. 18 Mar, 2021 1 commit
  3. 12 Mar, 2021 1 commit
    • Dduvall's avatar
      apt: Support configuration of http/https proxies · 7bb48691
      Dduvall authored
      Defines a new `apt.proxies` field that allows users to specify
      http/https proxy URLs that apply to one or all APT sources. APT
      configuration for the proxies will be written prior to package
      installation and then removed.
      
      Both a shorthand and longhand configuration are supported.
      
      Example:
      
          apt:
            packages: [foo]
            proxies: [http://proxy.example:8080]
      
      Is a shorthand equivalent of:
      
          apt:
            packages: [foo]
            proxies:
              - url: http://proxy.example:8080
      
      In which case, the following APT config is written before installing
      packages, proxying all http sources via `http://proxy.example:8080`.
      
          Acquire::http::Proxy "http://proxy.example:8080";
      
      Example:
      
          apt:
            packages: [foo]
            proxies:
              - url: https://proxy.example:8081
                source: https://security.debian.org
      
      In which case, the following APT config is written before installing
      packages, proxying only https requests to security.debian.org via
      `https://proxy.example:8081`.
      
          Acquire::https::Proxy::security.debian.org "https://proxy.example:8081";
      
      Other changes include a refactoring of the `AptConfig.Packages` type and
      unmarshalling function. A formal type `AptPackages` is now defined,
      derived from `map[string][]string`, and the `UnmarshalJSON` member
      function moved there. This was done to avoid bloating of the general
      `AptConfig` unmarshal function with implementation for both fields
      (`packages` and `proxies`).
      
      The new `AptConfig.Proxy` type defines its own unmarshal function to
      support both shorthand and longhand configurations.
      
      Bug: T277109
      Change-Id: I5b82efcc441e48890cfab62747519d8986d7e8ac
      7bb48691
  4. 23 Feb, 2021 1 commit
    • Dduvall's avatar
      copies: Allow copying directly from other images · db3a5397
      Dduvall authored
      To support image compositions that rely on copying files from one or
      more external images, allow `ArtifactsConfig.From` to be any valid image
      reference.
      
      Generalize `baseimage` validator as `imageref` and create a new
      `artifactfrom` validator alias for `imageref|variantref`. Use the new
      validator for `From`.
      
      Additionally, omits non-variant `From` references from the dependency
      graph.
      
      Change-Id: I3b92d815b62856fe5a8df836b37cce0043a74ffa
      db3a5397
  5. 12 Feb, 2021 1 commit
    • BryanDavis's avatar
      python: upgrade pip before installing requirements · 007009ea
      BryanDavis authored
      A `python` config block already triggered installing pip via
      easy_install which is pretty specific to python2 era usage. It then used
      pip to install/update pypi packages for setuptools, wheel, and tox. This
      change adds the pip package to that set of pypi install/update packages.
      This is useful to ensure that the resulting image is capable of
      installing manylinux2014 formatted wheels (pip >= 19.3). Debian Buster's
      python3-pip package is pip 18.1 which is too old to look for
      manylinux2014 wheels when installing packages such as cryptography which
      have stopped publishing wheels with the older manylinux1 and
      manylinux2010 platform tags.
      
      Existing pinning of pip to a version less than 21.0 is preserved for
      those poor, unloved python2 projects that linger in an ever colder and
      darker world hostile to their kind.
      
      Bug: T274435
      Change-Id: I81f0af69dbd4f9202bc9736faf653813c81fc0ad
      007009ea
  6. 04 Feb, 2021 1 commit
    • Dduvall's avatar
      python: Pin pip package to <21 for Python 2 · d91393a4
      Dduvall authored
      Pip 21 has dropped support for Python 2 but is still strangely
      selected for installation when running setuptools with a python2 binary.
      Let's pin pip when a `version` is specified that starts with "python2".
      
      Note this will require python 2 users to explicitly specify `version:
      python2` even if their default `python` binary in the base image is for
      python 2.
      
      Bug: T273793
      Change-Id: Id7d4315e345657ce7ac2efdd0b065d693e47b2f5
      d91393a4
  7. 03 Feb, 2021 1 commit
  8. 02 Feb, 2021 2 commits
  9. 01 Feb, 2021 1 commit
    • Jeena Huneidi's avatar
      apt.go: Add ability to target releases · eb038f2c
      Jeena Huneidi authored
      In order to allow packages from backports to be downloaded, the ability
      to target a release when running apt-get install is required.
      
      apt.packages now takes the previously defined list of string or a map
      of targets and packages to install. When using a map, in order to
      install the packages for the base image, the 'default' key should be
      used:
      
      packages:
        default: ["libfoo", "libbar"]
        buster-backports: ["libbaaz"]
      
      Bug: T272759
      Change-Id: I435c47794ffbc0264e0a440bfbd23e11570645d1
      eb038f2c
  10. 26 Jan, 2021 1 commit
  11. 15 Jan, 2021 1 commit
  12. 18 Dec, 2020 1 commit
    • Dduvall's avatar
      Refactor other builder types to use RequirementsConfig · 18c9968e
      Dduvall authored
      Copying of required files is now a generic operation implemented by
      `config.RequirementsConfig`. Other builder types should re-use this
      implementation.
      
      To support this change, some `RequirementsConfig` implementation around
      source and destination paths was moved to functions of
      `ArtifactsConfig`.
      
      Build macro functions `SortFilesByDir` and `SyncFiles` are no longer
      needed and were removed.
      
      Change-Id: Ieb0cd2a0b1e4f11b05d19c63bea9bdf9d578e3ea
      Follows-up: If813829bdace6851bdba56abcdfcab1cd967df03
      18c9968e
  13. 16 Dec, 2020 1 commit
  14. 15 Dec, 2020 1 commit
  15. 14 Dec, 2020 1 commit
    • BryanDavis's avatar
      requirements: Fix regression in short form handling · f063588e
      BryanDavis authored
      Flip the order of `path.Clean()` and `path.Dir()` in the NewFromShort()
      constructor. The prior order was stripping directory indicators from the
      source path prematurely.
      
      Bug: T263597
      Change-Id: I0d76b4632660a8c32fb1c8a7a90b287546271348
      f063588e
  16. 11 Dec, 2020 1 commit
    • Dduvall's avatar
      Make artifact destination optional · 058c2d54
      Dduvall authored
      Destination is now an optional field, defaulting to the source path
      when omitted.
      
      This makes for less redundancy when defining cross-variant builder
      requirements and copies in cases where the application directories
      are the same between variants.
      
      Change-Id: I44cc3fdff260670b07b6cb3d82d4d7a49ff221f7
      058c2d54
  17. 10 Dec, 2020 2 commits
    • BryanDavis's avatar
      builder: support cross variant copying for builder.requirements · a9a61206
      BryanDavis authored
      Allow `builder.requirements` to use a long form similar to `copies` when
      the user needs to copy files from one variant to another as part of the
      builder stage. This allows multi-stage builds to generate some artifact
      in stage "A" and then copy that artifact into stage "B" in the
      PreInstall phase before running `builder.command`.
      
      Usage example:
      ```
      variants:
        A:
          apt:
            packages:
              - wget
          builder:
            command:
              - /bin/bash
              - -c
              - >-
                wget https://example.org/some_cool_utility.tgz
      
      
                && tar xzvf some_cool_utility.tgz
                && rm some_cool_utility.tgz
        B:
          builder:
            requirements:
              - src
              - from: A
                source: some_cool_utility
                destination: .
            command:
              - some_cool_utility --do-something-cool src/
      ```
      
      Bug: T263597
      Co-authored-by: Dduvall's avatarDan Duvall <dduvall@wikimedia.org>
      Change-Id: If813829bdace6851bdba56abcdfcab1cd967df03
      a9a61206
    • BryanDavis's avatar
      Fix Makefile syntax for running linter & legacy errors · e81d1adf
      BryanDavis authored
      The `lint` make target contained a logic error which made make evaluate
      the test condition rather than bash. This in turn has prevented all
      gofmt warnings from being seen by developers. The fix was escaping the
      `$` character in the Makefile by doubling it.
      
      Following this fix, the existing linter errors are also fixed by this
      commit.
      
      Change-Id: Ieb0c91cbdd0b13602fba9b6f81c97d22f36a7fd7
      e81d1adf
  18. 16 Oct, 2020 1 commit
    • Dduvall's avatar
      Fix openapi spec and include variant schema · f34686ba
      Dduvall authored
      Fixes various issues with the openapi 3.0 blubberoid specification,
      includes the variant schema using `additionalProperties`'s object
      notation, and implements a test that performs validation.
      
      Change-Id: I203ff340a9753a1541512ab466ac845e0e7e4f64
      f34686ba
  19. 25 Sep, 2020 1 commit
    • BryanDavis's avatar
      python.go: Add support for Poetry package manager · 5718d4d6
      BryanDavis authored
      Poetry (<https://python-poetry.org/>) is a packaging and dependency
      management tool for Python. Poetry creates a fully versioned dependency
      tree for your project. The poetry.lock file tracks the exact versions of
      each library and its dependencies for repeatable installs.
      
      Usage example:
      ```
      variants:
        build:
          python:
            version: python3
            poetry:
              version: ==1.0.10
            requirements: [pyproject.toml, poetry.lock]
        test:
          includes: [build]
          python:
            poetry:
              devel: true
          copies: [local]
        production:
          includes: [build]
          copies: [local]
      ```
      
      python.poetry.version must be a simplified Python package version
      specification. A new "pypkgver" validator is provided to ensure that the
      value is usable for a `pip install poetry...` command. Typically it
      would be reasonable to pin the version of Poetry in use to an exact
      version as show in the sample above, but it may be desirable to use
      other more complex constraints such as ">=1.0,!=1.0.3,<2.0".
      
      python.poetry.devel is a boolean flag indicating whether or not to
      install development dependencies in the Poetry managed venv. It defaults
      to false.
      
      When python.poetry.version is non-empty various changes will be made to
      the generated Dockerfile.
      ; PhasePrivileged
      : A version of Poetry matching the python.poetry.version constraint
        will be installed using pip.
      : POETRY_VIRTUALENVS_PATH will be set in the environment.
      ; PhasePreInstall
      : PIP_WHEEL_DIR and PIP_FIND_LINKS will not be set in the environment.
      : `poetry install --no-root --no-dev` will be used to create and
        populate a venv within POETRY_VIRTUALENVS_PATH.
      : If python.poetry.devel is true, the `--no-dev` flag will be omitted.
      ; PhasePostInstall
      : PYTHONPATH and PATH will not be set in the environment.
      
      Change-Id: I33b356ff90983f8b8d5b76003851db139d97fe2e
      5718d4d6
  20. 22 Sep, 2020 1 commit
  21. 16 Sep, 2020 1 commit
    • Jeena Huneidi's avatar
      .pipeline/config.yaml: add promote step · 1fbe4e19
      Jeena Huneidi authored
      Adds a promote step to the pipeline config so that the blubber
      deployment chart will be updated with each newly published image.
      
      Bug: T255835
      Change-Id: I978792af67528009f7b964e9a55fcd594d44c72f
      1fbe4e19
  22. 09 Sep, 2020 1 commit
    • Dduvall's avatar
      Support scratch images · b31af34e
      Dduvall authored and Dduvall's avatar Dduvall committed
      Allow for variants that have no base image (scratch images), omitting
      compilation of all features but copies and entrypoint.
      
      One use case for this feature is to build images of minimal size. For
      example, the following produces a production blubberoid image of only
      11M compared to the current 66M image based on WMF's stretch.
      
          version: v4
          variants:
            build:
              base: docker-registry.wikimedia.org/golang:1.11.5-1
              apt: {packages: [gcc, git, make]}
            prep:
              includes: [build]
              runs:
                environment:
                  CGO_ENABLED: '0'
              builder:
                command: [make, blubberoid]
                requirements: [.]
            production:
              copies:
                - from: prep
                  source: /srv/app/blubberoid
                  destination: /srv/app/blubberoid
              entrypoint: [/srv/app/blubberoid]
      
      Another possible use case would be to allow for the intermediate
      publishing of individual MW extensions and skins that have gone through
      a build phase yet require further integration downstream into fully
      deployable images that include the core platform, configuration, l10n
      database, etc.  Basing such images on scratch would drastically lower
      storage costs.
      
      Bug: T260830
      Change-Id: I403206981f55b59246886dfcf2bbff9c316b285b
      b31af34e
  23. 02 Sep, 2020 1 commit
    • Jeena Huneidi's avatar
      Support PHP microservices · 8b88cb6a
      Jeena Huneidi authored
      Add ability to run 'composer install' in order to support php
      microservices.
      
      Bug: T261783
      Change-Id: I1a81378aac83db31c66fc013fc37bb1d75e41c3f
      8b88cb6a
  24. 01 Sep, 2020 1 commit
    • Jeena Huneidi's avatar
      .pipeline/config.yaml: Update chart object · fb636f37
      Jeena Huneidi authored
      Removes the chart url from the chart definition and adds a name
      property. Chart registry location is now stored in PipelineLib.
      
      Bug: T261346
      Depends-on: Ifeb5caec6bf120b86ae16afd0315d98ffbc63dad
      Change-Id: Ied1c2403a67581c45edde4b9527a41876ae33290
      fb636f37
  25. 28 Jul, 2020 1 commit
  26. 24 Jul, 2020 1 commit
    • Ahmon Dancy's avatar
      Fix handling of indirect 'copies' references · 5dcc587f
      Ahmon Dancy authored
      The following blubber configuration used to result in a broken
      Dockerfile which would build the 'two' stage but not the implicit
      'one' stage.  This is now fixed.
      
      Change-Id: I3f5d777c783de3ca113dafe5e0c728edf253db63
      
      ---
      version: v4
      
      variants:
        one:
          base: docker-registry.wikimedia.org/buster-nodejs10-slim
          builder:
            command: [touch, variant-one-artifact]
        two:
          base: docker-registry.wikimedia.org/releng/java8
          copies: [one]
          builder:
            command: [touch, variant-two-artifact]
        three:
          base: docker-registry.wikimedia.org/releng/ci-buster
          copies: [two]
      
      config/dependency_graph.go:
        A simple dependency graph implementation which can be used for both
        'includes' and 'copies'.
      
      Bug: T254629
      Change-Id: I2b11b01e27d8255bd35269d1c9f3f14c32b4e2e2
      5dcc587f
  27. 15 Jul, 2020 1 commit
  28. 18 Jun, 2020 1 commit
  29. 17 Jun, 2020 1 commit
  30. 08 Jun, 2020 1 commit
    • Jeena Huneidi's avatar
      Use ExpandIncludesAndCopies in tests · 33c79c19
      Jeena Huneidi authored
      Replaces most instances of ExpandVariant in tests with
      ExpandIncludesAndCopies, since ExpandVariant is not called outside
      of ExpandIncludesAndCopies.
      Adds a test for ExpandIncludesAndCopies.
      
      Bug:T248927
      Change-Id: I1bdce84c1bc85c341cdb92f48d3653e75011b32b
      33c79c19
  31. 06 Jun, 2020 1 commit
    • dcslagel's avatar
      Blubber/Blubberoid: mv expansion before verifying · 4bb38f63
      dcslagel authored
      - Separate variant expansion for 'includes' and 'copies' from
        docker/compose.go::Compose()
      - Move variant expansion to a separate function and preceding the
        policy check call in both blubber and blubberoid
      - Add tests for post expansion policy check
      - Add test for new GetVariant function
      
      Bug: T248927
      Change-Id: Id8aaccc09f81e03d205cdfdef5f99f7472c5fa12
      4bb38f63
  32. 01 May, 2020 2 commits
    • Jeena Huneidi's avatar
      PythonConfig: Change UseSystemFlag to Flag · 1429ca9e
      Jeena Huneidi authored
      Change UseSystemFlag from bool to Flag so that a false can be
      recorded when merging variants.
      
      Change-Id: I3e379021572cc6fcbbedb8bcfcf734c952ca0a75
      1429ca9e
    • Jeena Huneidi's avatar
      node.go: Add npm ci to build instructions · 0032f532
      Jeena Huneidi authored
      Add use-npm-ci as an option to run "npm ci" instead
      of "npm install."
      For compatibility with npm ci, change --production to
      --only=production, which was introduced in npm 6.
      
      Bug: T250764
      Change-Id: Ice2989675d704c9031156952332a553858a67f3f
      0032f532
  33. 15 Apr, 2020 1 commit
    • dcslagel's avatar
      Add golint as an optional requirement · 677ee9a8
      dcslagel authored
      golint is used in the `make lint` command and
      isn't included in Gopkg.toml for blubber. This change adds
      information on how to install it in CONTRIBUTING.md.
      
      Change-Id: I40d693fa870e1a53c3baae4b0218aa5dd99c434c
      677ee9a8
  34. 16 Dec, 2019 1 commit
    • Dduvall's avatar
      pipeline: Exclude binaries when copying source to docker images · 5591b500
      Dduvall authored
      The blubberoid binary is built within the context of the `prep` image
      variant before being copied to the `production` image. Any existing
      binaries in the local source context should be excluded from the build
      process.
      
      Change-Id: I66cd7782748493812b10f845de42997406353496
      5591b500
  35. 11 Dec, 2019 1 commit
  36. 10 Dec, 2019 1 commit
    • Dduvall's avatar
      Avoid extra image layers due to sparse user logs · 23c71563
      Dduvall authored
      Pass `-l` to `useradd` to avoid pre-initialization of `lastlog` and
      `faillog` sparse files, reducing final image layer sizes by around 42M.
      
      Change-Id: I4b301a4e3458d7d8107ff05f5233d44d7278455a
      23c71563
  37. 06 Dec, 2019 1 commit
    • Dduvall's avatar
      Fix `make clean` when project source is outside GOPATH · 6111a021
      Dduvall authored
      If the project source is outside GOPATH, invoking `go clean` without
      arguments results in "cannot find module for path .". Explicitly passing
      packages to `go clean` avoids this error.
      
      Change-Id: Ic732164917e775b5cc95534171c96e6cb377d685
      6111a021