CI.md 3.48 KB
Newer Older
Addshore's avatar
Addshore committed
1
2
# CI

Lens0021's avatar
Lens0021 committed
3
Continuous integration for this project is currently set up on a dedicated Cloud VPM machine.
Addshore's avatar
Addshore committed
4
5
6

Currently this CI will NOT work for forks of this project, only for actual project branches.

7
There are currently 2 runners:
Addshore's avatar
Addshore committed
8
 - gitlab-runner-addshore-1013.mwcli.eqiad1.wikimedia.cloud
Addshore's avatar
Addshore committed
9
 - gitlab-runner-addshore-1014.mwcli.eqiad1.wikimedia.cloud
10

Addshore's avatar
Addshore committed
11
12
13
14
15
16
17
18
19
## Maintenance

If the runner starts running out of space...

```sh
sudo docker system prune --force
sudo docker volume prune
```

Lens0021's avatar
Lens0021 committed
20
If this doesn't free up enough space the next step would be to nuke the registry container and volume and recreate it!
Addshore's avatar
Addshore committed
21
22
23
24
25

## Initial Setup

### Make a machine

Addshore's avatar
Addshore committed
26
Make a VM, such as `gitlab-runner-addshore-1013.mwcli.eqiad1.wikimedia.cloud`
Addshore's avatar
Addshore committed
27
28
29
30
31

### Install docker

```sh
sudo apt-get update
Addshore's avatar
Addshore committed
32
sudo apt-get remove docker docker-engine docker.io containerd runc
Addshore's avatar
Addshore committed
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
```

### Install gitlab runner

```sh
curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
sudo dpkg -i gitlab-runner_amd64.deb
54
rm gitlab-runner_amd64.deb
Addshore's avatar
Addshore committed
55
56
57
58
59
60
61
62
63
```

### Register the runner

```sh
sudo gitlab-runner register -n \
  --url https://gitlab.wikimedia.org/ \
  --registration-token XXXreleng-mwcli-tokenXXX \
  --executor docker \
64
  --limit 2 \
Addshore's avatar
Addshore committed
65
  --name "gitlab-runner-addshore-1013-docker-01" \
Addshore's avatar
Addshore committed
66
  --docker-image "docker:20.10.14" \
Addshore's avatar
Addshore committed
67
68
69
70
71
72
73
74
  --docker-privileged \
  --docker-volumes "/certs/client"
```

### Extra configuration

#### Configure "global" runner jobs

75
76
Allow 2 jobs at once globally on this runner and restart gitlab runner.
(Any more than this and things get slow, timeout, use too much storage, fail etc)
Addshore's avatar
Addshore committed
77
78

```sh
79
sudo sed -i 's/^concurrent =.*/concurrent = 2/' "/etc/gitlab-runner/config.toml"
Addshore's avatar
Addshore committed
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
sudo systemctl restart gitlab-runner
```

#### Register custom local docker mirror

Mainly from https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/

Create a mirror (using docker):

```sh
sudo docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    --restart always \
    --name registry registry:2
```

Get the IP address:

```sh
hostname --ip-address
```

Addshore's avatar
Addshore committed
102
Add the mirror (You might need to do this as root, not sudo...):
Addshore's avatar
Addshore committed
103
104

```sh
Addshore's avatar
Addshore committed
105
106
# If sudo doesn't work for the file change you may need to sudo su, and then run the echo...
sudo echo '{"registry-mirrors": ["http://<CUSTOM IP>:6000"]}' > /etc/docker/daemon.json
Addshore's avatar
Addshore committed
107
108
109
110
111
112
sudo service docker restart
```

Check with:

```sh
Addshore's avatar
Addshore committed
113
sudo docker system info
Addshore's avatar
Addshore committed
114
115
116
117
118
```

Also add the mirror for dind in `/etc/gitlab-runner/config.toml` to each runner it is needed for
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#enable-registry-mirror-for-dockerdind-service

Addshore's avatar
Addshore committed
119
120
You can also tweak the pull_policy to fallback to "if-not-present".

Addshore's avatar
Addshore committed
121
```sh
Addshore's avatar
Addshore committed
122
123
  [[runners.docker]]
    pull_policy = ["always", "if-not-present"]
Addshore's avatar
Addshore committed
124
    [[runners.docker.services]]
Addshore's avatar
Addshore committed
125
126
      name = "docker:20.10.14-dind"
      command = ["--registry-mirror", "http://172.16.7.194:6000"]
Addshore's avatar
Addshore committed
127
128
129
130
131
132
```

And restart the gitlab runner service:

```sh
sudo systemctl restart gitlab-runner
Lens0021's avatar
Lens0021 committed
133
```