Previously we were optimizing for cpu time, so we where building
gst-build once and then exporting that to be used by the test jobs.
However this meant that we where uploading 200mb (previously 600mb)
zipped of artifacts and then re-downloading them for each test job.
This caused big costs in terms of cloud egress since the runners
aren't hosted on the same cloud as the storage/artifacts instance.
Instead we are going to be rebuilding gst-build for each test
job from now, it also doesn't take more time than the network
i/o would of downloading the artifacts, so the impact of rebuilding
shouldn't be noticebly.
We are also using pinned git refs the modules we rebuild from
the manifest, so the binaries should be reproducible for the most
part (minus things like .pyc files).
Close#68
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/280>
docker builds can be big, and other shared runner have smaller
storage space which we should avoid filling if can be avoided.
The gst tagged runners are a better fit for such builds since
their disk storage is about 1-2 terabytes.
This avoids the need of using privilledged namespaces and dind
as buildah are able to build images unprivilledged.
One thing to note is that buildah inside docker is not a
supported configuration and not tested in upstream podman,
but the possible fallout is still easier to deal with than
dind and requiring privileged runners.
Previously we where accidently exporting the whole repo of
gst-integration-testsuites which includes 350mb of raw media
files and made the artifacts storage explode through the roof
along with the CI bills fd.o had to pay for uploading and
redownloading the artifacts
To deal with this, we clean all the media files from the builddir
and when needed we copy them over from the cache in the docker image,
and then git fetch the repo.
Close#69
We don't run the libnice testsuite, and when binaries are built
they consume ~45mb of space. This increases the size of the artifacts
we export from the gst-build job for the testsuite and drives up
the storage and bandwith costs when re-downloading the artifacts.
Similary disable the test targets of couple other subprojects as well
We have notice that a lot of CI activity is cause by user pushing to their
branch after having created an MR. To reduce our CI foot-print, the CI will
now only be automatically triggered when a reviewer assign the MR to the merge
bot. It will still be possible to run the CI manually but the result of that
CI won't be used by Marge.
It seems to be timing out with high frequency only on Windows runners.
```
Version: 12.8.0
00:47
Git revision: 1b659122
Git branch: 12-8-stable
GO version: go1.13.7
Built: 2020-02-22T03:03:07+0000
OS/Arch: windows/amd64
Uploading artifacts...
gst-build/build/meson-logs/: found 2 matching files
WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows
ERROR: Job failed (system failure): aborted: <nil>
```
See: https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/261
This might be related to the same issue described in the previous
commit: Till we can update the container image to the Feb 11 security
update, x86 executables and in general the container image will behave
badly because of:
https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t
vs2017 x86 has been failing with a runner system failure while
uploading artifacts / submitting job status:
```
Uploading artifacts...
gst-build/build/meson-logs/: found 2 matching files
WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows
ERROR: Job failed (system failure): aborted: <nil>
```
https://gitlab.freedesktop.org/slomo/gst-plugins-good/-/jobs/2084184
Disable it for now.
This will reduce the excessive load on the runners which are having issues
with this job in particuliar. We will revisit when we better understand the
runners issues.
Passing regex as variable does not really works, we ended up matching the
regex as a string instead. Replace all REGEX variable with rules: override.
It is longer but more reliable.
Related to !247Fixes#63
Rules is a new feature that replaces only/except and allow for finer grain
control on the workflow. With rules, we gain finer grain to pipeline and merge
request pipelines.
The windows runner has become a bit unstable lately, might be
due to some recent update. It frequently timeouts while waiting
to pick up a job or sometimes it goes missing in the middle of a job.
Declare an docker build-arg [1] and use it
whenever cloning one of our repositories. If the buildarg
is not specified, the variable defaults back to 'master'
and thus the current behavior doesn't change.
From the .gitlab-ci.yml file, when building pass
the GST_UPSTREAM_BRANCH that's defined from the ci_template
as the buildarg so we will be building the corresponding branches
for the docker images.
Close#33
[1] https://docs.docker.com/engine/reference/builder/#arg
This is where the WINEPREFIX is now in Cerbero. This used to be
share/wine, but was moved to var/tmp/wine for clarity. It was causing
two problems:
1. The size of these generated files are ~1GB, which were ~500MB after
tar.gz, and they were completely useless since they can just be
regenerated by Wine the next time it's run. Let's not waste egress
bandwidth.
2. Random build failures because wineserver and associated processes
would not always exit before we started tarring up the prefix, then
write to the directory on exit while tar was reading the directory
causing `tar -czf` to fail:
```
$ tar -C ${CERBERO_HOME} -czf $CERBERO_DEPS build-tools build-tools.cache dist/${ARCH} ${ARCH}.cache
tar: build-tools/share/wine: file changed as we read it
Uploading artifacts...
manifest.xml: found 1 matching files
cerbero-build/logs: found 461 matching files
cerbero-build/cerbero-deps.log: found 1 matching files
cerbero-deps.tar.gz: found 1 matching files
Uploading artifacts to coordinator... ok id=1807197 responseStatus=201 Created token=4_qFUP8z
ERROR: Job failed: exit code 1
```
