It seems to be timing out with high frequency only on Windows runners.
```
Version: 12.8.0
00:47
Git revision: 1b659122
Git branch: 12-8-stable
GO version: go1.13.7
Built: 2020-02-22T03:03:07+0000
OS/Arch: windows/amd64
Uploading artifacts...
gst-build/build/meson-logs/: found 2 matching files
WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows
ERROR: Job failed (system failure): aborted: <nil>
```
See: https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/261
This might be related to the same issue described in the previous
commit: Till we can update the container image to the Feb 11 security
update, x86 executables and in general the container image will behave
badly because of:
https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t
vs2017 x86 has been failing with a runner system failure while
uploading artifacts / submitting job status:
```
Uploading artifacts...
gst-build/build/meson-logs/: found 2 matching files
WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows
ERROR: Job failed (system failure): aborted: <nil>
```
https://gitlab.freedesktop.org/slomo/gst-plugins-good/-/jobs/2084184
Disable it for now.
This will reduce the excessive load on the runners which are having issues
with this job in particuliar. We will revisit when we better understand the
runners issues.
Passing regex as variable does not really works, we ended up matching the
regex as a string instead. Replace all REGEX variable with rules: override.
It is longer but more reliable.
Related to !247Fixes#63
Rules is a new feature that replaces only/except and allow for finer grain
control on the workflow. With rules, we gain finer grain to pipeline and merge
request pipelines.
The windows runner has become a bit unstable lately, might be
due to some recent update. It frequently timeouts while waiting
to pick up a job or sometimes it goes missing in the middle of a job.
This is where the WINEPREFIX is now in Cerbero. This used to be
share/wine, but was moved to var/tmp/wine for clarity. It was causing
two problems:
1. The size of these generated files are ~1GB, which were ~500MB after
tar.gz, and they were completely useless since they can just be
regenerated by Wine the next time it's run. Let's not waste egress
bandwidth.
2. Random build failures because wineserver and associated processes
would not always exit before we started tarring up the prefix, then
write to the directory on exit while tar was reading the directory
causing `tar -czf` to fail:
```
$ tar -C ${CERBERO_HOME} -czf $CERBERO_DEPS build-tools build-tools.cache dist/${ARCH} ${ARCH}.cache
tar: build-tools/share/wine: file changed as we read it
Uploading artifacts...
manifest.xml: found 1 matching files
cerbero-build/logs: found 461 matching files
cerbero-build/cerbero-deps.log: found 1 matching files
cerbero-deps.tar.gz: found 1 matching files
Uploading artifacts to coordinator... ok id=1807197 responseStatus=201 Created token=4_qFUP8z
ERROR: Job failed: exit code 1
```
This is slightly weird, cause I am not sure what causes the clone
to be there, since gitlab-runner supposedly always either use a
clean volume or at least runs git clean on the existing ones.
But its there and so we have to deal with failures like so
https://gitlab.freedesktop.org/tpm/gstreamer-sharp/-/jobs/1672137
That job was the slowest, now each jobs takes about 12 minutes, which
makes it slightly faster then msys2 jobs, and sometime iOS due to low
bandwidth and low availibility of OSX runners.
CCache tends to consume a lot of space which taxes heavily some
of the shared runners. Limit the mahcines the job can run
to those were we can ensure they will not have issues with
the storage.
```
At line:1 char:34
+ cd $env:CI_PROJECT_DIR/gst-build && python git-update --no-interactio ...
+ ~~
The token '&&' is not a valid statement separator in this version.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordEx
ception
+ FullyQualifiedErrorId : InvalidEndOfLine
```
This is not bash, but powershell, hue hue hue
Rebuild the windows docker image against the current ltsc [1]
of server 2019. This requires moving some of the msys setup
to the runner job cause it causes docker build to hang
Switch the job tags so they now use the 1809 runner, instead
of 1607.
Tweak the PATHs in the msys job so bash doesn't complain about
slashes..
Lastly, increase the timeout of the windows jobs, as msys2
installs its deps at runtime
[1] https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19
In case a build gets stuck for whatever reason,
happens from time to time on windows,
try to baild out quickly.
For cerbero builds, set the timeout to 3h which
according to the docs should also be able to
override the project defined timeout
Fix#19https://docs.gitlab.com/ce/ci/yaml/README.html#timeout
We have lots of tests that timeout on the CI due to a high load
of jobs on the CI runners. Let's try giving them a bit more time
and see how its going.
Check tests are being added to gstreamer/gstreamer-vaapi!181.
However, gstreamer-vaapi inherently requires specific hardware
drivers and platforms to function. The CI does not provide this
level of driver/platform selection. Thus, avoid running any
check tests in gstreamer-vaapi.
We are only using the builddir of the main fedora job to run
tests, the rest where exported by accident. Its especially
problematic cause static build eat a bunch of space and take
an eternity to be uploaded.
Part of #32
* Install git-lfs as its required now by gst-integration-suites
* Clone gst-build eache time to avoid dated gst-build checkouts
and overwritting .wrap files. Similar to !137
* Split the dockerfile and add a second run stage refresh the
powershell env inbetween calls
* Remove the msys2 workaround as its not needed anymore
This adds 3 new jobs that build against msys2 x86_64, msvc 2017 x86
and msvc 2017 x86_64. For the msvc build, some subprojects (like libnice)
don't satisfy all their deps, and are getting automatically disabled.
This doesn't add jobs that run the test suite also. Will hopefully
get implemented later on.
The git-update that is performed attempts to update gst-build however
will not use the updated git-update script for further operations. This
causes the CI to not use any updates to the git-update which is always
stuck on the version provided by the backing docker image.
The valgrind runs are there to spot obvious problems during the dev phase,
not sure we really need to run them in full after each merge.
Should reduce load on the build bots a little.
If any problems slip in they will be picked up soon enough by
the MR jobs again.
When the pipeline is based on top of a stable branch, we want
to track that branch isntead of the primary development branch.
This patch makes it so the upstream branch can be specified with
an env var.
part of #11
We exclude cerbero in the .build template but that key
is getting overwritten since !126. Valgrind needs to be
fixed first in order to remove this and the previous
workarounds.
For non-cerbero builds, pick a cerbero reference for which a build has
completed. This will reduce the number of cache miss, hence reduce the
number of timeouts and slow build we are facing each time cerbero is
updated.
Fixes#16
Don't valgrind everything for all changes though,
but only those modules most likely to be affected
by changes in the current ci project. So, valgrind
- gstreamer only for gstreamer core changes
- gst-plugins-base for core/base changes
- gst-plugins-good for core/base/good changes
- gst-plugins-ugly for core/base/ugly changes
- gst-plugins-bad for core/base/good changes
In other words: don't valgrind core/base if it's
good/bad/ugly that's being changed, for example.
Don't valgrind -good or -ugly for changes in -bad,
etc.
Meaning:
- for changes in core, valgrind core/base/good/ugly/bad
- for changes in base, valgrind base/good/ugly/bad
- for changes in good, valgrind good/bad
- for changes in ugly, valgrind ugly
- for changes in bad, valgrind bad
More modules to be added later once they're valgrind
clean on the CI.
https://gitlab.freedesktop.org/gstreamer/gst-ci/merge_requests/80
