Commit graph

495 commits

Author SHA1 Message Date
Jordan Petridis
f546834e6c ci_template: enable the windows x86 and msys native jobs again
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/281>
2020-04-23 07:57:02 +00:00
Jordan Petridis
b36d813f5b ci_template: don't use prebuilt binaries for test jobs
Previously we were optimizing for cpu time, so we where building
gst-build once and then exporting that to be used by the test jobs.
However this meant that we where uploading 200mb (previously 600mb)
zipped of artifacts and then re-downloading them for each test job.
This caused big costs in terms of cloud egress since the runners
aren't hosted on the same cloud as the storage/artifacts instance.

Instead we are going to be rebuilding gst-build for each test
job from now, it also doesn't take more time than the network
i/o would of downloading the artifacts, so the impact of rebuilding
shouldn't be noticebly.

We are also using pinned git refs the modules we rebuild from
the manifest, so the binaries should be reproducible for the most
part (minus things like .pyc files).

Close #68

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/280>
2020-04-22 09:23:18 +03:00
Matthew Waters
0c315875be oss-fuzz: update for autotools removal 2020-04-21 08:47:28 +00:00
Jordan Petridis
f3aedd930d build docker image: tag the jobs to use the gstreamer runners
docker builds can be big, and other shared runner have smaller
storage space which we should avoid filling if can be avoided.
The gst tagged runners are a better fit for such builds since
their disk storage is about 1-2 terabytes.
2020-04-20 07:04:16 +00:00
Jordan Petridis
dd69902fbf Replace docker-in-docker with buildah
This avoids the need of using privilledged namespaces and dind
as buildah are able to build images unprivilledged.

One thing to note is that buildah inside docker is not a
supported configuration and not tested in upstream podman,
but the possible fallout is still easier to deal with than
dind and requiring privileged runners.
2020-04-20 07:04:16 +00:00
Nicolas Dufresne
95f9b25683 ci: Always allow redirect when using curl
This done by specifying -L parameter. Some of the FDO infrastructure is using
redirect.
2020-04-19 14:30:43 -04:00
Jordan Petridis
815976e4aa
ci_template: avoid exporting the mediafiles into build artifacts
Previously we where accidently exporting the whole repo of
gst-integration-testsuites which includes 350mb of raw media
files and made the artifacts storage explode through the roof
along with the CI bills fd.o had to pay for uploading and
redownloading the artifacts

To deal with this, we clean all the media files from the builddir
and when needed we copy them over from the cache in the docker image,
and then git fetch the repo.

Close #69
2020-04-17 22:46:34 +03:00
Jordan Petridis
7676363645
ci_template: avoid building libnice tests and examples in the subproject
We don't run the libnice testsuite, and when binaries are built
they consume ~45mb of space. This increases the size of the artifacts
we export from the gst-build job for the testsuite and drives up
the storage and bandwith costs when re-downloading the artifacts.

Similary disable the test targets of couple other subprojects as well
2020-04-17 21:59:10 +03:00
Nicolas Dufresne
8cb3a84d97 Don't let marge run post-merge pipeline
This is a small regression from previous change.
2020-04-17 13:48:18 -04:00
Jordan Petridis
26c45d6ca3 clone_manifest_ref: add the ability to fetch instead of cloning the repo 2020-04-17 05:30:01 +00:00
Nicolas Dufresne
d692406ab1 ci: Let Marge the merge bot trigger our CI
We have notice that a lot of CI activity is cause by user pushing to their
branch after having created an MR. To reduce our CI foot-print, the CI will
now only be automatically triggered when a reviewer assign the MR to the merge
bot. It will still be possible to run the CI manually but the result of that
CI won't be used by Marge.
2020-04-16 13:04:31 -04:00
Jordan Petridis
2871b293e1
ci_template: reduce the size of exported artifacts in gst-build jobs
This is initial work looking over the deps subprojects and
cleaning up the low hanging fruits. Saves about 90mb compressed.
2020-04-10 21:57:45 +03:00
Jordan Petridis
42830e91f6
windows: udpate the docker image
This is a rebuild of the image to acount for an issue
with a security update during February 2020

https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t

This also means we can drop the gstreamer-windows tag
that was added in 32c91139a4
2020-04-07 18:55:56 +03:00
Jordan Petridis
ae1b75b51f
windows: switch the image tag to ltsc2019
This should have no functional difference as 1809 is an alias
to ltsc2019 but it makes it more explicit that what branch we
are tracking
2020-04-07 14:45:08 +03:00
Matthew Waters
45d4e8ebc0 Update cerbero and android docker images
Partial revert of 9ed6305e1f

For post-merge of android ndk-21 as part of:
https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/266
https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/463
2020-04-06 21:52:59 +10:00
Matthew Waters
3b273b01e8 Partial revert of 9ed6305e1f
For post-merge of android ndk-21 as part of:
https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/266
https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/463
2020-04-06 18:38:41 +10:00
Matthew Waters
9ed6305e1f docker/cerbero: checkout proposed ndk-21 cerbero branch 2020-04-06 11:50:21 +10:00
Matthew Waters
afab7cc48b docker: update android image container for ndk-21 2020-04-06 11:50:21 +10:00
Nicolas Dufresne
e137093def ci: Update to use new external storage for cerbero deps
The goal of this change is to reduce egress on the FDO servers.
2020-04-03 16:31:45 -04:00
Matthew Waters
b00c68e69e gitlab: don't upload gst-build artifacts we never use 2020-04-02 23:42:41 +11:00
Matthew Waters
18f285f15c gitlab: compress deps using xz on master
Results in a better compression for artifacts that are going to be used
multiple times.
2020-03-31 13:30:22 +11:00
Nirbheek Chauhan
6c379762c2 gst-build: Disabling uploading of artifacts for now
It seems to be timing out with high frequency only on Windows runners.

```
Version:      12.8.0
00:47
Git revision: 1b659122
Git branch:   12-8-stable
GO version:   go1.13.7
Built:        2020-02-22T03:03:07+0000
OS/Arch:      windows/amd64
Uploading artifacts...
gst-build/build/meson-logs/: found 2 matching files
WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows
ERROR: Job failed (system failure): aborted: <nil>
```

See: https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/261
2020-03-28 16:34:03 +05:30
Seungha Yang
e0544b256f ci_template: Increase timeout of -good valgrind
splitmux unit test includes encoding task which might require more time
to finish test
2020-03-28 18:15:47 +09:00
Nirbheek Chauhan
a5686ae478 gst-build: Disable vs2017 x86 job too
This might be related to the same issue described in the previous
commit: Till we can update the container image to the Feb 11 security
update, x86 executables and in general the container image will behave
badly because of:

https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t

vs2017 x86 has been failing with a runner system failure while
uploading artifacts / submitting job status:

```
Uploading artifacts...
gst-build/build/meson-logs/: found 2 matching files
WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows
ERROR: Job failed (system failure): aborted: <nil>
```

https://gitlab.freedesktop.org/slomo/gst-plugins-good/-/jobs/2084184

Disable it for now.
2020-03-27 20:54:28 +05:30
Nirbheek Chauhan
32c91139a4 gst-build: Only use gstreamer runners for windows jobs
Till we can update the container image to the Feb 11 security update,
all x86 executables will crash because of:

https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t

So restrict to the gstreamer runners that are running an older Windows
version. For example:

```
subprojects\gstreamer\gst\parse\meson.build:10:0: ERROR: Command
  "c:\python38\python.exe
  C:\builds\thaytan\gst-plugins-good\gst-build\subprojects\gstreamer\gst\parse\get_flex_version.py
  C:\builds\thaytan\gst-plugins-good\gst-build\subprojects\win-flex-bison-binaries\win_flex.exe"
  failed with status 1.
```

https://gitlab.freedesktop.org/thaytan/gst-plugins-good/-/jobs/2066909

```
meson.build:1:0: ERROR: Executables created by c compiler cl are not runnable.
```

https://gitlab.freedesktop.org/nirbheek/gst-ci/-/jobs/2070394
2020-03-26 19:57:18 +05:30
Nirbheek Chauhan
3d06b08d49 gst-build: Always include meson-logs in the artifacts
Can't figure out build errors without this.
2020-03-26 18:56:38 +05:30
Nicolas Dufresne
8ba527416c ci: Re-enable example builds for MR
As we should still build the examples to make sure they are not broken.

Reported-by: Matthew Waters <matthew@centricular.com>
2020-03-25 17:22:09 -04:00
Nicolas Dufresne
54d94f7444 ci: Always run CI for gst-docs
We have an external cron that rely on this to pull and update the doc on the
webpage.

Fixes #64
2020-03-25 14:36:48 -04:00
Nicolas Dufresne
70ac883275 ci: Remove uneeded cerbero jobs from gst-docs CI 2020-03-25 14:00:20 -04:00
Guillaume Desmottes
9405f9245b docker: install intel-mediasdk on Fedora image
Will be used to build gst-msdk in gst-plugins-bad.
Fix #52
2020-03-24 16:33:04 +01:00
Guillaume Desmottes
d016fa7453 docker: fedora: update to Fedora 31
aom-extra-tools has been removed from Fedora:
038d7b2164

gstreamer1-plugins-bad-nonfree has been removed from rpmfusion:
https://pkgs.rpmfusion.org/cgit/nonfree/gstreamer1-plugins-bad-nonfree.git/commit/?id=30641c2ded63d12e099d32cb814bab8244f48c90
It was only used of its build deps of which only faac-devel was not
already installed, so I added it instead.

Need to manually install pip3 now.
2020-03-24 14:15:21 +00:00
Nicolas Dufresne
cf21e723e3 ci: Switch msys2 job to manual
This will reduce the excessive load on the runners which are having issues
with this job in particuliar. We will revisit when we better understand the
runners issues.
2020-03-23 15:35:35 -04:00
Nicolas Dufresne
a8941a8566 ci: Replace REGEX variables
Passing regex as variable does not really works, we ended up matching the
regex as a string instead. Replace all REGEX variable with rules: override.
It is longer but more reliable.

Related to !247
Fixes #63
2020-03-18 09:45:31 -04:00
Guillaume Desmottes
bbc624175d ci: disable Rust jobs for now
They are not reliable enough (#53) and generated artifacts are too big (#61)
so disable them until those issues are solved.

Fix https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/issues/97
2020-03-18 09:00:17 +01:00
Nicolas Dufresne
21584e4610 ci: Fix gstreamer-full job to run in gst-build
It was set to gst-omx by accident instead.
2020-03-17 21:38:34 -04:00
Nicolas Dufresne
c02cb97dc2 ci: Fix exceptions for gst indent job
The name of the variable passing the exception regex wasn't correct. As a side
effect, gst-indent was not run on all repository.
2020-03-17 21:27:09 -04:00
Nicolas Dufresne
45901ee69b ci: Remove spurious except regex in valgrind test
This was ported but unused from the previous code. In the new code it was
preventing valgrind jobs from being run in gstreamer repository.
2020-03-17 11:59:34 -04:00
Nicolas Dufresne
ee77993ccd ci: Remove outdated commemt
We no longer use awk or cat int hat context.
2020-03-17 11:58:33 -04:00
Nicolas Dufresne
8b8f1e0dfe ci: Cleanup gst-omx from unneeded jobs 2020-03-17 09:37:00 -04:00
Nicolas Dufresne
2fca5b7981 Only run CI automatically on merge_request 2020-03-17 09:32:44 -04:00
Nicolas Dufresne
f551f4a7be Make jobs interruptible by default
This was added in 12.3 and is false by default. Make it to true, so that we
stop the previous pipelien on force push.
2020-03-17 09:32:44 -04:00
Nicolas Dufresne
e12ea82a23 Port from only/except to rules
Rules is a new feature that replaces only/except and allow for finer grain
control on the workflow. With rules, we gain finer grain to pipeline and merge
request pipelines.
2020-03-17 09:32:44 -04:00
Xavier Claessens
b0eaeb69f3 gst-full: Only build 1 plugin and disable all artifacts
This is enough to validate that it works for now.

Fixes: #60
2020-03-16 21:01:06 -04:00
Jordan Petridis
d8c5432793 ci_template: extend the scenarios where retries are attempted
The windows runner has become a bit unstable lately, might be
due to some recent update. It frequently timeouts while waiting
to pick up a job or sometimes it goes missing in the middle of a job.
2020-03-10 12:25:46 +00:00
Guillaume Desmottes
604928a40b gitlab: build gst-omx with tizonia as target
Build just the minimum to be able to build gst-omx with this specific
target.
2020-03-10 03:48:14 +00:00
Jordan Petridis
a99cdc8909 docker: allow to specify the branch to pull for various repos
Declare an docker build-arg [1] and use it
whenever cloning one of our repositories. If the buildarg
is not specified, the variable defaults back to 'master'
and thus the current behavior doesn't change.

From the .gitlab-ci.yml file, when building pass
the GST_UPSTREAM_BRANCH that's defined from the ci_template
as the buildarg so we will be building the corresponding branches
for the docker images.

Close #33

[1] https://docs.docker.com/engine/reference/builder/#arg
2020-03-08 18:53:23 +00:00
Nirbheek Chauhan
3608d69c1f gitlab/cerbero: Use new prefix for jobs built with MinGW 2020-03-05 08:39:17 +00:00
Xavier Claessens
28881dddb4 Add integration test for gstreamer-full-1.0 2020-03-04 18:50:21 +00:00
Nirbheek Chauhan
7d31018ce9 gitlab/cerbero: Do not add tmp files to cerbero-deps tarball
This is where the WINEPREFIX is now in Cerbero. This used to be
share/wine, but was moved to var/tmp/wine for clarity. It was causing
two problems:

1. The size of these generated files are ~1GB, which were ~500MB after
   tar.gz, and they were completely useless since they can just be
   regenerated by Wine the next time it's run. Let's not waste egress
   bandwidth.

2. Random build failures because wineserver and associated processes
   would not always exit before we started tarring up the prefix, then
   write to the directory on exit while tar was reading the directory
   causing `tar -czf` to fail:

```
$ tar -C ${CERBERO_HOME} -czf $CERBERO_DEPS build-tools build-tools.cache dist/${ARCH} ${ARCH}.cache
tar: build-tools/share/wine: file changed as we read it
Uploading artifacts...
manifest.xml: found 1 matching files
cerbero-build/logs: found 461 matching files
cerbero-build/cerbero-deps.log: found 1 matching files
cerbero-deps.tar.gz: found 1 matching files
Uploading artifacts to coordinator... ok            id=1807197 responseStatus=201 Created token=4_qFUP8z
ERROR: Job failed: exit code 1
```
2020-03-04 14:33:16 +00:00
Matthew Waters
fbd1eb9283 cerbero-android: don't upload the '-runtime' package
It's useless for android
2020-03-04 11:46:50 +11:00