Commit graph

74 commits

Author SHA1 Message Date
Jordan Petridis
3a004f7ca4 citemplate: use the fd.o pre existing buildah image to build containers
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
2020-06-26 12:14:28 +00:00
Jordan Petridis
cece45eacb ci_template: merge docker builds into the normal pipeline
Port the build docker jobs to the template and integrate it into
the pipeline. Then change the image uris to the new system,
and the user forked registries.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
2020-06-26 12:14:28 +00:00
Jordan Petridis
e73ced3676 .gitlab-ci.yml: kill local jobs
The idea behind local jobs was that you would have jobs that
share the same setup as the citemplate ones but would always
be pointing to your forked registry with your custom build of
the image, since the citemplate hardcoded the image it was
running against.

With the changes introduced in the previous commit
we now mirror the setup from freedesktop/ci-templates and the idea
is that jobs always run from the registry in your fork. If the
image sha/id matches the one from the upstream registry, its copied
over else a new one is build, pushed and tested.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
2020-06-26 12:14:28 +00:00
Jordan Petridis
fe959679b8 .gitlab-ci.yml: adapt the docker build jobs to be more like citemplates
This is the inital step towards migrating our docker images setup
to something closer and eventually freedesktop/citemplates [1]

The idea is that jobs always run from the registry in your fork. If the
image sha/id matches the one from the upstream registry, its copied
over else a new one is build, pushed and tested.

Only change the fedora job for now while testing.

[1]: https://gitlab.freedesktop.org/freedesktop/ci-templates

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
2020-06-26 12:14:28 +00:00
Jordan Petridis
c731fab0ca image_builds: simplify the tags
We can use docker labels to pass the metadata we want
and make the tag simpoler.

also small fixup, make it so buildah it uses chroot isolation
and update the base image

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
2020-06-26 12:14:28 +00:00
Nicolas Dufresne
f0eea49641 CI: Combine deps and build for cerbero builds
This change combine the deps and the build step when running cerbero CI in
cerbero repository itself. This removes avoidable usage of large artifacts.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/303>
2020-06-17 13:28:44 +00:00
Jordan Petridis
822706e473
.gitlab-ci.yml: fix buildah syntax for tagging images 2020-05-14 12:46:54 +03:00
Jordan Petridis
0fb6955c5e
.gitlab-ci.yml: fix typo in buildah images 2020-05-14 11:40:50 +03:00
Jordan Petridis
b36d813f5b ci_template: don't use prebuilt binaries for test jobs
Previously we were optimizing for cpu time, so we where building
gst-build once and then exporting that to be used by the test jobs.
However this meant that we where uploading 200mb (previously 600mb)
zipped of artifacts and then re-downloading them for each test job.
This caused big costs in terms of cloud egress since the runners
aren't hosted on the same cloud as the storage/artifacts instance.

Instead we are going to be rebuilding gst-build for each test
job from now, it also doesn't take more time than the network
i/o would of downloading the artifacts, so the impact of rebuilding
shouldn't be noticebly.

We are also using pinned git refs the modules we rebuild from
the manifest, so the binaries should be reproducible for the most
part (minus things like .pyc files).

Close #68

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/280>
2020-04-22 09:23:18 +03:00
Jordan Petridis
f3aedd930d build docker image: tag the jobs to use the gstreamer runners
docker builds can be big, and other shared runner have smaller
storage space which we should avoid filling if can be avoided.
The gst tagged runners are a better fit for such builds since
their disk storage is about 1-2 terabytes.
2020-04-20 07:04:16 +00:00
Jordan Petridis
dd69902fbf Replace docker-in-docker with buildah
This avoids the need of using privilledged namespaces and dind
as buildah are able to build images unprivilledged.

One thing to note is that buildah inside docker is not a
supported configuration and not tested in upstream podman,
but the possible fallout is still easier to deal with than
dind and requiring privileged runners.
2020-04-20 07:04:16 +00:00
Nicolas Dufresne
2fca5b7981 Only run CI automatically on merge_request 2020-03-17 09:32:44 -04:00
Nicolas Dufresne
e12ea82a23 Port from only/except to rules
Rules is a new feature that replaces only/except and allow for finer grain
control on the workflow. With rules, we gain finer grain to pipeline and merge
request pipelines.
2020-03-17 09:32:44 -04:00
Jordan Petridis
a99cdc8909 docker: allow to specify the branch to pull for various repos
Declare an docker build-arg [1] and use it
whenever cloning one of our repositories. If the buildarg
is not specified, the variable defaults back to 'master'
and thus the current behavior doesn't change.

From the .gitlab-ci.yml file, when building pass
the GST_UPSTREAM_BRANCH that's defined from the ci_template
as the buildarg so we will be building the corresponding branches
for the docker images.

Close #33

[1] https://docs.docker.com/engine/reference/builder/#arg
2020-03-08 18:53:23 +00:00
Nirbheek Chauhan
10326927d5 gitlab-ci: Fix docker tag when pushing image
Missed in https://gitlab.freedesktop.org/gstreamer/gst-ci/merge_requests/213
2020-01-08 23:46:35 +05:30
Nirbheek Chauhan
eedde1884a gitlab-ci.yml: Allow tagging images built on upstream branches
If we're on the gstreamer namespace, we should push all images built
on that namespace to the registry. This is needed to, f.ex., update
the docker image used in stable builds. This is needed for:
https://gitlab.freedesktop.org/gstreamer/cerbero/merge_requests/367
2020-01-07 12:58:21 +05:30
Jordan Petridis
20c2bb8080
ci_template: refresh the docker images used for the CI 2019-10-24 15:13:03 +03:00
Jordan Petridis
1e063c8539 .gitlab-ci.yml: Rename the UID variable in the docker build job
UID is a read-only protected variable in bash 5.0+
2019-09-23 10:54:40 +00:00
Jordan Petridis
c98b4accb3 .gitlab-ci.yml: export artifacts for the fedora build job
20f5d7d646 stopped exporting artifacts
from the .build template which broke the local jobs since they
no longer exported the builddir.
2019-09-23 10:54:40 +00:00
Nirbheek Chauhan
17fc9a66d4 gitlab: Rename all cross jobs to have a cross- prefix
For consistency, and to differentiate, f.ex., cross-windows jobs from
native windows jobs.
2019-09-11 13:32:56 +05:30
Matthew Waters
a596162ab5 gitlab: use the new needs yaml key
Allows implementing a DAG where a dependant job can be built before the
entirety of the previous stage has completed.
2019-08-28 11:01:18 +10:00
Matthew Waters
a9cd95f7ba gitlab: consolidate stages
This now optimizes for a successful build.
2019-06-13 06:08:01 +00:00
Thibault Saunier
3c5db7d449 Add support to build the documentation
Since we can't run valgrind tests on fedora 30 yet (see
https://gitlab.freedesktop.org/gstreamer/gst-ci/merge_requests/125),
we build the nodebug variant with the F30 image and use the artifacts
of that build to build the documentation and not the main one.
2019-05-15 18:29:32 -04:00
Thibault Saunier
68fa0afea7 Revert "Add support for building the doc"
This reverts commit 58861fd410.
2019-05-13 21:48:36 -04:00
Thibault Saunier
58861fd410 Add support for building the doc 2019-05-13 19:41:20 +00:00
Thibault Saunier
edd8ff52e3 ci: Run 'validate' and 'ges' integration testsuites
And use new validate feature to print debug logs URL directly
in the logs instead of dumping the whole files.
2019-04-03 19:13:21 -03:00
Matthew Waters
c4a3df759d add CI for gst-examples android build 2019-03-27 12:11:31 +11:00
Jordan Petridis
5fe97b9f1c
registry: Change the way in which images are tagged
Previously we tried using the git commit ref as a UID for the
images. This does not work though cause multiple jobs that rebuild
the image can be triggered and override the same image tag. Instead
use the CI_JOB_ID to provide better semantics wrt naming conflicts.

Additionally add the date as part of the tag to better indicate
the age of the image. Gitlab WEBGUI doens't indicate the age
in a good way nor makes it easy to short the images.
2019-03-26 17:18:24 +02:00
Matthew Waters
497415544b docker: add android-specific image
pre-includes the NDK/SDK
2019-03-26 04:00:57 +00:00
Jordan Petridis
0831f87561
gitlab/ci_template: refresh the image builds 2019-03-14 20:51:08 +02:00
Matthew Waters
775c0b3520 gitlab: remove the android gst-build target
android is now being tested using cerbero for the binaries we produce.
2019-03-13 13:00:01 -04:00
Nicolas Dufresne
4221153836 CI: Add Android universal builds 2019-03-13 10:35:00 -04:00
Nicolas Dufresne
ae80b863c3 gitlab: Move cerbero jobs in gst-ci
This will allow extending the CI at one place.
2019-03-12 22:56:21 -04:00
Nicolas Dufresne
b289400a04 Update docker hash
While cleaning up, I got confused since some docker were rebuilt without
but unused. So I broke the build, this should fix it.
2019-01-31 11:39:11 -05:00
Nicolas Dufresne
5a55d5103d CI: Add Win32 and Win64 cross builds 2019-01-08 10:31:19 -05:00
Xavier Claessens
aa13631e38 android: Use cross file shipped by gst-build
We want to test that cross files shipped by gst-build are working,
instead of a generating our own.
2018-12-13 17:08:16 +00:00
Xavier Claessens
791757de6c Rename 'android' docker image back to 'ubuntu'
There is no point in doing a different docker image for each build we
are going to do inside an Ubuntu distro. We can later use the same image
for native ubuntu builds, or other cross builds (e.g. android API 21,
etc).
2018-12-13 17:08:16 +00:00
Xavier Claessens
f7331ab4f8 Move Android NDK to /opt
It was installed in '/' which doesn't feel right. Installing it in /root
is also not correct because we want to run the build as user instead of
root in the future and cleanup.sh removes everything in /root. /opt
seems the best place because that's also the default location when
installing Android Studio.
2018-12-13 17:08:16 +00:00
Jordan Petridis
c8c9cad685 CI: Make the docker build and local jobs manual
While they are very useful, each time we create a branch,
gitlab tries to build all the images which is very resource
intesinve. Thus make all the local images and everything that
depends upon them a manual job and only trigger them before
merging an MR.
2018-12-11 21:34:08 +00:00
Jordan Petridis
7151af0270
ci_template.yml: Run the check test suite!!!
Add a fedora job that runs the 'check' tests with
gst-validate-launcher. Its fairly well abstracted so the same
template can be used to add the rest of the test-suites fairly
soon.
2018-12-10 14:45:56 +02:00
Nicolas Dufresne
b1e3b9e6ab gitlab: Add cerbero build template and x86_64 build 2018-12-09 19:12:07 +00:00
Nicolas Dufresne
f9435d7312 docker: Create Cerbero Fedora Docker 2018-12-09 19:12:07 +00:00
Thibault Saunier
bb2c819c77 Revert "Revert me" and fix test_manifest image to be used
This reverts commit 4717e446a9.
2018-12-07 02:52:34 +00:00
Jordan Petridis
96c5f89d79
CI: do not run local jobs for master branch
They are identical to the normal jobs
2018-12-06 15:43:43 +02:00
Thibault Saunier
4717e446a9 Revert me 2018-12-06 07:23:30 -03:00
Thibault Saunier
5aad1e149d Run test_manifest when the docker is changed 2018-12-06 07:23:30 -03:00
Jordan Petridis
06c76e1d63
CI: Update image uris 2018-12-01 15:20:38 +02:00
Jordan Petridis
6f7e01e1e3
Docker: fix a type in the test-manifest image 2018-12-01 13:17:46 +02:00
Jordan Petridis
8683c2940b
docker: add an image to run the tests of the manifest
So in the futuere we can avoid installing pytest each time
2018-11-30 20:43:39 +02:00
Jordan Petridis
e87e1fb6ca
docker: Add a new image for the gst-indent job
This is meant run GNU indent along with the gstreamer wrapper
through the codebase and error out for missconfigured code.

Part of https://gitlab.freedesktop.org/gstreamer/gstreamer-project/issues/33
2018-11-30 20:38:51 +02:00