In case you are developing a set of changes in a module, in
conjunction with a branch in gst-ci, you will end up with
a template with a different tag than the upstream repo, which
will be refferencing your gst-ci forked registry.
But that image won't existin in the namespace the
module would be running at, so we need to copy it from there.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
The idea behind local jobs was that you would have jobs that
share the same setup as the citemplate ones but would always
be pointing to your forked registry with your custom build of
the image, since the citemplate hardcoded the image it was
running against.
With the changes introduced in the previous commit
we now mirror the setup from freedesktop/ci-templates and the idea
is that jobs always run from the registry in your fork. If the
image sha/id matches the one from the upstream registry, its copied
over else a new one is build, pushed and tested.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
This is the inital step towards migrating our docker images setup
to something closer and eventually freedesktop/citemplates [1]
The idea is that jobs always run from the registry in your fork. If the
image sha/id matches the one from the upstream registry, its copied
over else a new one is build, pushed and tested.
Only change the fedora job for now while testing.
[1]: https://gitlab.freedesktop.org/freedesktop/ci-templates
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/308>
In all our other builds, we are using the clone_manifest_ref script
to fetch the revision of gst-build that we discover in the manifest.
For the windows job this was missed it seems, but didn't cause
any issues till now cause it only affected the gst-build MRs.
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/296>
Previously we were optimizing for cpu time, so we where building
gst-build once and then exporting that to be used by the test jobs.
However this meant that we where uploading 200mb (previously 600mb)
zipped of artifacts and then re-downloading them for each test job.
This caused big costs in terms of cloud egress since the runners
aren't hosted on the same cloud as the storage/artifacts instance.
Instead we are going to be rebuilding gst-build for each test
job from now, it also doesn't take more time than the network
i/o would of downloading the artifacts, so the impact of rebuilding
shouldn't be noticebly.
We are also using pinned git refs the modules we rebuild from
the manifest, so the binaries should be reproducible for the most
part (minus things like .pyc files).
Close#68
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/280>
docker builds can be big, and other shared runner have smaller
storage space which we should avoid filling if can be avoided.
The gst tagged runners are a better fit for such builds since
their disk storage is about 1-2 terabytes.
This avoids the need of using privilledged namespaces and dind
as buildah are able to build images unprivilledged.
One thing to note is that buildah inside docker is not a
supported configuration and not tested in upstream podman,
but the possible fallout is still easier to deal with than
dind and requiring privileged runners.
Previously we where accidently exporting the whole repo of
gst-integration-testsuites which includes 350mb of raw media
files and made the artifacts storage explode through the roof
along with the CI bills fd.o had to pay for uploading and
redownloading the artifacts
To deal with this, we clean all the media files from the builddir
and when needed we copy them over from the cache in the docker image,
and then git fetch the repo.
Close#69
We don't run the libnice testsuite, and when binaries are built
they consume ~45mb of space. This increases the size of the artifacts
we export from the gst-build job for the testsuite and drives up
the storage and bandwith costs when re-downloading the artifacts.
Similary disable the test targets of couple other subprojects as well
We have notice that a lot of CI activity is cause by user pushing to their
branch after having created an MR. To reduce our CI foot-print, the CI will
now only be automatically triggered when a reviewer assign the MR to the merge
bot. It will still be possible to run the CI manually but the result of that
CI won't be used by Marge.
