Commit graph

55 commits

Author SHA1 Message Date
Nirbheek Chauhan 0c7275212a manifest: Check all deps logs when picking cerbero commit
Instead of picking the 'slowest build', look at all the deps logs and
pick the commit that is mentioned in the maximum number of deps logs.

In practice, this will either be the newest commit or the previous
commit. If it's not the newest commit, we will warn and use an older
one.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/349>
2020-09-22 16:59:48 +05:30
Nicolas Dufresne dec76861ea manifest: Fix URL for cerbero deps log
There was a missing 'f' prefix, so the {branch} was not expanded.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/348>
2020-09-21 11:57:59 -04:00
Nicolas Dufresne 808fc283f3 manifest: Build useful error message
This helps detecting issue with optional steps like cerbero deps
lookup.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-ci/-/merge_requests/348>
2020-09-21 11:57:26 -04:00
Nicolas Dufresne e137093def ci: Update to use new external storage for cerbero deps
The goal of this change is to reduce egress on the FDO servers.
2020-04-03 16:31:45 -04:00
Jordan Petridis 2cde4bef1e gitlab/build_manifest: allow for upstream branch to be specified
When the pipeline is based on top of a stable branch, we want
to track that branch isntead of the primary development branch.

This patch makes it so the upstream branch can be specified with
an env var.

part of #11
2019-06-05 22:51:04 +00:00
Jordan Petridis e6cf53329e
gitlab/build_manifest: add gst-sharp to the module list 2019-05-20 19:29:46 +03:00
Nicolas Dufresne 1e27cc81b1 gitlab: manifest: Use a cerbero ref with build
For non-cerbero builds, pick a cerbero reference for which a build has
completed. This will reduce the number of cache miss, hence reduce the
number of timeouts and slow build we are facing each time cerbero is
updated.

Fixes #16
2019-05-03 20:05:33 +00:00
Matthew Waters c4a3df759d add CI for gst-examples android build 2019-03-27 12:11:31 +11:00
Thibault Saunier 74a954ca91 Track gst-integration-testsuites 2019-03-21 18:15:13 -03:00
Nicolas Dufresne d7322403f2 gitlab: Add orc and cerbero to manifest 2018-12-09 19:12:07 +00:00
Thibault Saunier 5856b05f30 build_manifest: fetch remote refname before checking out right commit
Working around https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/95
2018-12-06 07:23:30 -03:00
Thibault Saunier 7cb3feec19 build_manifest: Include remote refname in the manifest 2018-12-06 07:23:30 -03:00
Thibault Saunier 8932df0c2d buildmanifest: Move away from the gitlab API and use plain git to retrieve branch names
Makes the code much simpler
2018-12-06 00:09:17 -03:00
Nicolas Dufresne 597544cd61 build_manifest: Increase the number of search results
This increase the number of search results when looking up user
branches to 100 items. This should be sufficient for all of our users.

Fixes #3
2018-12-01 11:33:44 -05:00
Thibault Saunier 2414895a53 build_manifest: Add the "path" attribute to projects
When the local path and the name are different, both have to be
specified, it is now required since we add the '.git' to the name.
2018-11-21 20:37:12 -03:00
Thibault Saunier 10ffa6f1af build_manifest: Add the to repo names to avoid spamming about redirections 2018-11-14 10:55:14 -03:00
Thibault Saunier 10ae6d0326 build_manifest: Never use users 'master' branch for repos we are not explicitely building
People very often do not update them leading to random and weird failures
2018-11-14 10:55:14 -03:00
Thibault Saunier d8ebbea1f0 build_manifest: Make sure to respect GitLab currently built SHA 2018-11-14 10:14:08 -03:00
Thibault Saunier 0cfe2cee41 gitlab: Handle checking out the right gst-ci branch to call build_manifest 2018-11-13 16:51:36 +00:00
Jordan Petridis 48f3063daf
build_manifest.py: Use CI_PROJECT_NAMESPACE instead of GITLAB_USER
GITLAB_USER_* variables represent the user *that triggered* the job,
which might not match the user that owns the namespace of the fork.

This happens frequently with rebase, where the one who triggers the
rebase is the one that appears in the GITLAB_USER_* vars.

Instead use the CI_PROJECT_NAMESPACE variable to search the user's
projects. If CI_PROJECT_NAMESPACE has the value of a Group namespace
search_user_namespace returns 404.
2018-11-13 13:20:32 +02:00
Víctor Manuel Jáquez Leal d1b80b835b build_manifest.py: Add gstreamer-vaapi 2018-11-12 16:04:13 +00:00
Jordan Petridis b335b4416b
build_manifest.py: Silence irrelevant logs
The group API querries where removed in 2320cfaf but the
log was forgotten.

The headers print was meant to be just for debugging.
2018-11-12 16:51:22 +02:00
Edward Hervey 7a51b846f5 build-manifest: Remove reference to anongit
And make the gitlab one the "origin"
2018-11-09 14:08:17 +00:00
Jordan Petridis fa20357221
build_manifest.py: remove get_hostname function
Not really needed anymore now that it just calls urlparse.
2018-11-06 16:25:53 +02:00
Jordan Petridis 9d637f6694
build_manifest.py: Hardcode the project ids of upstream repos.
Since we can't use the group api to query their id, see previous
commit, we are going to hard code the upstream project id for the
time being.
2018-11-06 16:25:53 +02:00
Jordan Petridis 2320cfaffd
build_manifest: Drop the group search for now
Since we can't authenticate with the CI_JOB_TOKEN it needs a
workaroud. Currently it falls back to the upstream repo which is
good enough to get us going.
2018-11-05 15:18:38 +02:00
Jordan Petridis 7ff131879b
build_manifest.py: Fix the api querries
Previously the script assumed that the best match will be the first
result, but that's not always the case apparently.
2018-11-05 14:41:50 +02:00
Jordan Petridis 39e2ddf788
build_manifest.py: avoid duplicate / in requests 2018-11-05 14:41:50 +02:00
Jordan Petridis 76d38faf20
build_manifest.py: remove request assertion
This does indeed brake things
2018-11-05 14:41:50 +02:00
Jordan Petridis b5b71baddb
build_manifest.py: Prefer a custom token if set
In order to be able to query the Gitlab Group API we need to be
authenticated. CI_JOB_TOKEN for public jobs has a non-meaningfull
value which does not actually authenticate the Runner to the
intance.
2018-11-05 14:41:50 +02:00
Jordan Petridis 02b57af288
build_manifest.py: add some basic logging 2018-11-05 14:41:50 +02:00
Jordan Petridis 41a4ca4c8e
build_manifest.py: preserve env vars when running tests
pytest will happily run all the tests in the same proccess,
which means if one mutates an environment variable, the change
will bleed to the rest of the tests.
2018-11-05 14:31:52 +02:00
Jordan Petridis f4d6e7e9c6
build_manifest.py: Avoid duplicating some code 2018-11-05 14:31:52 +02:00
Jordan Petridis 07a1480fc9
build_manifest.py: Avoid making the CI variables global
There are half of the variables compared to what was used before,
and now they are only used once so its not needed for them anymore
to be global.

This allows running the tests without needing to pre-define them
in your local environment, since the ones needed are mocked in the
tests code anyway.
2018-11-05 14:31:52 +02:00
Jordan Petridis 2e2864cde7
build_manifest.py: Properly query the group namepsace
Add a function to query groups, so the fallback code will
be functional once gstreamer migrates to gitlab.
2018-11-05 14:31:52 +02:00
Jordan Petridis 490e936dd1
build_manifest.py: Remove an unnecessary if branch
I can't think of a scenario where this would be hit, and can't
trigger make the find_reposiroty_sha function return None either.
2018-11-05 14:31:52 +02:00
Jordan Petridis 64c0e6fba3
build_manifest.py: Remove an early exit branch
find_repository_sha will conclude to the same result so
we can remove the whole if branch and extra variables
that it needed.
2018-11-05 14:31:52 +02:00
Jordan Petridis e042a76031
build_manifest.py: Test the find_repository_sha function 2018-11-05 14:31:52 +02:00
Jordan Petridis 412d980b62
build_manifest.py: Handle bad requests 2018-11-05 14:31:52 +02:00
Jordan Petridis 10b1812e70
build_manifest.py: Use more descriprive variable name 2018-11-05 14:31:52 +02:00
Jordan Petridis 45c89ece12
build_manifest.py: Make the needed env vars global variables 2018-11-05 14:31:52 +02:00
Jordan Petridis 17195ea3aa
build_manifest.py: add a failure test case for branches 2018-11-05 14:31:52 +02:00
Jordan Petridis ea5459e127
build_manifest.py: search user namespaces instead of global
Previously find_repository_sha would do a global search of
all the projects in the gitlab instance. This ports it to
use the user namespace endpoint.

Additionally seems like the else: block never worked.
2018-11-05 14:31:52 +02:00
Jordan Petridis 0b9a8b5cac
build_manifest.py: Add a function to search user namespace of projects
Also fix a type annotation error in the request functions.
2018-11-05 14:31:52 +02:00
Jordan Petridis ca9b1362b3
build_manifest.py: Make sure the remote always ends with a slash
that's how `git-update` in `gst-build` expects it to be
2018-11-05 14:31:52 +02:00
Jordan Petridis dc275a204e
build_manifest.py: Query the exact branch needed 2018-11-05 14:31:51 +02:00
Jordan Petridis 7ad193f276
build_manifest.py: Add a FIXME comment 2018-11-05 11:24:12 +02:00
Jordan Petridis 7ede52d7c7
build_manifest.py: Use the project path not the name
The project path is what's shown in the url bar, ex
gitlab.example.com/john/gstreamer -> path == gstreamer

The project name is defined in project settings and its just.
Its more common to change the name of a fork than its path.
2018-11-05 11:24:12 +02:00
Jordan Petridis 61c5d7b6cf
build_manifest.py: split the request function in two
Make it so it does not depend on env variables so it will be easier
to test.
2018-11-05 11:24:12 +02:00
Jordan Petridis 6772fa76aa
build_manifest.py: Split the hostname resolution from the requests 2018-11-05 11:24:12 +02:00