closes#1801closes#1815closes#1144
closes #983
closes #557closes#1827
regression of #1791
# TODO
- [x] adjust log model
- [x] add migration for logs
- [x] send log line via grpc using step-id
- [x] save log-line to db
- [x] stream log-lines to UI
- [x] use less structs for log-data
- [x] make web UI work
- [x] display logs loaded from db
- [x] display streaming logs
- [ ] ~~make migration work~~ -> dedicated pull (#1828)
# TESTED
- [x] new logs are stored in database
- [x] log retrieval via cli (of new logs) works
- [x] log streaming works (tested via curl & webui)
- [x] log retrieval via web (of new logs) works
---------
Co-authored-by: 6543 <6543@obermui.de>
This isolates single migration tasks from each other.
The migration itself is now not atomic anymore but each single migration
now on it's own.
This takes load away from databases, as new sessions have a committed
schema available.
We also disable xorm.cache, as the speed improvements are minor but
invalid cache caused by schema changes did happen already in the past.
---------
Reverts #1817Closes#1821
---------
Co-authored-by: 6543 <6543@obermui.de>
# Summary
This PR drops the outdated former swagger.yaml/json and introduced
automatic API document generation from Go code.
The generated code is also used to generate documentation/markdown for
the community page,
as well as enable the Woodpecker server to serve a Swagger Web UI for
manual tinkering.
I did opt-in for gin-swagger, a middleware for the Gin framework, to
ease implementation and have a sophisticated output.
This middleware only produces Swagger v2 specs. AFAIK the newer OpenApi
3x tooling is not yet that mature,
so I guess that's fine for now.
## Implemenation notes
- former swagger.json files removed
- former // swagger godocs removed
- introduced new dependency gin-swagger, which uses godoc annotations on
top of Gin Handler functions.
- reworked Makefile to automatically generate Go code for the server
- introduce new dependency go-swagger, to generate Markdown for
documentation purposes
- add a Swagger Web UI, incl. capabilities for manual API exploration
- consider relative root paths in the implementation
- write documentation for all exposed API endpoints
- incl. API docs in the community website (auto-generated)
- provide developer documentation, for the Woodpecker authors
- no other existing logic/code was intentionally changed
---------
close#292
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
it did make sense to have it still supported within v0.15.0,
but as we move future away and with the release of v1.0.0
we should not give the appearance of still support the original drone
v0.8 config
Gogs support is broken (and we won't fix it because we don't care about
it...) because it does not support OAuth, at least after we introduced
the new Vue UI.
See:
77d830d5b5/server/forge/gogs/gogs.go (L84)
This route is not present in the new UI.
closes#1636closes#1429
supersedes #1586
Uses a different approach: just take the index.html compiled by vite and
replace the paths to js and other files using regex. This is not
compatible with the dev proxy which is also the reason why we can't use
go templates for this.
Do not sync repos with forge if the repo is not necessary in DB.
In the DB, only repos that were active once or repos that are currently
active are stored. When trying to enable new repos, the repos list is
fetched from the forge instead and displayed directly. In addition to
this, the forge func `Perm` was removed and is now merged with `Repo`.
Solves a TODO on RepoBatch.
---------
Co-authored-by: Anbraten <anton@ju60.de>
Save which agent is running a task. This is now visible in the admin UI
in the queue and in the agent details screen.
# changes
- [x] save id of agent executing a task
- [x] add endpoint to get tasks of an agent for #999
- [x] show assigned agent-id in queue
- [x] (offtopic) use same colors for queue stats and icons (similar to
the ones used by pipelines)
- [x] (offtopic) use badges for queue labels & dependencies
# Changes
- Adds an admin view to see the whole work-queue of the server.
- The admin can also pause / resume the queue.
- The view is reloading data every 5 seconds automatically.
- The task model from queue got removed in favor of the one from models.
close#1114
As long as the `VersionResponse` type is not changed the check will
fail/pass gracefully
example output:
```
{"level":"error","error":"GRPC version mismatch","time":"2023-03-19T19:49:09+01:00","message":"Server version next-6923e7ab does report grpc version 2 but we only understand 1"}
GRPC version mismatch
```
- allow repo names to be case-insensitive
- improve backend error handling on DB get errors (record not found ->
404, else -> 500)
- replace magic numbers of http response codes
- unify the look and feel of cancel / save buttons on forms and view
them in one line
---------
Co-authored-by: Lauris BH <lauris@nix.lv>
Coding support is likely broken and nobody will ever fix it. Also it
looks like nobody wants to use it, otherwise we would have get some bug
reports.
---------
Co-authored-by: 6543 <6543@obermui.de>
Closes#1582
When `WOODPECKER_FLAT_PERMISSIONS=true` workaround is applied all
permissions are set to false (default) and query never returns any
matches.
This fixes it by always assigning Pull/Push/Admin to true when flatPermissions is enabled.
When a server such as Codeberg has unusually high response time, three
seconds may not be enough to fetch the configuration.
Signed-off-by: Earl Warren <contact@earl-warren.org>
Co-authored-by: 6543 <6543@obermui.de>
closes#101
Added secrets encryption in database
- Google TINK or simple AES as encryption mechanisms
- Keys rotation support on TINK
- Existing SecretService is wrapped by encryption layer
- Encryption can be enabled and disabled at any time
Co-authored-by: Kuzmin Ilya <ilia.kuzmin@indrive.com>
Co-authored-by: 6543 <6543@obermui.de>
This implements #1073, adds .yaml to the accepted endings for woodpecker configs.
This currently adds some more lines to the duplication (tried to compensate by fixing the other duplication in the configFetcher) as the CLI and Server are still separate.
Crude fix to allow to correctly list workspaces for bitbucket cloud
(https://bitbucket.org) and so run a pipeline.
Last year they removed a bunch of deprecated APIs and replaced them with
new ones.
Signed-off-by: Martin Herren <martin.herren@gmail.com>
Co-authored-by: Martin Herren <martin.herren@ecorobotix.com>
Co-authored-by: 6543 <6543@obermui.de>
Closes#1169
Replaces structs that were added inline in hook structs with structs of
the corresponding SDKs. This makes it more readable and error-proof.
Use IDs of the forge to fetch repositories instead of their names and owner names. This improves handling of renamed and transferred repos.
TODO
- [ ] try to support as many forges as possible
- [x] Gogs (no API)
- [ ] Bitbucket Server
- [x] Coding (no API?)
- [x] update repo every time it is fetched or received from the forge
- [x] if repo remote IDs are not available, use owner / name to get it
- [x] handle redirections (redirect a renamed repo to its new path)
- [x] ~~pull all repos once during migration to update ID (?)~~ issue fixed by on-demand loading of remote IDs
- [x] handle redirections in web UI
- [ ] improve handling of hooks after a repo was renamed (currently it checks for a redirection to the repo)
- [x] tests
- [x] `UNIQUE` constraint for remote IDs after migration shouldn't work (all repos have an empty string as remote ID)
close#854close#648 partial
close https://codeberg.org/Codeberg-CI/feedback/issues/46
Possible follow-up PRs
- apply the same scheme on everything fetched from the remote (currently only users)
Co-authored-by: 6543 <6543@obermui.de>
breakout from #934
when new events are added you don't have to worry that pipeline will behave different as it does now with this
Co-authored-by: Anbraten <anton@ju60.de>
at some point (~7years ago) the oauth2 implementation was copied into the code-base and never touched.
We only use it for gitlab the rest is already back using std.
This migrates to the std oauth2 implementation
* Implement database changes and store methods for global and organization secrets
* Add tests for new store methods
* Add organization secret API and UI
* Add global secrets API and UI
* Add suggestions
* Update warning style
* Apply suggestions from code review
Co-authored-by: Anbraten <anton@ju60.de>
* Fix lint warning
Co-authored-by: Anbraten <anton@ju60.de>
* make global environment variables available for pipeline substitution
* lint fixes
* global env support in cli exec; procBuilder tests
* drop GLOBAL_ prefix
* docs
* documentation typo
* Update docs/docs/20-usage/50-environment.md
as suggested by anbraten
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Anbraten <anton@ju60.de>
to make it easier for devs to find the right place for code
close#655
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
- refactor
- create new errors to handle on them
- dedup code
- split server pipeline functionality's into dedicated functions
- add code comments to document what goes on
- add TODOs for next refactor
Officially support labels for pipelines and agents to improve pipeline picking.
* add pipeline labels
* update, improve docs and add migration
* update proto file
---
closes#304 & #860
closes#11
Added support:
1. Environment variable `WOODPECKER_DELETE_MULTIPLE_RUNS_ON_EVENTS` (Default pull_request, push)
2. Builds will be marked as killed when they "override" another build
* Do not filter on linux/amd64 per default & add tests
Tasks with no platform would otherwise not perform on runners with different OS/ARCH combos
Signed-off-by: Lukas Bachschwell <lukas@lbsfilm.at>
Co-authored-by: 6543 <6543@obermui.de>
We previously got the machine hostname for Netrc from the url of the remote, but in cases where the clone-url does not match the api url this can lead to errors.
If the repo was renamed, there's an issue with Gitea: it redirects the /api/v1/repos/<owner>/<repo>/hooks POST request to a GET request at the same URL.
This URL returns the list of all hooks, thus the Gitea SDK can't parse the response into a single gitea.Hook type.
A better error is also visisble if the repo was deleted.
* only calculate time on running builds
* Add updated timestamp into database and use it in frontend
* add more trace logging
* refactor (move grpc unrelated func into related package)
* fix xorm schema
* add todo
Some flags where unused and / or unnecessary as they are covered by alternatives implemented in PRs of milestone 0.15.0 and just complicated the setup.
closes#681
* use flag value
* fix test
* sed -i 's/STATUS_CONTEXT/WOODPECKER_STATUS_CONTEXT/g'
* docs
* Update docs/docs/91-migrations.md
Co-authored-by: Anbraten <anton@ju60.de>
- Add field for image list in Secrets Repo Settings (Web UI)
Simple comma separated input field, split into images array
- validate secret images in backend
- trim spaces and filter empty list items
Signed-off-by: 6543 <6543@obermui.de>
Co-authored-by: 6543 <6543@obermui.de>
- link to specific proc (only general build before)
- set status for all procs (before: only for the whole build on some SCMs)
- set status after restart
- set status to pending after waiting for approval
- make status of gitlab, gitea & github equal
- dedupe status update code
- dedupe `PostBuild` code
close#410, close#297, close#459, close#521
Benefits:
- the webhook delivery history of the drone-ci-service is broken (no way to check if a webhook was successfully delivered by Gitlab)
- drone-ci-service has limited events support (for example no comment or branch deleted event)
- independent from drone integration in general
Fixed problems:
- `c.AbortWithError()` stops the chain, but does not return a message as http response
- only return woodpeckers own messages and hide errors from http response (could be dangerous)
- added missing returns
- log all messages send to the user (at least on debug level)
* write back to webhook caller what happend
* skip sound like an error - it is none change that
* improve hook func
* dedup code & fix bugs that only existed on gated builds
* startBuild use std context
* wordings
Co-authored-by: Anbraten <anton@ju60.de>
* nit
* todo done
Co-authored-by: Anbraten <anton@ju60.de>
* UI: let remove be a remove
* UI: add deactivate repo btn
* Store: DeleteRepo also delete related
* Store: more test coverage
Co-authored-by: 6543 <6543@obermui.de>
