There was no permission check when looking up repos so you were able to
get basic repo information even if you're not allowed to.
This uses `session.MustPull` (and set repo/perms before) to fix this.
Currently it would fail with panic:
```
runtime error: invalid memory address or nil pointer dereference
...
/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/forge/gitea/gitea.go:492 (0xdfb32e)
```
For "woodpecker-ci" the `name` is "Woodpecker CI"
and the `login` is "woodpecker-ci"
Fixes#2092
This was causing the organization lookup to fail, because it looks up
using the `login`, when it did not find the organization, it would try
to create it. The creation would fail, because it uses the `name`, and
an organization with that `name` already exists.
Resulting in:
```
pq: duplicate key value violates unique constraint "UQE_orgs_name"
```
error logs like:
```
{"level":"warn","error":"sql: no rows in result set","time":"2023-07-25T21:29:56Z"}
```
have to less context to be helpfull.
this will cange it as the message now looks like:
```
{"level":"warn","error":"GetPipelineLast: sql: no rows in result set", "time":"2023-07-27T02:54:25+02:00"}
```
closes#1743
fixes: setting secrets for own user namespace
- create org in database
- use orgID for org related APIs
Co-authored-by: 6543 <6543@obermui.de>
In order to test this functionality, we had to expose the `BranchHead()`
through an API endpoint
```
GET /repos/{repo_id}/branches/{branch}/head
```
The response is a string that contains the latest commit hash of the
requested branch.
for now it's not clear defined, what to do on an unsupported event.
e.g. gitea webhook panel shows 500 error and no message.
now we have a successful webhook and a message to show an info
This PR introduces two new server configuration options, for providing a
custom .JS and .CSS file.
These can be used to show custom banner messages, add
environment-dependent signals, or simply a corporate logo.
### Motivation (what problem I try to solve)
I'm operating Woodpecker in multiple k8s clusters for different
environments.
When having multiple browser tabs open, I prefer strong indicators for
each environment.
E.g. a red "PROD" banner, or just a blue "QA" banner.
Also, we sometimes need to have the chance for maintenance, and instead
of broadcasting emails,
I prefer a banner message, stating something like: "Heads-up: there's a
planned downtime, next Friday, blabla...".
Also, I like to have the firm's logo visible, which makes Woodpecker
look more like an integral part of our platform.
### Implementation notes
* Two new config options are introduced ```WOODPECKER_CUSTOM_CSS_FILE```
and ```WOODPECKER_CUSTOM_JS_FILE```
* I've piggy-bagged the existing handler for assets, as it seemed to me
a minimally invasive approach
* the option along with an example is documented
* a simple unit test for the Gin-handler ensures some regression safety
* no extra dependencies are introduced
### Visual example
The documented example will look like this.
### Areas of uncertainty
This is my first contribution to Woodpecker and I tried my best to align
with your conventions.
That said, I found myself uncertain about these things and would be glad
about getting feedback.
* The handler tests are somewhat different than the other ones because I
wanted to keep them simple - I hope that still matches your coding
guidelines
* caching the page sometimes will let the browser not recognize changes
and a user must reload. I'm not fully into the details of how caching is
implemented and neither can judge if it's a real problem. Another pair
of eyes would be good.
Using an empty token for an agent was returning the first agent from the
database as the orm is not adding where clauses for empty strings of a
model when querying.
# Huge thanks for reporting and explaining the issue ❤️
- Dominik Heidler
- Timo Tomasini
closes#1801closes#1815closes#1144
closes #983
closes #557closes#1827
regression of #1791
# TODO
- [x] adjust log model
- [x] add migration for logs
- [x] send log line via grpc using step-id
- [x] save log-line to db
- [x] stream log-lines to UI
- [x] use less structs for log-data
- [x] make web UI work
- [x] display logs loaded from db
- [x] display streaming logs
- [ ] ~~make migration work~~ -> dedicated pull (#1828)
# TESTED
- [x] new logs are stored in database
- [x] log retrieval via cli (of new logs) works
- [x] log streaming works (tested via curl & webui)
- [x] log retrieval via web (of new logs) works
---------
Co-authored-by: 6543 <6543@obermui.de>
This isolates single migration tasks from each other.
The migration itself is now not atomic anymore but each single migration
now on it's own.
This takes load away from databases, as new sessions have a committed
schema available.
We also disable xorm.cache, as the speed improvements are minor but
invalid cache caused by schema changes did happen already in the past.
---------
Reverts #1817Closes#1821
---------
Co-authored-by: 6543 <6543@obermui.de>
# Summary
This PR drops the outdated former swagger.yaml/json and introduced
automatic API document generation from Go code.
The generated code is also used to generate documentation/markdown for
the community page,
as well as enable the Woodpecker server to serve a Swagger Web UI for
manual tinkering.
I did opt-in for gin-swagger, a middleware for the Gin framework, to
ease implementation and have a sophisticated output.
This middleware only produces Swagger v2 specs. AFAIK the newer OpenApi
3x tooling is not yet that mature,
so I guess that's fine for now.
## Implemenation notes
- former swagger.json files removed
- former // swagger godocs removed
- introduced new dependency gin-swagger, which uses godoc annotations on
top of Gin Handler functions.
- reworked Makefile to automatically generate Go code for the server
- introduce new dependency go-swagger, to generate Markdown for
documentation purposes
- add a Swagger Web UI, incl. capabilities for manual API exploration
- consider relative root paths in the implementation
- write documentation for all exposed API endpoints
- incl. API docs in the community website (auto-generated)
- provide developer documentation, for the Woodpecker authors
- no other existing logic/code was intentionally changed
---------
close#292
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
it did make sense to have it still supported within v0.15.0,
but as we move future away and with the release of v1.0.0
we should not give the appearance of still support the original drone
v0.8 config
Gogs support is broken (and we won't fix it because we don't care about
it...) because it does not support OAuth, at least after we introduced
the new Vue UI.
See:
77d830d5b5/server/forge/gogs/gogs.go (L84)
This route is not present in the new UI.
closes#1636closes#1429
supersedes #1586
Uses a different approach: just take the index.html compiled by vite and
replace the paths to js and other files using regex. This is not
compatible with the dev proxy which is also the reason why we can't use
go templates for this.
Do not sync repos with forge if the repo is not necessary in DB.
In the DB, only repos that were active once or repos that are currently
active are stored. When trying to enable new repos, the repos list is
fetched from the forge instead and displayed directly. In addition to
this, the forge func `Perm` was removed and is now merged with `Repo`.
Solves a TODO on RepoBatch.
---------
Co-authored-by: Anbraten <anton@ju60.de>
Save which agent is running a task. This is now visible in the admin UI
in the queue and in the agent details screen.
# changes
- [x] save id of agent executing a task
- [x] add endpoint to get tasks of an agent for #999
- [x] show assigned agent-id in queue
- [x] (offtopic) use same colors for queue stats and icons (similar to
the ones used by pipelines)
- [x] (offtopic) use badges for queue labels & dependencies
# Changes
- Adds an admin view to see the whole work-queue of the server.
- The admin can also pause / resume the queue.
- The view is reloading data every 5 seconds automatically.
- The task model from queue got removed in favor of the one from models.
close#1114
As long as the `VersionResponse` type is not changed the check will
fail/pass gracefully
example output:
```
{"level":"error","error":"GRPC version mismatch","time":"2023-03-19T19:49:09+01:00","message":"Server version next-6923e7ab does report grpc version 2 but we only understand 1"}
GRPC version mismatch
```
- allow repo names to be case-insensitive
- improve backend error handling on DB get errors (record not found ->
404, else -> 500)
- replace magic numbers of http response codes
- unify the look and feel of cancel / save buttons on forms and view
them in one line
---------
Co-authored-by: Lauris BH <lauris@nix.lv>
Coding support is likely broken and nobody will ever fix it. Also it
looks like nobody wants to use it, otherwise we would have get some bug
reports.
---------
Co-authored-by: 6543 <6543@obermui.de>
Closes#1582
When `WOODPECKER_FLAT_PERMISSIONS=true` workaround is applied all
permissions are set to false (default) and query never returns any
matches.
This fixes it by always assigning Pull/Push/Admin to true when flatPermissions is enabled.
When a server such as Codeberg has unusually high response time, three
seconds may not be enough to fetch the configuration.
Signed-off-by: Earl Warren <contact@earl-warren.org>
Co-authored-by: 6543 <6543@obermui.de>
closes#101
Added secrets encryption in database
- Google TINK or simple AES as encryption mechanisms
- Keys rotation support on TINK
- Existing SecretService is wrapped by encryption layer
- Encryption can be enabled and disabled at any time
Co-authored-by: Kuzmin Ilya <ilia.kuzmin@indrive.com>
Co-authored-by: 6543 <6543@obermui.de>
This implements #1073, adds .yaml to the accepted endings for woodpecker configs.
This currently adds some more lines to the duplication (tried to compensate by fixing the other duplication in the configFetcher) as the CLI and Server are still separate.
Crude fix to allow to correctly list workspaces for bitbucket cloud
(https://bitbucket.org) and so run a pipeline.
Last year they removed a bunch of deprecated APIs and replaced them with
new ones.
Signed-off-by: Martin Herren <martin.herren@gmail.com>
Co-authored-by: Martin Herren <martin.herren@ecorobotix.com>
Co-authored-by: 6543 <6543@obermui.de>
Closes#1169
Replaces structs that were added inline in hook structs with structs of
the corresponding SDKs. This makes it more readable and error-proof.
Use IDs of the forge to fetch repositories instead of their names and owner names. This improves handling of renamed and transferred repos.
TODO
- [ ] try to support as many forges as possible
- [x] Gogs (no API)
- [ ] Bitbucket Server
- [x] Coding (no API?)
- [x] update repo every time it is fetched or received from the forge
- [x] if repo remote IDs are not available, use owner / name to get it
- [x] handle redirections (redirect a renamed repo to its new path)
- [x] ~~pull all repos once during migration to update ID (?)~~ issue fixed by on-demand loading of remote IDs
- [x] handle redirections in web UI
- [ ] improve handling of hooks after a repo was renamed (currently it checks for a redirection to the repo)
- [x] tests
- [x] `UNIQUE` constraint for remote IDs after migration shouldn't work (all repos have an empty string as remote ID)
close#854close#648 partial
close https://codeberg.org/Codeberg-CI/feedback/issues/46
Possible follow-up PRs
- apply the same scheme on everything fetched from the remote (currently only users)
Co-authored-by: 6543 <6543@obermui.de>
breakout from #934
when new events are added you don't have to worry that pipeline will behave different as it does now with this
Co-authored-by: Anbraten <anton@ju60.de>
at some point (~7years ago) the oauth2 implementation was copied into the code-base and never touched.
We only use it for gitlab the rest is already back using std.
This migrates to the std oauth2 implementation
* Implement database changes and store methods for global and organization secrets
* Add tests for new store methods
* Add organization secret API and UI
* Add global secrets API and UI
* Add suggestions
* Update warning style
* Apply suggestions from code review
Co-authored-by: Anbraten <anton@ju60.de>
* Fix lint warning
Co-authored-by: Anbraten <anton@ju60.de>
* make global environment variables available for pipeline substitution
* lint fixes
* global env support in cli exec; procBuilder tests
* drop GLOBAL_ prefix
* docs
* documentation typo
* Update docs/docs/20-usage/50-environment.md
as suggested by anbraten
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Anbraten <anton@ju60.de>
to make it easier for devs to find the right place for code
close#655
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
- refactor
- create new errors to handle on them
- dedup code
- split server pipeline functionality's into dedicated functions
- add code comments to document what goes on
- add TODOs for next refactor
Officially support labels for pipelines and agents to improve pipeline picking.
* add pipeline labels
* update, improve docs and add migration
* update proto file
---
closes#304 & #860
closes#11
Added support:
1. Environment variable `WOODPECKER_DELETE_MULTIPLE_RUNS_ON_EVENTS` (Default pull_request, push)
2. Builds will be marked as killed when they "override" another build
* Do not filter on linux/amd64 per default & add tests
Tasks with no platform would otherwise not perform on runners with different OS/ARCH combos
Signed-off-by: Lukas Bachschwell <lukas@lbsfilm.at>
Co-authored-by: 6543 <6543@obermui.de>
We previously got the machine hostname for Netrc from the url of the remote, but in cases where the clone-url does not match the api url this can lead to errors.
If the repo was renamed, there's an issue with Gitea: it redirects the /api/v1/repos/<owner>/<repo>/hooks POST request to a GET request at the same URL.
This URL returns the list of all hooks, thus the Gitea SDK can't parse the response into a single gitea.Hook type.
A better error is also visisble if the repo was deleted.
* only calculate time on running builds
* Add updated timestamp into database and use it in frontend
* add more trace logging
* refactor (move grpc unrelated func into related package)
* fix xorm schema
* add todo
Some flags where unused and / or unnecessary as they are covered by alternatives implemented in PRs of milestone 0.15.0 and just complicated the setup.
closes#681
* use flag value
* fix test
* sed -i 's/STATUS_CONTEXT/WOODPECKER_STATUS_CONTEXT/g'
* docs
* Update docs/docs/91-migrations.md
Co-authored-by: Anbraten <anton@ju60.de>
- Add field for image list in Secrets Repo Settings (Web UI)
Simple comma separated input field, split into images array
- validate secret images in backend
- trim spaces and filter empty list items
Signed-off-by: 6543 <6543@obermui.de>
Co-authored-by: 6543 <6543@obermui.de>
- link to specific proc (only general build before)
- set status for all procs (before: only for the whole build on some SCMs)
- set status after restart
- set status to pending after waiting for approval
- make status of gitlab, gitea & github equal
- dedupe status update code
- dedupe `PostBuild` code
close#410, close#297, close#459, close#521
Benefits:
- the webhook delivery history of the drone-ci-service is broken (no way to check if a webhook was successfully delivered by Gitlab)
- drone-ci-service has limited events support (for example no comment or branch deleted event)
- independent from drone integration in general
Fixed problems:
- `c.AbortWithError()` stops the chain, but does not return a message as http response
- only return woodpeckers own messages and hide errors from http response (could be dangerous)
- added missing returns
- log all messages send to the user (at least on debug level)
* write back to webhook caller what happend
* skip sound like an error - it is none change that
* improve hook func
* dedup code & fix bugs that only existed on gated builds
* startBuild use std context
* wordings
Co-authored-by: Anbraten <anton@ju60.de>
* nit
* todo done
Co-authored-by: Anbraten <anton@ju60.de>
* UI: let remove be a remove
* UI: add deactivate repo btn
* Store: DeleteRepo also delete related
* Store: more test coverage
Co-authored-by: 6543 <6543@obermui.de>
Dropped support for `DRONE_*` environment variables in pipeline steps. Pipeline meta-data can be accessed with `CI_*` variables.
- `CI_*` prefix replaces `DRONE_*`
- `CI` value is now `woodpecker`
- `DRONE=true` has been removed
Resolve some todos in server/model:
* Move persistent queue into its own package
* Create Types: StatusValue, SCMKind, RepoVisibly
* Rename struct Repo fields: SCMKind, IsSCMPrivate
Rewrite of the UI using Typescript, Vue3, Windicss and Vite. The design should be close to the current one with some changes:
- latest pipeline in a sidebar on the right
- secrets and registry as part of the repo-settings (secrets and registry entries shouldn't be used as much so they can be "hidden" under settings IMO)
- start page shows list of active repositories with button to enable / add new ones (currently you see all repositories and in most cases you only add new repositories once in a while)
test for a bug where pipelines that use `depends_on` were not built in multi-pipeline builds. The problem is that pipelines names keep a leading `'/'` when the pipeline path does not have a trailing `'/'`.
This is a preparation for #245 and adds a new endpoint (`.js``file) which can be included by the web-ui to get some config and credentials like the user profile or access-token.
The goal here is to make consistent use of configuration environment variables prefixed `WOODPECKER_`. Where several variants existed, this PR aims to remove all but one option, leaving the most explicit.
This PR only changes server and agent code, but not documentation, in order to keep the PR digestible. Once we have consensus that this is correct, I'll change docs accordingly.
User (rather: admin) facing changes in this PR:
- In general, support for all server and agent config environment variables (env vars) starting with `DRONE_` is removed. The according `WOODPECKER_*` variables must be used instead.
- The env var `WOODPECKER_HOST` replaces `DRONE_HOST`, and `DRONE_SERVER_HOST`.
- The env var `WOODPECKER_AGENT_SECRET` is used to configure the shared secret which agents use to authenticate against the server. It replaces `WOODPECKER_SECRET`, `DRONE_SECRET`, `WOODPECKER_PASSWORD`, `DRONE_PASSWORD`, and `DRONE_AGENT_SECRET`.
- The env var `WOODPECKER_DATABASE_DRIVER` replaces `DRONE_DATABASE_DRIVER` and `DATABASE_DRIVER`.
- The env var `WOODPECKER_DATABASE_DATASOURCE` replaces `DRONE_DATABASE_DATASOURCE` and `DATABASE_CONFIG`.
* pass context down to remote clients
* make tests work
* add ctx to Refresh() and use it
* bitbucketserver
* code format
* plugin interface: add todo context
* solve todo
* RM TODO by using context.WithTimeout
* refactor & fix
* Apply suggestions from code review
Co-authored-by: Anbraten <anton@ju60.de>
* go fmt
* Update server/remote/coding/coding.go
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: Anbraten <anton@ju60.de>
By adding a new ENV variable called `WOODPECKER_WWW_PROXY` it is possible to serve a webinterface via a proxy configured by the `WOODPECKER_WWW_PROXY` value for development instead of serving the interface from the bundled code or from some folder location as the old `WOODPECKER_WWW` option allowed. Using a proxy allows developing the UI with hot-reloading.
#299 who closed#133 did not take into account, that that gitea (and eventually) other forges do return 200 and empty string if file was not found - this make configFetcher more resilient
* Move cncd/{logging,pubsub,queue}/ to server/{logging,pubsub,queue}/
* Update REAMDEs and include history
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: Anbraten <anton@ju60.de>
* move api code to server/api
* move grpc server for agent communication to server/grpc
* move server.Config to server/config.go as it is used by both server/api and server/grpc
* move shared code used by server/api and server/grpc to server/shared
* delete obsolete cncd code
jsonrpc2 client+server, and the utils that used them (piped, pipec)
are not used anymore.
jsonrpc2 was replaced with grpc
piped+pipec were replaced by agents
* delete duplicate section in swagger file
* comment typos
* go mod tidy (websocket+jsonrpc2 no longer used)
* go mod vendor
Co-authored-by: Anbraten <anton@ju60.de>
With #299 the `sanitizePath` has a bug now. If no special config (empty string) has been set it falls back to default values, but the `sanitizePath` function the used the pipelince-config-path to sanitize pipeline names.
# Config resolution
- pipeline-config setting empty / not specified (default): `.woodpecker/` => `.woodpecker.yml` => `.drone.yml`
- pipeline-config setting defined by user: try that file / folder and no fallback (if a user sets some special value that is normally done for some reason)
# Changes
- pipeline-config setting will be empty by default
- remove fallback setting for config loading (simplifies config)
---
closes#133
---
* adjust config fetching mechanism
* default path empty
* remove fallback flag from ui and db
closes#271
- filter pipeline config folders for `.yml` and `.yaml` files
- improve `fetchConfig` tests
- update remote mock and correct wrong folder name `mock` => `mocks` to match package name
- fix: return correct filename for fallback
- improve config loading by checking if folder or not before sending api call
Example:
```yaml
when:
path: '*.md'
```
should match only builds in which the commit added/removed or modified files with the *.md extension
Co-authored-by: 6543 <6543@obermui.de>
Woodpecker should use configured server URL when creating self
URL-s for VCS UI and not depend on URL read from request. In
case of internal calls like answers for webhooks from Gitea
on backend, address in URL may be internal and this causes
Gitea to display CI status for commit with intenal URL
to woodpecker not external one.
