closes#1801closes#1815closes#1144
closes #983
closes #557closes#1827
regression of #1791
# TODO
- [x] adjust log model
- [x] add migration for logs
- [x] send log line via grpc using step-id
- [x] save log-line to db
- [x] stream log-lines to UI
- [x] use less structs for log-data
- [x] make web UI work
- [x] display logs loaded from db
- [x] display streaming logs
- [ ] ~~make migration work~~ -> dedicated pull (#1828)
# TESTED
- [x] new logs are stored in database
- [x] log retrieval via cli (of new logs) works
- [x] log streaming works (tested via curl & webui)
- [x] log retrieval via web (of new logs) works
---------
Co-authored-by: 6543 <6543@obermui.de>
# Summary
This PR drops the outdated former swagger.yaml/json and introduced
automatic API document generation from Go code.
The generated code is also used to generate documentation/markdown for
the community page,
as well as enable the Woodpecker server to serve a Swagger Web UI for
manual tinkering.
I did opt-in for gin-swagger, a middleware for the Gin framework, to
ease implementation and have a sophisticated output.
This middleware only produces Swagger v2 specs. AFAIK the newer OpenApi
3x tooling is not yet that mature,
so I guess that's fine for now.
## Implemenation notes
- former swagger.json files removed
- former // swagger godocs removed
- introduced new dependency gin-swagger, which uses godoc annotations on
top of Gin Handler functions.
- reworked Makefile to automatically generate Go code for the server
- introduce new dependency go-swagger, to generate Markdown for
documentation purposes
- add a Swagger Web UI, incl. capabilities for manual API exploration
- consider relative root paths in the implementation
- write documentation for all exposed API endpoints
- incl. API docs in the community website (auto-generated)
- provide developer documentation, for the Woodpecker authors
- no other existing logic/code was intentionally changed
---------
close#292
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
Gogs support is broken (and we won't fix it because we don't care about
it...) because it does not support OAuth, at least after we introduced
the new Vue UI.
See:
77d830d5b5/server/forge/gogs/gogs.go (L84)
This route is not present in the new UI.
closes#1636closes#1429
supersedes #1586
Uses a different approach: just take the index.html compiled by vite and
replace the paths to js and other files using regex. This is not
compatible with the dev proxy which is also the reason why we can't use
go templates for this.
Do not sync repos with forge if the repo is not necessary in DB.
In the DB, only repos that were active once or repos that are currently
active are stored. When trying to enable new repos, the repos list is
fetched from the forge instead and displayed directly. In addition to
this, the forge func `Perm` was removed and is now merged with `Repo`.
Solves a TODO on RepoBatch.
---------
Co-authored-by: Anbraten <anton@ju60.de>
close#1114
As long as the `VersionResponse` type is not changed the check will
fail/pass gracefully
example output:
```
{"level":"error","error":"GRPC version mismatch","time":"2023-03-19T19:49:09+01:00","message":"Server version next-6923e7ab does report grpc version 2 but we only understand 1"}
GRPC version mismatch
```
Coding support is likely broken and nobody will ever fix it. Also it
looks like nobody wants to use it, otherwise we would have get some bug
reports.
---------
Co-authored-by: 6543 <6543@obermui.de>
When a server such as Codeberg has unusually high response time, three
seconds may not be enough to fetch the configuration.
Signed-off-by: Earl Warren <contact@earl-warren.org>
Co-authored-by: 6543 <6543@obermui.de>
closes#101
Added secrets encryption in database
- Google TINK or simple AES as encryption mechanisms
- Keys rotation support on TINK
- Existing SecretService is wrapped by encryption layer
- Encryption can be enabled and disabled at any time
Co-authored-by: Kuzmin Ilya <ilia.kuzmin@indrive.com>
Co-authored-by: 6543 <6543@obermui.de>
As discussed in the comments in PR #1197. Also add documenation
accordingly.
One thing I'm not sure about is the simple check in health.go if the
address is usable in the GET request or not. From reading
https://pkg.go.dev/net#Dial it seems that the only non-standard address
format that would work in the `net` package but not in a GET url would
likely only be `:port`, as the others listed here are actually also
valid urls:
`For TCP, UDP and IP networks, if the host is empty or a literal
unspecified IP address, as in ":80", "0.0.0.0:80" or "[::]:80" for TCP
and UDP, "", "0.0.0.0" or "::" for IP, the local system is assumed.`
One additional thing I noticed is that while `WOODPECKER_SERVER_ADDR`
and `WOODPECKER_SERVER_ADDR` use the default value format of `:PORT`,
`WOODPECKER_SERVER` actually uses `localhost:9000`. I guess it makes a
bit of sense, considering the server might not be local to the agent,
but it looks a bit inconsistent this way. I don't think it would hurt to
make the `WOODPECKER_HEALTHCHECK_ADDR` in this format too, but then it's
different from the server flags again... :-)
Officially support labels for pipelines and agents to improve pipeline picking.
* add pipeline labels
* update, improve docs and add migration
* update proto file
---
closes#304 & #860
closes#11
Added support:
1. Environment variable `WOODPECKER_DELETE_MULTIPLE_RUNS_ON_EVENTS` (Default pull_request, push)
2. Builds will be marked as killed when they "override" another build
With systems like docker swarm or docker compose it is usually a little awkward to manage secrets.
There is no way to directly inject them into the environment config. So you often have to write your secrets directly into the compose file
There are hacky workarounds such as overriding the entry-point of the container and loading a script which then fetches secrets from /run/secrets and replaces the environment variables, but this becomes very difficult once we are using docker images built from "scratch" (which is a really great practice otherwise) as there is no shell or standard tooling available
This adds a *_FILE variant of their Environment config values to work around this issue.
Signed-off-by: Lukas Bachschwell <lukas@lbsfilm.at>
We previously got the machine hostname for Netrc from the url of the remote, but in cases where the clone-url does not match the api url this can lead to errors.
* Added documentation of all configuration options.
* sort some flags
* adjust config docs to current flags
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Anton Bracke <anton@ju60.de>
Some flags where unused and / or unnecessary as they are covered by alternatives implemented in PRs of milestone 0.15.0 and just complicated the setup.
closes#681
* use flag value
* fix test
* sed -i 's/STATUS_CONTEXT/WOODPECKER_STATUS_CONTEXT/g'
* docs
* Update docs/docs/91-migrations.md
Co-authored-by: Anbraten <anton@ju60.de>
Resolve some todos in server/model:
* Move persistent queue into its own package
* Create Types: StatusValue, SCMKind, RepoVisibly
* Rename struct Repo fields: SCMKind, IsSCMPrivate
#494 introduced a bug, where a migration function can remove the sqlite3 file outside of the mounted docker volume.
that would result in a data lose after a container recreate.
this fix it by only rename the file if in same folder else just use the old path as fallback and put warnings into the log
Co-authored-by: Anbraten <anton@ju60.de>
This is a preparation for #245 and adds a new endpoint (`.js``file) which can be included by the web-ui to get some config and credentials like the user profile or access-token.
The goal here is to make consistent use of configuration environment variables prefixed `WOODPECKER_`. Where several variants existed, this PR aims to remove all but one option, leaving the most explicit.
This PR only changes server and agent code, but not documentation, in order to keep the PR digestible. Once we have consensus that this is correct, I'll change docs accordingly.
User (rather: admin) facing changes in this PR:
- In general, support for all server and agent config environment variables (env vars) starting with `DRONE_` is removed. The according `WOODPECKER_*` variables must be used instead.
- The env var `WOODPECKER_HOST` replaces `DRONE_HOST`, and `DRONE_SERVER_HOST`.
- The env var `WOODPECKER_AGENT_SECRET` is used to configure the shared secret which agents use to authenticate against the server. It replaces `WOODPECKER_SECRET`, `DRONE_SECRET`, `WOODPECKER_PASSWORD`, `DRONE_PASSWORD`, and `DRONE_AGENT_SECRET`.
- The env var `WOODPECKER_DATABASE_DRIVER` replaces `DRONE_DATABASE_DRIVER` and `DATABASE_DRIVER`.
- The env var `WOODPECKER_DATABASE_DATASOURCE` replaces `DRONE_DATABASE_DATASOURCE` and `DATABASE_CONFIG`.
- Add json schema file
- Add tests to validate sample pipeline files
- Add new command `lint` to cli to test a directory or single file to use correct schema
Example: `woodpecker-cli lint ./pipeline/schema/.woodpecker/`
---
close#275
preparation for #276
Co-authored-by: 6543 <6543@obermui.de>
By adding a new ENV variable called `WOODPECKER_WWW_PROXY` it is possible to serve a webinterface via a proxy configured by the `WOODPECKER_WWW_PROXY` value for development instead of serving the interface from the bundled code or from some folder location as the old `WOODPECKER_WWW` option allowed. Using a proxy allows developing the UI with hot-reloading.
* Move cncd/{logging,pubsub,queue}/ to server/{logging,pubsub,queue}/
* Update REAMDEs and include history
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: Anbraten <anton@ju60.de>
* Refactor: Move agent stats structs to agent/
Based in part on changes by laszlocph (kube branch):
7f4a0c8c12 Factor into Runner
f7fe9edd0b Using Runner in server
* Refactor: Add Runner constructor with counter field
Based in part on changes by laszlocph (kube branch):
7f4a0c8c12 Factor into Runner
f7fe9edd0b Using Runner in server
* Refactor: Move Runner to agent/ package
Based in part on changes by laszlocph (kube branch):
7f4a0c8c12 Factor into Runner
f7fe9edd0b Using Runner in server
* Refactor: pass pipeline.backend.Engine into agent.Runner
Based in part on changes by laszlocph (kube branch):
7f4a0c8 Factor into Runner
f7fe9ed Using Runner in server
* Use well-known function signature for WriteTo
* Rename stats.go -> state.go
Co-authored-by: Anbraten <anton@ju60.de>
* move api code to server/api
* move grpc server for agent communication to server/grpc
* move server.Config to server/config.go as it is used by both server/api and server/grpc
* move shared code used by server/api and server/grpc to server/shared
- move cli files from `cli/drone` to `cli/`
- move cli main to `cmd/cli/main.go` to match agent and server
- use version from `version/version.go` to match agent and server
# Config resolution
- pipeline-config setting empty / not specified (default): `.woodpecker/` => `.woodpecker.yml` => `.drone.yml`
- pipeline-config setting defined by user: try that file / folder and no fallback (if a user sets some special value that is normally done for some reason)
# Changes
- pipeline-config setting will be empty by default
- remove fallback setting for config loading (simplifies config)
---
closes#133
---
* adjust config fetching mechanism
* default path empty
* remove fallback flag from ui and db
Add flags to allow the agent to connect to a secure grpc endpoint.
This can be done by placing nginx in front of the drone-server or updating the
code to accept tls servers for the grpc server.
Added a feature to obtain the initial Vault token from the Kubernetes
auth method.
This works by making a request to the Vault server at the specified auth
method mount point's login path and presenting the JWT located in a file
on a running pod, along with the Kubernetes role to authenticate as.
Vault will then respond with a token and its TTL, if the request is valid.
This commit sets the http handler to the redirect function for let's encrypt
enabled drone instances. In addition, the `Strict-Transport-Security` header is
added to the redirect given `header.Secure` will only be added for gin routes.
This commit resolves#2261.
In our current drone setup, we are not using a proxy, thus letting drone handle
SSL termination. In addition, we are not exposing port 80 (effectively disabling
insecure drone access). When new engineers join and attempt to access drone,
they are not sent a 301 and often complain that they either do not have access
or that drone is broken (when in reality they are just accessing drone via the
incorrect protocol/port).
This commit changes the default behavior when running drone with a server-cert
by only sending redirects on port 80 rather than allowing both secure and
insecure access.
