When in local mode, `getWorkflowIDFromStep` can handle normal steps with
a name like `wp_01h2a6qggwz68zekrkbwqq9rny_0_step_0`.
However, it will fail on clone (unless `skip_clone: true`) with an
`invalid step name` error.
```
invalid step name wp_01h2a2ebppp43bwjdfdsyj1m6m_0_clone
```
This patch handles either `_stage_` or `_clone` as the separator that
the local backend can use to extract the workflowID.
closes#1801closes#1815closes#1144
closes #983
closes #557closes#1827
regression of #1791
# TODO
- [x] adjust log model
- [x] add migration for logs
- [x] send log line via grpc using step-id
- [x] save log-line to db
- [x] stream log-lines to UI
- [x] use less structs for log-data
- [x] make web UI work
- [x] display logs loaded from db
- [x] display streaming logs
- [ ] ~~make migration work~~ -> dedicated pull (#1828)
# TESTED
- [x] new logs are stored in database
- [x] log retrieval via cli (of new logs) works
- [x] log streaming works (tested via curl & webui)
- [x] log retrieval via web (of new logs) works
---------
Co-authored-by: 6543 <6543@obermui.de>
This add a simple implementation of requests/limits for individual
steps. There is no validation of what the resource actually is beyond
checking that it can successfully be converted to a Quantity, so it can
be used for things other than just memory/CPU.
close#1809
Do not sync repos with forge if the repo is not necessary in DB.
In the DB, only repos that were active once or repos that are currently
active are stored. When trying to enable new repos, the repos list is
fetched from the forge instead and displayed directly. In addition to
this, the forge func `Perm` was removed and is now merged with `Repo`.
Solves a TODO on RepoBatch.
---------
Co-authored-by: Anbraten <anton@ju60.de>
- Kubernetes v1.26 on VKE causes error when creating persistent volume
claim because of uppercase characters in name field
This patch is trivial just in order to get it working - happy to
implement differently.
The error in question:
```
The PersistentVolumeClaim "wp-01G1131R63FWBSPMA4ZAZTKLE-0-clone-0" is invalid: metadata.name: Invalid value: "wp-01G1131R63FWBSPMA4ZAZTKLE-0-clone-0": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
```
close#1114
As long as the `VersionResponse` type is not changed the check will
fail/pass gracefully
example output:
```
{"level":"error","error":"GRPC version mismatch","time":"2023-03-19T19:49:09+01:00","message":"Server version next-6923e7ab does report grpc version 2 but we only understand 1"}
GRPC version mismatch
```
Closes https://github.com/woodpecker-ci/woodpecker/issues/1617
The `woodpecker exec` auto-detects the backend by iterating over a map
of backends. However, since Go 1 the runtime randomizes map iteration
order, so a random backend might be chosen during each execution.
PR changes to auto-detection to iterate over the backends list with
predefined priority: `docker`, `local`, `ssh`, `kubernetes`.
---------
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Closes https://github.com/woodpecker-ci/woodpecker/issues/1615
The error described in
https://github.com/woodpecker-ci/woodpecker/issues/1615 is happening
because `Tail` method of the docker backend closes the instance of
`io.ReadCloser` it returns in `defer` function. As a result anything
that try to read data returned by `Tail` method eventually will attempt
to read from closes reader and get an error:
2171212c5a/pipeline/backend/docker/docker.go (L229)
The fix is just don't close returned reader and let the consumer of
`Tail` method do it. Good thing is that `Tail` is used only in one place
and reader is correctly closed:
2171212c5a/pipeline/pipeline.go (L231-L237)
Example of `woodpecker exec` output using pipeline from
https://github.com/woodpecker-ci/woodpecker/issues/1615 with the fix:
```
woodpecker exec .woodpecker.yaml
[step1:L0:0s] + echo step1
[step1:L1:0s] step1
[step2:L0:0s] + echo step2
[step2:L1:0s] step2
```
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Try to fix#1495
It's very hard to reproduce it and only way to fix when it gets in this
state is woodpecker agent restart.
This anyway fixes problem if step mounts and
`WOODPECKER_BACKEND_DOCKER_VOLUMES` conflict
Since "success" and "failure" are the only two possible values, and
"success" is considered to be included by default, the existing code can
also be simplified a little.
This has the side effect of ignoring the "exclude" part of the
constraint completely. I put it in the tests just to make sure the
workaround in
https://github.com/woodpecker-ci/woodpecker/issues/1181#issuecomment-1347253585
continues to work as expected, but couldn't think of any legitimate use
cases for it.
Fixes#1181
Provide up to date drone compatibility environment variables to each step execution.
closes#1416
Before a step is executed, some environemnt variables are updated.
This ensures, that the updated environment variables are copied to their corresponding `DRONE_` environemt variables.
Side effect is that the `DRONE_` environemnt variables are no longer available in the metadata which should not harm as they are not used inside woodpecker.
closes#1181closes#834
Adds `ignore_failure` to pipeline steps. When it's set to true,
if the step fails the following steps continue to execute as if no failure had occurred.
---
failure enums idea:
* fail (default) = if other steps run in parallel, wait for them and
then let workflow fail
* cancel = if other steps run in parallel, kill them
* ignore = we mark the step as failed but it wont have any impact
at the moment we compile a script that we can pipe in as single command
this is because of the constrains the docker backend gives us.
so we move it into the docker backend and eventually get rid of it altogether
breakout from #934
when new events are added you don't have to worry that pipeline will behave different as it does now with this
Co-authored-by: Anbraten <anton@ju60.de>
use yaml aliases (https://yaml.org/spec/1.2.2/#3222-anchors-and-aliases) to have pipeline `variables`
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
Co-authored-by: Anbraten <anton@ju60.de>
/bin/env was used to resolve a command name against PATH and pass
additional environment variables.
All of this can also be achieved using functionality already provided by
go's exec lib, which will then internally pass the appropriate arguments
to e.g. execve.
A non-zero exit code signifies a pipeline failure, but is not a fatal error in the agent.
Since exec reports this as exec.ExitError, this has to be handled explicitly.
This also fixes logs not being shown on build errors.
- Support for pipeline/containers as list
- Support for container name in logs (step.Name)
Co-authored-by: Zav Shotan <zshotan@bloomberg.net>
Co-authored-by: 6543 <6543@obermui.de>
Officially support labels for pipelines and agents to improve pipeline picking.
* add pipeline labels
* update, improve docs and add migration
* update proto file
---
closes#304 & #860
When executing a backend step, in case of failure of the specific step, the run is marked as errored but the step error is missing.
Added:
1. Log for the backend error (without trace)
2. Mark the step as errored with exit code 126 (Could not execute).
Co-authored-by: Zav Shotan <zshotan@bloomberg.net>
Co-authored-by: Anton Bracke <anton@ju60.de>
Since /tmp is writable by everybody, a user could precreate
/tmp/woodpecker with 777 permissions, allowing them to modify the
pipeline while it is being run, or preventing the pipeline from running.
And since os.MkdirAll error code wasn't checked, the same attacker
could have precreated the directory where the pipeline is executed to
mess with the run, allowing code execution under the UID of the
agent (who has access to the toke, to communicate with the server, which
mean a attacker could inject a fake agent, steal credentials, etc)
* Do not filter on linux/amd64 per default & add tests
Tasks with no platform would otherwise not perform on runners with different OS/ARCH combos
Signed-off-by: Lukas Bachschwell <lukas@lbsfilm.at>
Co-authored-by: 6543 <6543@obermui.de>
hotfix #717
This comes from the agent being inactive / not sending and requesting any data if there a no pipelines waiting for him to execute. GRPC seems to only allow 2 pings without calling an actual endpoint before closing the connection. I think this will be indirectly solved in the moment we implement something like #536https://github.com/grpc/grpc/blob/master/doc/keepalive.md
Co-authored-by: Anbraten <anton@ju60.de>
Refactor
- use constants for strings
- more tests
- move constraint code into own package
Enhance
- all constrains use doublestart (glob pattern matching) now
Co-authored-by: Anbraten <anton@ju60.de>
Dropped support for `DRONE_*` environment variables in pipeline steps. Pipeline meta-data can be accessed with `CI_*` variables.
- `CI_*` prefix replaces `DRONE_*`
- `CI` value is now `woodpecker`
- `DRONE=true` has been removed
since github.com/docker/libcompose is deprecated, unmaintained and archived.
and license is the same as woodpecker's, we can just copy stuff into woodpecker directly.
(we only use types of that project anyway)
- Add json schema file
- Add tests to validate sample pipeline files
- Add new command `lint` to cli to test a directory or single file to use correct schema
Example: `woodpecker-cli lint ./pipeline/schema/.woodpecker/`
---
close#275
preparation for #276
Co-authored-by: 6543 <6543@obermui.de>
