Currently, backend options are parsed in the yaml parser.
This has some issues:
- backend specific code should be in the backend folders
- it is not possible to add backend options for backends added via
addons
This pull request addresses the issue https://github.com/woodpecker-ci/woodpecker/issues/3210.
Ideally, the Bitbucket API should include repository permissions when
utilizing the 'get all repositories' endpoint. However, as it currently
does not provide this information, a viable solution is to fetch all
permissions for every repository and then employ a dictionary to
associate each repository with its respective permissions.
Without implementing this fix, logging in becomes problematic for users
with access to a substantial number of repositories (300+), as the
process takes over 2 minutes to complete.
---------
Co-authored-by: Alberto Alcón <albertoalcon@bit2me.com>
Mostly those that did not require much work.
From #2960
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Because of the check `if step.Stopped == 0`
without the check there are edgecases where could be the case a stoped
steped can be markt as running again, witch is wrong.
I do remember we have "running" steps indefinetly in cancled pipelines.
This could be the fix, i just did not test that specific.
Anyway the func hat a good testcoverage ... so just look at the tests
_Source:
https://github.com/woodpecker-ci/woodpecker/pull/3143#discussion_r1446138088_
close#3109
~~also fix start time of steps to be set correctly~~ edgecase do not hit
anymore as we have a clear sepperation between workflows and steps now
:)
---------
Co-authored-by: Anbraten <anton@ju60.de>
closes https://github.com/woodpecker-ci/woodpecker/issues/3071
1. If a secret can be used on PRs, it can also be used on PR close.
2. If no events are set, disallow access to secret. This was different
before, secrets without any event set were allowed for all events.
3. Compare strings instead of patterns.
---------
Co-authored-by: 6543 <6543@obermui.de>
Replaces #2972Fixes#2483
Removed etag header as etag is used incorrectly, it should be based on
content not startup time and we don't handle it from request headers
anyway.
Transfer the agent hostname to the server's model.agent.Name field.
Previously the autoscaler had to create the agent with the server and
get a unique token to pass to the agent at boot up. This allows to get
the right name for the agent while using the shared master agent token
at agent boot up.
Both ways are still supported.
`[POST] /api/repos?forge_remote_id=12345`
```json
{"level":"error","error":"GetOrgByName","time":"2023-12-19T01:03:36Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/repo.go:143","message":"Could not fetch organization from forge."}
```
because `https://codeberg.org/api/v1/orgs/6543` returns an error.
`https://codeberg.org/api/v1/users/6543` do not.
close #2714
I just discovered that there is an organization created with name being empty.
we should at least catch it for now in the storage - and later trace
down why we get it in the first place
https://go.dev/doc/modules/release-workflow#breaking
Fixes https://github.com/woodpecker-ci/woodpecker/issues/2913 fixes
#2654
```
runephilosof@fedora:~/code/platform-woodpecker/woodpecker-repo-configurator (master)$ go get go.woodpecker-ci.org/woodpecker@v2.0.0
go: go.woodpecker-ci.org/woodpecker@v2.0.0: invalid version: module contains a go.mod file, so module path must match major version ("go.woodpecker-ci.org/woodpecker/v2")
```
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
implement this fix but with an additional field on workflows to not
change the workflow name
closes#1840closes#713
---------
Co-authored-by: 6543 <6543@obermui.de>
Previously, *all* log streams of the pipeline were closed everytime when
a workflow is done, if you have multiple workflows, streams will be
closed multiple times.
Closes https://github.com/woodpecker-ci/woodpecker/issues/2036
Add scopes needed when creating Github application.
Removed "repo:status" scope, because it is included in already requested
"repo" scope.
Fixes: #1081
- refactor pipeline parsing
- do not parse the pipeline multiple times to perform filter checks, do
this once and perform checks on the result directly
- code deduplication
- refactor forge token refreshing
- move refreshing to a helper func to reduce code
---------
Co-authored-by: Anbraten <anton@ju60.de>
closes#2514
The fix is simple, just providing a file name, so
`http.ServeContent(...)` can set the correct mimeype in case the content
is zero bytes.
The test was just extended.
PS: I would appreciate a `hacktoberfest-accepted` label ;)
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
I had experienced some issues running Woodpecker behind a reverse-proxy,
resulting from not defining the `WOODPECKER_ROOT_PATH` environment
variable in #2477.
As suggested by @qwerty287, specifying `WOODPECKER_ROOT_PATH=/foo`
*mostly* solved the issue of running the woodpecker server at an url
like `https://example.org/foo`.
However, the webhook urls and badge urls were generated excluding the
configured `WOODPECKER_ROOT_PATH`.
This PR (mostly) fixes issues related to non-empty
`WOODPECKER_ROOT_PATH`.
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/golang-jwt/jwt/v4](https://togithub.com/golang-jwt/jwt) |
require | major | `v4.5.0` -> `v5.0.0` |
---
### ⚠ Dependency Lookup Warnings ⚠
Warnings were logged while processing this repo. Please check the
Dependency Dashboard for more information.
---
### Release Notes
<details>
<summary>golang-jwt/jwt (github.com/golang-jwt/jwt/v4)</summary>
### [`v5.0.0`](https://togithub.com/golang-jwt/jwt/releases/tag/v5.0.0)
[Compare
Source](https://togithub.com/golang-jwt/jwt/compare/v4.5.0...v5.0.0)
### 🚀 New Major Version `v5` 🚀
It's finally here, the release you have been waiting for! We don't take
breaking changes lightly, but the changes outlined below were necessary
to address some of the challenges of the previous API. A big thanks for
[@​mfridman](https://togithub.com/mfridman) for all the reviews,
all contributors for their commits and of course
[@​dgrijalva](https://togithub.com/dgrijalva) for the original
code. I hope we kept some of the spirit of your original `v4` branch
alive in the approach we have taken here.
\~[@​oxisto](https://togithub.com/oxisto), on behalf of
[@​golang-jwt/maintainers](https://togithub.com/golang-jwt/maintainers)
Version `v5` contains a major rework of core functionalities in the
`jwt-go` library. This includes support for several validation options
as well as a re-design of the `Claims` interface. Lastly, we reworked
how errors work under the hood, which should provide a better overall
developer experience.
Starting from
[v5.0.0](https://togithub.com/golang-jwt/jwt/releases/tag/v5.0.0), the
import path will be:
"github.com/golang-jwt/jwt/v5"
For most users, changing the import path *should* suffice. However,
since we intentionally changed and cleaned some of the public API,
existing programs might need to be updated. The following sections
describe significant changes and corresponding updates for existing
programs.
#### Parsing and Validation Options
Under the hood, a new `validator` struct takes care of validating the
claims. A long awaited feature has been the option to fine-tune the
validation of tokens. This is now possible with several `ParserOption`
functions that can be appended to most `Parse` functions, such as
`ParseWithClaims`. The most important options and changes are:
- Added `WithLeeway` to support specifying the leeway that is allowed
when validating time-based claims, such as `exp` or `nbf`.
- Changed default behavior to not check the `iat` claim. Usage of this
claim is OPTIONAL according to the JWT RFC. The claim itself is also
purely informational according to the RFC, so a strict validation
failure is not recommended. If you want to check for sensible values in
these claims, please use the `WithIssuedAt` parser option.
- Added `WithAudience`, `WithSubject` and `WithIssuer` to support
checking for expected `aud`, `sub` and `iss`.
- Added `WithStrictDecoding` and `WithPaddingAllowed` options to allow
previously global settings to enable base64 strict encoding and the
parsing of base64 strings with padding. The latter is strictly speaking
against the standard, but unfortunately some of the major identity
providers issue some of these incorrect tokens. Both options are
disabled by default.
#### Changes to the `Claims` interface
##### Complete Restructuring
Previously, the claims interface was satisfied with an implementation of
a `Valid() error` function. This had several issues:
- The different claim types (struct claims, map claims, etc.) then
contained similar (but not 100 % identical) code of how this validation
was done. This lead to a lot of (almost) duplicate code and was hard to
maintain
- It was not really semantically close to what a "claim" (or a set of
claims) really is; which is a list of defined key/value pairs with a
certain semantic meaning.
Since all the validation functionality is now extracted into the
validator, all `VerifyXXX` and `Valid` functions have been removed from
the `Claims` interface. Instead, the interface now represents a list of
getters to retrieve values with a specific meaning. This allows us to
completely decouple the validation logic with the underlying storage
representation of the claim, which could be a struct, a map or even
something stored in a database.
```go
type Claims interface {
GetExpirationTime() (*NumericDate, error)
GetIssuedAt() (*NumericDate, error)
GetNotBefore() (*NumericDate, error)
GetIssuer() (string, error)
GetSubject() (string, error)
GetAudience() (ClaimStrings, error)
}
```
##### Supported Claim Types and Removal of `StandardClaims`
The two standard claim types supported by this library, `MapClaims` and
`RegisteredClaims` both implement the necessary functions of this
interface. The old `StandardClaims` struct, which has already been
deprecated in `v4` is now removed.
Users using custom claims, in most cases, will not experience any
changes in the behavior as long as they embedded `RegisteredClaims`. If
they created a new claim type from scratch, they now need to implemented
the proper getter functions.
##### Migrating Application Specific Logic of the old `Valid`
Previously, users could override the `Valid` method in a custom claim,
for example to extend the validation with application-specific claims.
However, this was always very dangerous, since once could easily disable
the standard validation and signature checking.
In order to avoid that, while still supporting the use-case, a new
`ClaimsValidator` interface has been introduced. This interface consists
of the `Validate() error` function. If the validator sees, that a
`Claims` struct implements this interface, the errors returned to the
`Validate` function will be *appended* to the regular standard
validation. It is not possible to disable the standard validation
anymore (even only by accident).
Usage examples can be found in [example_test.go](./example_test.go), to
build claims structs like the following.
```go
// MyCustomClaims includes all registered claims, plus Foo.
type MyCustomClaims struct {
Foo string `json:"foo"`
jwt.RegisteredClaims
}
// Validate can be used to execute additional application-specific claims
// validation.
func (m MyCustomClaims) Validate() error {
if m.Foo != "bar" {
return errors.New("must be foobar")
}
return nil
}
```
#### Changes to the `Token` and `Parser` struct
The previously global functions `DecodeSegment` and `EncodeSegment` were
moved to the `Parser` and `Token` struct respectively. This will allow
us in the future to configure the behavior of these two based on options
supplied on the parser or the token (creation). This also removes two
previously global variables and moves them to parser options
`WithStrictDecoding` and `WithPaddingAllowed`.
In order to do that, we had to adjust the way signing methods work.
Previously they were given a base64 encoded signature in `Verify` and
were expected to return a base64 encoded version of the signature in
`Sign`, both as a `string`. However, this made it necessary to have
`DecodeSegment` and `EncodeSegment` global and was a less than perfect
design because we were repeating encoding/decoding steps for all signing
methods. Now, `Sign` and `Verify` operate on a decoded signature as a
`[]byte`, which feels more natural for a cryptographic operation anyway.
Lastly, `Parse` and `SignedString` take care of the final
encoding/decoding part.
In addition to that, we also changed the `Signature` field on `Token`
from a `string` to `[]byte` and this is also now populated with the
decoded form. This is also more consistent, because the other parts of
the JWT, mainly `Header` and `Claims` were already stored in decoded
form in `Token`. Only the signature was stored in base64 encoded form,
which was redundant with the information in the `Raw` field, which
contains the complete token as base64.
```go
type Token struct {
Raw string // Raw contains the raw token
Method SigningMethod // Method is the signing method used or to be used
Header map[string]interface{} // Header is the first segment of the token in decoded form
Claims Claims // Claims is the second segment of the token in decoded form
Signature []byte // Signature is the third segment of the token in decoded form
Valid bool // Valid specifies if the token is valid
}
```
Most (if not all) of these changes should not impact the normal usage of
this library. Only users directly accessing the `Signature` field as
well as developers of custom signing methods should be affected.
#### What's Changed
- Added GitHub Actions Markdown by
[@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/260](https://togithub.com/golang-jwt/jwt/pull/260)
- Remove `StandardClaims` in favor of `RegisteredClaims` by
[@​oxisto](https://togithub.com/oxisto) in
[#​235](https://togithub.com/golang-jwt/jwt/issues/235)
- Adding more coverage by [@​oxisto](https://togithub.com/oxisto)
in [#​268](https://togithub.com/golang-jwt/jwt/issues/268)
- More consistent way of handling validation errors by
[@​oxisto](https://togithub.com/oxisto) in
[#​274](https://togithub.com/golang-jwt/jwt/issues/274)
- New Validation API by [@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/236](https://togithub.com/golang-jwt/jwt/pull/236)
- `v5` Pre-Release by [@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/234](https://togithub.com/golang-jwt/jwt/pull/234)
- no need for string slice and call to strings.join by
[@​moneszarrugh](https://togithub.com/moneszarrugh) in
[https://github.com/golang-jwt/jwt/pull/115](https://togithub.com/golang-jwt/jwt/pull/115)
- Update MIGRATION_GUIDE.md by
[@​liam-verta](https://togithub.com/liam-verta) in
[https://github.com/golang-jwt/jwt/pull/289](https://togithub.com/golang-jwt/jwt/pull/289)
- Moving `DecodeSegement` to `Parser` by
[@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/278](https://togithub.com/golang-jwt/jwt/pull/278)
- Adjusting the error checking example by
[@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/270](https://togithub.com/golang-jwt/jwt/pull/270)
- add documentation to hmac `Verify` & `Sign` to detail why string is
not an advisable input for key by
[@​dillonstreator](https://togithub.com/dillonstreator) in
[https://github.com/golang-jwt/jwt/pull/249](https://togithub.com/golang-jwt/jwt/pull/249)
- Add golangci-lint by [@​mfridman](https://togithub.com/mfridman)
in
[https://github.com/golang-jwt/jwt/pull/279](https://togithub.com/golang-jwt/jwt/pull/279)
- Added dependabot updates for GitHub actions by
[@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/298](https://togithub.com/golang-jwt/jwt/pull/298)
- Bump actions/checkout from 2 to 3 by
[@​dependabot](https://togithub.com/dependabot) in
[https://github.com/golang-jwt/jwt/pull/299](https://togithub.com/golang-jwt/jwt/pull/299)
- Bump actions/setup-go from 3 to 4 by
[@​dependabot](https://togithub.com/dependabot) in
[https://github.com/golang-jwt/jwt/pull/300](https://togithub.com/golang-jwt/jwt/pull/300)
- Added coverage reporting by
[@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/304](https://togithub.com/golang-jwt/jwt/pull/304)
- Last Documentation cleanups for `v5` release by
[@​oxisto](https://togithub.com/oxisto) in
[https://github.com/golang-jwt/jwt/pull/291](https://togithub.com/golang-jwt/jwt/pull/291)
- enable jwt.ParsePublicKeyFromPEM to parse PKCS1 Public Key by
[@​twocs](https://togithub.com/twocs) in
[https://github.com/golang-jwt/jwt/pull/120](https://togithub.com/golang-jwt/jwt/pull/120)
#### New Contributors
- [@​moneszarrugh](https://togithub.com/moneszarrugh) made their
first contribution in
[https://github.com/golang-jwt/jwt/pull/115](https://togithub.com/golang-jwt/jwt/pull/115)
- [@​liam-verta](https://togithub.com/liam-verta) made their first
contribution in
[https://github.com/golang-jwt/jwt/pull/289](https://togithub.com/golang-jwt/jwt/pull/289)
- [@​dillonstreator](https://togithub.com/dillonstreator) made
their first contribution in
[https://github.com/golang-jwt/jwt/pull/249](https://togithub.com/golang-jwt/jwt/pull/249)
- [@​dependabot](https://togithub.com/dependabot) made their first
contribution in
[https://github.com/golang-jwt/jwt/pull/299](https://togithub.com/golang-jwt/jwt/pull/299)
- [@​twocs](https://togithub.com/twocs) made their first
contribution in
[https://github.com/golang-jwt/jwt/pull/120](https://togithub.com/golang-jwt/jwt/pull/120)
**Full Changelog**:
https://github.com/golang-jwt/jwt/compare/v4.5.0...v5.0.0
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/woodpecker-ci/woodpecker).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi44My4wIiwidXBkYXRlZEluVmVyIjoiMzYuODMuMCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: qwerty287 <ndev@web.de>
There was no permission check when looking up repos so you were able to
get basic repo information even if you're not allowed to.
This uses `session.MustPull` (and set repo/perms before) to fix this.
Currently it would fail with panic:
```
runtime error: invalid memory address or nil pointer dereference
...
/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/forge/gitea/gitea.go:492 (0xdfb32e)
```
For "woodpecker-ci" the `name` is "Woodpecker CI"
and the `login` is "woodpecker-ci"
Fixes#2092
This was causing the organization lookup to fail, because it looks up
using the `login`, when it did not find the organization, it would try
to create it. The creation would fail, because it uses the `name`, and
an organization with that `name` already exists.
Resulting in:
```
pq: duplicate key value violates unique constraint "UQE_orgs_name"
```
error logs like:
```
{"level":"warn","error":"sql: no rows in result set","time":"2023-07-25T21:29:56Z"}
```
have to less context to be helpfull.
this will cange it as the message now looks like:
```
{"level":"warn","error":"GetPipelineLast: sql: no rows in result set", "time":"2023-07-27T02:54:25+02:00"}
```
closes#1743
fixes: setting secrets for own user namespace
- create org in database
- use orgID for org related APIs
Co-authored-by: 6543 <6543@obermui.de>
In order to test this functionality, we had to expose the `BranchHead()`
through an API endpoint
```
GET /repos/{repo_id}/branches/{branch}/head
```
The response is a string that contains the latest commit hash of the
requested branch.
for now it's not clear defined, what to do on an unsupported event.
e.g. gitea webhook panel shows 500 error and no message.
now we have a successful webhook and a message to show an info
This PR introduces two new server configuration options, for providing a
custom .JS and .CSS file.
These can be used to show custom banner messages, add
environment-dependent signals, or simply a corporate logo.
### Motivation (what problem I try to solve)
I'm operating Woodpecker in multiple k8s clusters for different
environments.
When having multiple browser tabs open, I prefer strong indicators for
each environment.
E.g. a red "PROD" banner, or just a blue "QA" banner.
Also, we sometimes need to have the chance for maintenance, and instead
of broadcasting emails,
I prefer a banner message, stating something like: "Heads-up: there's a
planned downtime, next Friday, blabla...".
Also, I like to have the firm's logo visible, which makes Woodpecker
look more like an integral part of our platform.
### Implementation notes
* Two new config options are introduced ```WOODPECKER_CUSTOM_CSS_FILE```
and ```WOODPECKER_CUSTOM_JS_FILE```
* I've piggy-bagged the existing handler for assets, as it seemed to me
a minimally invasive approach
* the option along with an example is documented
* a simple unit test for the Gin-handler ensures some regression safety
* no extra dependencies are introduced
### Visual example
The documented example will look like this.
### Areas of uncertainty
This is my first contribution to Woodpecker and I tried my best to align
with your conventions.
That said, I found myself uncertain about these things and would be glad
about getting feedback.
* The handler tests are somewhat different than the other ones because I
wanted to keep them simple - I hope that still matches your coding
guidelines
* caching the page sometimes will let the browser not recognize changes
and a user must reload. I'm not fully into the details of how caching is
implemented and neither can judge if it's a real problem. Another pair
of eyes would be good.
Using an empty token for an agent was returning the first agent from the
database as the orm is not adding where clauses for empty strings of a
model when querying.
# Huge thanks for reporting and explaining the issue ❤️
- Dominik Heidler
- Timo Tomasini
closes#1801closes#1815closes#1144
closes #983
closes #557closes#1827
regression of #1791
# TODO
- [x] adjust log model
- [x] add migration for logs
- [x] send log line via grpc using step-id
- [x] save log-line to db
- [x] stream log-lines to UI
- [x] use less structs for log-data
- [x] make web UI work
- [x] display logs loaded from db
- [x] display streaming logs
- [ ] ~~make migration work~~ -> dedicated pull (#1828)
# TESTED
- [x] new logs are stored in database
- [x] log retrieval via cli (of new logs) works
- [x] log streaming works (tested via curl & webui)
- [x] log retrieval via web (of new logs) works
---------
Co-authored-by: 6543 <6543@obermui.de>
This isolates single migration tasks from each other.
The migration itself is now not atomic anymore but each single migration
now on it's own.
This takes load away from databases, as new sessions have a committed
schema available.
We also disable xorm.cache, as the speed improvements are minor but
invalid cache caused by schema changes did happen already in the past.
---------
Reverts #1817Closes#1821
---------
Co-authored-by: 6543 <6543@obermui.de>
# Summary
This PR drops the outdated former swagger.yaml/json and introduced
automatic API document generation from Go code.
The generated code is also used to generate documentation/markdown for
the community page,
as well as enable the Woodpecker server to serve a Swagger Web UI for
manual tinkering.
I did opt-in for gin-swagger, a middleware for the Gin framework, to
ease implementation and have a sophisticated output.
This middleware only produces Swagger v2 specs. AFAIK the newer OpenApi
3x tooling is not yet that mature,
so I guess that's fine for now.
## Implemenation notes
- former swagger.json files removed
- former // swagger godocs removed
- introduced new dependency gin-swagger, which uses godoc annotations on
top of Gin Handler functions.
- reworked Makefile to automatically generate Go code for the server
- introduce new dependency go-swagger, to generate Markdown for
documentation purposes
- add a Swagger Web UI, incl. capabilities for manual API exploration
- consider relative root paths in the implementation
- write documentation for all exposed API endpoints
- incl. API docs in the community website (auto-generated)
- provide developer documentation, for the Woodpecker authors
- no other existing logic/code was intentionally changed
---------
close#292
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
