Deprecate alternative names on secrets (#3406)

Closes https://github.com/woodpecker-ci/woodpecker/discussions/2274

# deprecation of alternative names

Instead of
```yaml
secrets:
  - source: some_secret
    target: some_env
```
you now write:
```yaml
environment:
  some_env:
    from_secret: some_secret
```

Also, it's possible to use complex yaml objects in `environment`,
they're turned into json (just like `settings`).

docs/docs/20-usage/20-workflow-syntax.md

@@ -164,13 +164,13 @@ Allows you to specify the entrypoint for containers. Note that this must be a li
 
 Woodpecker provides the ability to pass environment variables to individual steps.
 
-For more details check the [environment docs](./50-environment.md).
+For more details, check the [environment docs](./50-environment.md).
 
 ### `secrets`
 
 Woodpecker provides the ability to store named parameters external to the YAML configuration file, in a central secret store. These secrets can be passed to individual steps of the workflow at runtime.
 
-For more details check the [secrets docs](./40-secrets.md).
+For more details, check the [secrets docs](./40-secrets.md).
 
 ### `failure`
 
@@ -574,10 +574,10 @@ For more details check the [matrix build docs](./30-matrix-workflows.md).
 
 You can set labels for your workflow to select an agent to execute the workflow on. An agent will pick up and run a workflow when **every** label assigned to it matches the agents labels.
 
-To set additional agent labels check the [agent configuration options](../30-administration/15-agent-config.md#woodpecker_filter_labels). Agents will have at least four default labels: `platform=agent-os/agent-arch`, `hostname=my-agent`, `backend=docker` (type of the agent backend) and `repo=*`. Agents can use a `*` as a wildcard for a label. For example `repo=*` will match every repo.
+To set additional agent labels, check the [agent configuration options](../30-administration/15-agent-config.md#woodpecker_filter_labels). Agents will have at least four default labels: `platform=agent-os/agent-arch`, `hostname=my-agent`, `backend=docker` (type of the agent backend) and `repo=*`. Agents can use a `*` as a wildcard for a label. For example `repo=*` will match every repo.
 
 Workflow labels with an empty value will be ignored.
-By default each workflow has at least the `repo=your-user/your-repo-name` label. If you have set the [platform attribute](#platform) for your workflow it will have a label like `platform=your-os/your-arch` as well.
+By default, each workflow has at least the `repo=your-user/your-repo-name` label. If you have set the [platform attribute](#platform) for your workflow it will have a label like `platform=your-os/your-arch` as well.
 
 You can add additional labels as a key value map:
 

docs/docs/20-usage/40-secrets.md

@@ -28,20 +28,20 @@ once their usage is declared in the `secrets` section:
 
 The case of the environment variables is not changed, but secret matching is done case-insensitively. In the example above, `DOCKER_PASSWORD` would also match if the secret is called `docker_password`.
 
-### Use secrets in settings
+### Use secrets in settings and environment
 
-Alternatively, you can get a `setting` from secrets using the `from_secret` syntax.
-In this example, the secret named `secret_token` would be passed to the setting named `token`, which will be available in the plugin as environment variable named `PLUGIN_TOKEN`. See [Plugins](./51-plugins/20-creating-plugins.md#settings) for details.
+You can set an setting or environment value from secrets using the `from_secret` syntax.
 
-:::note
-The `from_secret` syntax only works with the newer `settings` block.
-:::
+In this example, the secret named `secret_token` would be passed to the setting named `token`,which will be available in the plugin as environment variable named `PLUGIN_TOKEN` (See [plugins](./51-plugins/20-creating-plugins.md#settings) for details), and to the environment variable `TOKEN_ENV`.
 
 ```diff
  steps:
    - name: docker
      image: my-plugin
-     settings:
++    environment:
++      TOKEN_ENV:
++        from_secret: secret_token
++    settings:
 +      token:
 +        from_secret: secret_token
 ```
@@ -62,21 +62,6 @@ Please note parameter expressions are subject to pre-processing. When using secr
      secrets: [ docker_username, DOCKER_PASSWORD ]
 ```
 
-### Alternate Names
-
-There may be scenarios where you are required to store secrets using alternate names. You can map the alternate secret name to the expected name using the below syntax:
-
-```diff
- steps:
-   - name: docker
-     image: plugins/docker
-     repo: octocat/hello-world
-     tags: latest
-+    secrets:
-+      - source: docker_prod_password
-+        target: docker_password
-```
-
 ### Use in Pull Requests events
 
 Secrets are not exposed to pull requests by default. You can override this behavior by creating the secret and enabling the `pull_request` event type, either in UI or by CLI, see below.

docs/docs/20-usage/50-environment.md

@@ -7,9 +7,9 @@ Woodpecker provides the ability to pass environment variables to individual pipe
    - name: build
      image: golang
 +    environment:
-+      - CGO=0
-+      - GOOS=linux
-+      - GOARCH=amd64
++      CGO: 0
++      GOOS: linux
++      GOARCH: amd64
      commands:
        - go build
        - go test

docs/docs/91-migrations.md

@@ -9,6 +9,8 @@ Some versions need some changes to the server configuration or the pipeline conf
 - Pipelines without a config file will now be skipped instead of failing
 - Deprecated `includes` and `excludes` support from **event** filter
 - Deprecated uppercasing all secret env vars, instead, the value of the `secrets` property is used. [Read more](./20-usage/40-secrets.md#use-secrets-in-commands)
+- Deprecated alternative names for secrets, use `environment` with `from_secret`
+- Deprecated slice definition for env vars
 
 ## 2.0.0
 

pipeline/frontend/yaml/compiler/compiler.go

@@ -179,10 +179,12 @@ func (c *Compiler) Compile(conf *yaml_types.Workflow) (*backend_types.Config, er
 			cloneSettings["tags"] = "true"
 		}
 		container := &yaml_types.Container{
-			Name:        defaultCloneName,
-			Image:       cloneImage,
-			Settings:    cloneSettings,
-			Environment: c.cloneEnv,
+			Name:     defaultCloneName,
+			Image:    cloneImage,
+			Settings: cloneSettings,
+		}
+		for k, v := range c.cloneEnv {
+			container.Environment[k] = v
 		}
 		step, err := c.createProcess(container, backend_types.StepTypeClone)
 		if err != nil {

pipeline/frontend/yaml/compiler/convert.go

@@ -76,7 +76,6 @@ func (c *Compiler) createProcess(container *yaml_types.Container, stepType backe
 
 	// append default environment variables
 	environment := map[string]string{}
-	maps.Copy(environment, container.Environment)
 	maps.Copy(environment, c.env)
 
 	environment["CI_WORKSPACE"] = path.Join(c.base, c.path)
@@ -112,6 +111,10 @@ func (c *Compiler) createProcess(container *yaml_types.Container, stepType backe
 		}
 	}
 
+	if err := settings.ParamsToEnv(container.Environment, environment, getSecretValue); err != nil {
+		return nil, err
+	}
+
 	for _, requested := range container.Secrets.Secrets {
 		secretValue, err := getSecretValue(requested.Source)
 		if err != nil {

pipeline/frontend/yaml/linter/linter.go

@@ -295,6 +295,23 @@ func (l *Linter) lintDeprecations(config *WorkflowConfig) (err error) {
 		}
 	}
 
+	for _, step := range parsed.Steps.ContainerList {
+		for i, c := range step.Secrets.Secrets {
+			if c.Source != c.Target {
+				err = multierr.Append(err, &errors.PipelineError{
+					Type:    errors.PipelineErrorTypeDeprecation,
+					Message: "Secrets alternative names are deprecated, use environment with from_secret",
+					Data: errors.DeprecationErrorData{
+						File:  config.File,
+						Field: fmt.Sprintf("steps.%s.secrets[%d]", step.Name, i),
+						Docs:  "https://woodpecker-ci.org/docs/usage/workflow-syntax#event",
+					},
+					IsWarning: true,
+				})
+			}
+		}
+	}
+
 	return err
 }
 

pipeline/frontend/yaml/parse.go

@@ -21,7 +21,6 @@ import (
 
 	"go.woodpecker-ci.org/woodpecker/v2/pipeline/frontend/yaml/constraint"
 	"go.woodpecker-ci.org/woodpecker/v2/pipeline/frontend/yaml/types"
-	"go.woodpecker-ci.org/woodpecker/v2/pipeline/frontend/yaml/types/base"
 )
 
 // ParseBytes parses the configuration from bytes b.
@@ -53,7 +52,7 @@ func ParseBytes(b []byte) (*types.Workflow, error) {
 	// support deprecated platform filter
 	if out.PlatformDoNotUseIt != "" {
 		if out.Labels == nil {
-			out.Labels = make(base.SliceOrMap)
+			out.Labels = make(map[string]string)
 		}
 		if _, set := out.Labels["platform"]; !set {
 			out.Labels["platform"] = out.PlatformDoNotUseIt

pipeline/frontend/yaml/types/base/base_types_test.go

@@ -1,121 +0,0 @@
-// Copyright 2023 Woodpecker Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package base
-
-import (
-	"fmt"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-	"gopkg.in/yaml.v3"
-)
-
-type StructStringorInt struct {
-	Foo StringOrInt
-}
-
-func TestStringorIntYaml(t *testing.T) {
-	for _, str := range []string{`{foo: 10}`, `{foo: "10"}`} {
-		s := StructStringorInt{}
-		assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
-
-		assert.Equal(t, StringOrInt(10), s.Foo)
-
-		d, err := yaml.Marshal(&s)
-		assert.NoError(t, err)
-
-		s2 := StructStringorInt{}
-		assert.NoError(t, yaml.Unmarshal(d, &s2))
-
-		assert.Equal(t, StringOrInt(10), s2.Foo)
-	}
-}
-
-type StructStringOrSlice struct {
-	Foo StringOrSlice
-}
-
-func TestStringOrSliceYaml(t *testing.T) {
-	str := `{foo: [bar, "baz"]}`
-	s := StructStringOrSlice{}
-	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
-	assert.Equal(t, StringOrSlice{"bar", "baz"}, s.Foo)
-
-	d, err := yaml.Marshal(&s)
-	assert.NoError(t, err)
-
-	s = StructStringOrSlice{}
-	assert.NoError(t, yaml.Unmarshal(d, &s))
-	assert.Equal(t, StringOrSlice{"bar", "baz"}, s.Foo)
-
-	str = `{foo: []}`
-	s = StructStringOrSlice{}
-	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
-	assert.Equal(t, StringOrSlice{}, s.Foo)
-
-	str = `{}`
-	s = StructStringOrSlice{}
-	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
-	assert.Nil(t, s.Foo)
-}
-
-type StructSliceorMap struct {
-	Foos SliceOrMap `yaml:"foos,omitempty"`
-	Bars []string   `yaml:"bars"`
-}
-
-func TestSliceOrMapYaml(t *testing.T) {
-	str := `{foos: [bar=baz, far=faz]}`
-
-	s := StructSliceorMap{}
-	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
-
-	assert.Equal(t, SliceOrMap{"bar": "baz", "far": "faz"}, s.Foos)
-
-	d, err := yaml.Marshal(&s)
-	assert.NoError(t, err)
-
-	s2 := StructSliceorMap{}
-	assert.NoError(t, yaml.Unmarshal(d, &s2))
-
-	assert.Equal(t, SliceOrMap{"bar": "baz", "far": "faz"}, s2.Foos)
-}
-
-var sampleStructSliceorMap = `
-foos:
-  io.rancher.os.bar: baz
-  io.rancher.os.far: true
-bars: []
-`
-
-func TestUnmarshalSliceOrMap(t *testing.T) {
-	s := StructSliceorMap{}
-	err := yaml.Unmarshal([]byte(sampleStructSliceorMap), &s)
-	assert.Equal(t, fmt.Errorf("cannot unmarshal 'true' of type bool into a string value"), err)
-}
-
-func TestStr2SliceOrMapPtrMap(t *testing.T) {
-	s := map[string]*StructSliceorMap{"udav": {
-		Foos: SliceOrMap{"io.rancher.os.bar": "baz", "io.rancher.os.far": "true"},
-		Bars: []string{},
-	}}
-	d, err := yaml.Marshal(&s)
-	assert.NoError(t, err)
-
-	s2 := map[string]*StructSliceorMap{}
-	assert.NoError(t, yaml.Unmarshal(d, &s2))
-
-	assert.Equal(t, s, s2)
-}

pipeline/frontend/yaml/types/base/int_test.go

@@ -0,0 +1,43 @@
+// Copyright 2024 Woodpecker Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package base
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"gopkg.in/yaml.v3"
+)
+
+type StructStringOrInt struct {
+	Foo StringOrInt
+}
+
+func TestStringOrIntYaml(t *testing.T) {
+	for _, str := range []string{`{foo: 10}`, `{foo: "10"}`} {
+		s := StructStringOrInt{}
+		assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
+
+		assert.Equal(t, StringOrInt(10), s.Foo)
+
+		d, err := yaml.Marshal(&s)
+		assert.NoError(t, err)
+
+		s2 := StructStringOrInt{}
+		assert.NoError(t, yaml.Unmarshal(d, &s2))
+
+		assert.Equal(t, StringOrInt(10), s2.Foo)
+	}
+}

pipeline/frontend/yaml/types/base/map.go

@@ -21,13 +21,13 @@ import (
 )
 
 // SliceOrMap represents a map of strings, string slice are converted into a map
-type SliceOrMap map[string]string
+type SliceOrMap map[string]any
 
 // UnmarshalYAML implements the Unmarshaler interface.
 func (s *SliceOrMap) UnmarshalYAML(unmarshal func(any) error) error {
 	var sliceType []any
 	if err := unmarshal(&sliceType); err == nil {
-		parts := map[string]string{}
+		parts := map[string]any{}
 		for _, s := range sliceType {
 			if str, ok := s.(string); ok {
 				str := strings.TrimSpace(str)
@@ -47,21 +47,9 @@ func (s *SliceOrMap) UnmarshalYAML(unmarshal func(any) error) error {
 		return nil
 	}
 
-	var mapType map[any]any
+	var mapType map[string]any
 	if err := unmarshal(&mapType); err == nil {
-		parts := map[string]string{}
-		for k, v := range mapType {
-			if sk, ok := k.(string); ok {
-				if sv, ok := v.(string); ok {
-					parts[sk] = sv
-				} else {
-					return fmt.Errorf("cannot unmarshal '%v' of type %T into a string value", v, v)
-				}
-			} else {
-				return fmt.Errorf("cannot unmarshal '%v' of type %T into a string value", k, k)
-			}
-		}
-		*s = parts
+		*s = mapType
 		return nil
 	}
 

pipeline/frontend/yaml/types/base/map_test.go

@@ -0,0 +1,58 @@
+// Copyright 2024 Woodpecker Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package base
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"gopkg.in/yaml.v3"
+)
+
+type StructSliceOrMap struct {
+	Foos SliceOrMap `yaml:"foos,omitempty"`
+	Bars []string   `yaml:"bars"`
+}
+
+func TestSliceOrMapYaml(t *testing.T) {
+	str := `{foos: [bar=baz, far=faz]}`
+
+	s := StructSliceOrMap{}
+	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
+
+	assert.Equal(t, SliceOrMap{"bar": "baz", "far": "faz"}, s.Foos)
+
+	d, err := yaml.Marshal(&s)
+	assert.NoError(t, err)
+
+	s2 := StructSliceOrMap{}
+	assert.NoError(t, yaml.Unmarshal(d, &s2))
+
+	assert.Equal(t, SliceOrMap{"bar": "baz", "far": "faz"}, s2.Foos)
+}
+
+func TestStr2SliceOrMapPtrMap(t *testing.T) {
+	s := map[string]*StructSliceOrMap{"udav": {
+		Foos: SliceOrMap{"io.rancher.os.bar": "baz", "io.rancher.os.far": "true"},
+		Bars: []string{},
+	}}
+	d, err := yaml.Marshal(&s)
+	assert.NoError(t, err)
+
+	s2 := map[string]*StructSliceOrMap{}
+	assert.NoError(t, yaml.Unmarshal(d, &s2))
+
+	assert.Equal(t, s, s2)
+}

pipeline/frontend/yaml/types/base/slice_test.go

@@ -0,0 +1,50 @@
+// Copyright 2024 Woodpecker Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package base
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"gopkg.in/yaml.v3"
+)
+
+type StructStringOrSlice struct {
+	Foo StringOrSlice
+}
+
+func TestStringOrSliceYaml(t *testing.T) {
+	str := `{foo: [bar, "baz"]}`
+	s := StructStringOrSlice{}
+	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
+	assert.Equal(t, StringOrSlice{"bar", "baz"}, s.Foo)
+
+	d, err := yaml.Marshal(&s)
+	assert.NoError(t, err)
+
+	s = StructStringOrSlice{}
+	assert.NoError(t, yaml.Unmarshal(d, &s))
+	assert.Equal(t, StringOrSlice{"bar", "baz"}, s.Foo)
+
+	str = `{foo: []}`
+	s = StructStringOrSlice{}
+	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
+	assert.Equal(t, StringOrSlice{}, s.Foo)
+
+	str = `{}`
+	s = StructStringOrSlice{}
+	assert.NoError(t, yaml.Unmarshal([]byte(str), &s))
+	assert.Nil(t, s.Foo)
+}

pipeline/frontend/yaml/types/container.go

@@ -38,19 +38,22 @@ type (
 		Entrypoint     base.StringOrSlice `yaml:"entrypoint,omitempty"`
 		Detached       bool               `yaml:"detach,omitempty"`
 		Directory      string             `yaml:"directory,omitempty"`
-		Environment    base.SliceOrMap    `yaml:"environment,omitempty"`
 		Failure        string             `yaml:"failure,omitempty"`
 		Group          string             `yaml:"group,omitempty"`
 		Image          string             `yaml:"image,omitempty"`
 		Name           string             `yaml:"name,omitempty"`
 		Pull           bool               `yaml:"pull,omitempty"`
-		Secrets        Secrets            `yaml:"secrets,omitempty"`
 		Settings       map[string]any     `yaml:"settings"`
 		Volumes        Volumes            `yaml:"volumes,omitempty"`
 		When           constraint.When    `yaml:"when,omitempty"`
 		Ports          []string           `yaml:"ports,omitempty"`
 		DependsOn      base.StringOrSlice `yaml:"depends_on,omitempty"`
 
+		// TODO make []string in 3.x
+		Secrets Secrets `yaml:"secrets,omitempty"`
+		// TODO make map[string]any in 3.x
+		Environment base.SliceOrMap `yaml:"environment,omitempty"`
+
 		// Docker and Kubernetes Specific
 		Privileged bool `yaml:"privileged,omitempty"`
 

pipeline/frontend/yaml/types/workflow.go

@@ -22,15 +22,15 @@ import (
 type (
 	// Workflow defines a workflow configuration.
 	Workflow struct {
-		When      constraint.When `yaml:"when,omitempty"`
-		Workspace Workspace       `yaml:"workspace,omitempty"`
-		Clone     ContainerList   `yaml:"clone,omitempty"`
-		Steps     ContainerList   `yaml:"steps,omitempty"`
-		Services  ContainerList   `yaml:"services,omitempty"`
-		Labels    base.SliceOrMap `yaml:"labels,omitempty"`
-		DependsOn []string        `yaml:"depends_on,omitempty"`
-		RunsOn    []string        `yaml:"runs_on,omitempty"`
-		SkipClone bool            `yaml:"skip_clone"`
+		When      constraint.When   `yaml:"when,omitempty"`
+		Workspace Workspace         `yaml:"workspace,omitempty"`
+		Clone     ContainerList     `yaml:"clone,omitempty"`
+		Steps     ContainerList     `yaml:"steps,omitempty"`
+		Services  ContainerList     `yaml:"services,omitempty"`
+		Labels    map[string]string `yaml:"labels,omitempty"`
+		DependsOn []string          `yaml:"depends_on,omitempty"`
+		RunsOn    []string          `yaml:"runs_on,omitempty"`
+		SkipClone bool              `yaml:"skip_clone"`
 
 		// Undocumented
 		Cache    base.StringOrSlice `yaml:"cache,omitempty"`