gotosocial/testrig/transportcontroller.go

294 lines
9.9 KiB
Go
Raw Normal View History

/*
GoToSocial
2021-12-20 17:42:19 +00:00
Copyright (C) 2021-2022 GoToSocial Authors admin@gotosocial.org
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
package testrig
import (
"bytes"
"encoding/json"
"io"
"net/http"
"strings"
"sync"
2021-11-13 16:29:43 +00:00
"github.com/superseriousbusiness/activity/pub"
"github.com/superseriousbusiness/activity/streams"
"github.com/superseriousbusiness/activity/streams/vocab"
apimodel "github.com/superseriousbusiness/gotosocial/internal/api/model"
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 09:16:43 +00:00
"github.com/superseriousbusiness/gotosocial/internal/concurrency"
"github.com/superseriousbusiness/gotosocial/internal/db"
"github.com/superseriousbusiness/gotosocial/internal/federation"
[chore] use our own logging implementation (#716) * first commit Signed-off-by: kim <grufwub@gmail.com> * replace logging with our own log library Signed-off-by: kim <grufwub@gmail.com> * fix imports Signed-off-by: kim <grufwub@gmail.com> * fix log imports Signed-off-by: kim <grufwub@gmail.com> * add license text Signed-off-by: kim <grufwub@gmail.com> * fix package import cycle between config and log package Signed-off-by: kim <grufwub@gmail.com> * fix empty kv.Fields{} being passed to WithFields() Signed-off-by: kim <grufwub@gmail.com> * fix uses of log.WithFields() with whitespace issues and empty slices Signed-off-by: kim <grufwub@gmail.com> * *linter related grumbling* Signed-off-by: kim <grufwub@gmail.com> * gofmt the codebase! also fix more log.WithFields() formatting issues Signed-off-by: kim <grufwub@gmail.com> * update testrig code to match new changes Signed-off-by: kim <grufwub@gmail.com> * fix error wrapping in non fmt.Errorf function Signed-off-by: kim <grufwub@gmail.com> * add benchmarking of log.Caller() vs non-cached Signed-off-by: kim <grufwub@gmail.com> * fix syslog tests, add standard build tags to test runner to ensure consistency Signed-off-by: kim <grufwub@gmail.com> * make syslog tests more robust Signed-off-by: kim <grufwub@gmail.com> * fix caller depth arithmatic (is that how you spell it?) Signed-off-by: kim <grufwub@gmail.com> * update to use unkeyed fields in kv.Field{} instances Signed-off-by: kim <grufwub@gmail.com> * update go-kv library Signed-off-by: kim <grufwub@gmail.com> * update libraries list Signed-off-by: kim <grufwub@gmail.com> * fuck you linter get nerfed Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 08:47:55 +00:00
"github.com/superseriousbusiness/gotosocial/internal/log"
"github.com/superseriousbusiness/gotosocial/internal/messages"
"github.com/superseriousbusiness/gotosocial/internal/transport"
)
[chore] use our own logging implementation (#716) * first commit Signed-off-by: kim <grufwub@gmail.com> * replace logging with our own log library Signed-off-by: kim <grufwub@gmail.com> * fix imports Signed-off-by: kim <grufwub@gmail.com> * fix log imports Signed-off-by: kim <grufwub@gmail.com> * add license text Signed-off-by: kim <grufwub@gmail.com> * fix package import cycle between config and log package Signed-off-by: kim <grufwub@gmail.com> * fix empty kv.Fields{} being passed to WithFields() Signed-off-by: kim <grufwub@gmail.com> * fix uses of log.WithFields() with whitespace issues and empty slices Signed-off-by: kim <grufwub@gmail.com> * *linter related grumbling* Signed-off-by: kim <grufwub@gmail.com> * gofmt the codebase! also fix more log.WithFields() formatting issues Signed-off-by: kim <grufwub@gmail.com> * update testrig code to match new changes Signed-off-by: kim <grufwub@gmail.com> * fix error wrapping in non fmt.Errorf function Signed-off-by: kim <grufwub@gmail.com> * add benchmarking of log.Caller() vs non-cached Signed-off-by: kim <grufwub@gmail.com> * fix syslog tests, add standard build tags to test runner to ensure consistency Signed-off-by: kim <grufwub@gmail.com> * make syslog tests more robust Signed-off-by: kim <grufwub@gmail.com> * fix caller depth arithmatic (is that how you spell it?) Signed-off-by: kim <grufwub@gmail.com> * update to use unkeyed fields in kv.Field{} instances Signed-off-by: kim <grufwub@gmail.com> * update go-kv library Signed-off-by: kim <grufwub@gmail.com> * update libraries list Signed-off-by: kim <grufwub@gmail.com> * fuck you linter get nerfed Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 08:47:55 +00:00
const (
applicationJSON = "application/json"
applicationActivityJSON = "application/activity+json"
)
// NewTestTransportController returns a test transport controller with the given http client.
//
// Obviously for testing purposes you should not be making actual http calls to other servers.
// To obviate this, use the function NewMockHTTPClient in this package to return a mock http
// client that doesn't make any remote calls but just returns whatever you tell it to.
//
// Unlike the other test interfaces provided in this package, you'll probably want to call this function
// PER TEST rather than per suite, so that the do function can be set on a test by test (or even more granular)
// basis.
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564) * cache transports in controller by privkey-generated pubkey, add retry logic to transport requests Signed-off-by: kim <grufwub@gmail.com> * update code comments, defer mutex unlocks Signed-off-by: kim <grufwub@gmail.com> * add count to 'performing request' log message Signed-off-by: kim <grufwub@gmail.com> * reduce repeated conversions of same url.URL object Signed-off-by: kim <grufwub@gmail.com> * move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue Signed-off-by: kim <grufwub@gmail.com> * fix security advisories regarding max outgoing conns, max rsp body size - implemented by a new httpclient.Client{} that wraps an underlying client with a queue to limit connections, and limit reader wrapping a response body with a configured maximum size - update pub.HttpClient args passed around to be this new httpclient.Client{} Signed-off-by: kim <grufwub@gmail.com> * add httpclient tests, move ip validation to separate package + change mechanism Signed-off-by: kim <grufwub@gmail.com> * fix merge conflicts Signed-off-by: kim <grufwub@gmail.com> * use singular mutex in transport rather than separate signer mus Signed-off-by: kim <grufwub@gmail.com> * improved useragent string Signed-off-by: kim <grufwub@gmail.com> * add note regarding missing test Signed-off-by: kim <grufwub@gmail.com> * remove useragent field from transport (instead store in controller) Signed-off-by: kim <grufwub@gmail.com> * shutup linter Signed-off-by: kim <grufwub@gmail.com> * reset other signing headers on each loop iteration Signed-off-by: kim <grufwub@gmail.com> * respect request ctx during retry-backoff sleep period Signed-off-by: kim <grufwub@gmail.com> * use external pkg with docs explaining performance "hack" Signed-off-by: kim <grufwub@gmail.com> * use http package constants instead of string method literals Signed-off-by: kim <grufwub@gmail.com> * add license file headers Signed-off-by: kim <grufwub@gmail.com> * update code comment to match new func names Signed-off-by: kim <grufwub@gmail.com> * updates to user-agent string Signed-off-by: kim <grufwub@gmail.com> * update signed testrig models to fit with new transport logic (instead uses separate signer now) Signed-off-by: kim <grufwub@gmail.com> * fuck you linter Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 09:16:43 +00:00
func NewTestTransportController(client pub.HttpClient, db db.DB, fedWorker *concurrency.WorkerPool[messages.FromFederator]) transport.Controller {
return transport.NewController(db, NewTestFederatingDB(db, fedWorker), &federation.Clock{}, client)
}
type MockHTTPClient struct {
do func(req *http.Request) (*http.Response, error)
testRemoteStatuses map[string]vocab.ActivityStreamsNote
testRemotePeople map[string]vocab.ActivityStreamsPerson
testRemoteGroups map[string]vocab.ActivityStreamsGroup
testRemoteServices map[string]vocab.ActivityStreamsService
testRemoteAttachments map[string]RemoteAttachmentFile
SentMessages sync.Map
}
// NewMockHTTPClient returns a client that conforms to the pub.HttpClient interface.
//
// If do is nil, then a standard response set will be mocked out, which includes models stored in the
// testrig, and webfinger responses as well.
//
// If do is not nil, then the given do function will always be used, which allows callers
// to customize how the client is mocked.
//
// Note that you should never ever make ACTUAL http calls with this thing.
func NewMockHTTPClient(do func(req *http.Request) (*http.Response, error), relativeMediaPath string) *MockHTTPClient {
mockHTTPClient := &MockHTTPClient{}
if do != nil {
mockHTTPClient.do = do
return mockHTTPClient
}
mockHTTPClient.testRemoteStatuses = NewTestFediStatuses()
mockHTTPClient.testRemotePeople = NewTestFediPeople()
mockHTTPClient.testRemoteGroups = NewTestFediGroups()
mockHTTPClient.testRemoteServices = NewTestFediServices()
mockHTTPClient.testRemoteAttachments = NewTestFediAttachments(relativeMediaPath)
mockHTTPClient.do = func(req *http.Request) (*http.Response, error) {
responseCode := http.StatusNotFound
responseBytes := []byte(`{"error":"404 not found"}`)
responseContentType := applicationJSON
responseContentLength := len(responseBytes)
if req.Method == http.MethodPost {
b, err := io.ReadAll(req.Body)
if err != nil {
panic(err)
}
if sI, loaded := mockHTTPClient.SentMessages.LoadOrStore(req.URL.String(), [][]byte{b}); loaded {
s, ok := sI.([][]byte)
if !ok {
panic("SentMessages entry wasn't [][]byte")
}
s = append(s, b)
mockHTTPClient.SentMessages.Store(req.URL.String(), s)
}
responseCode = http.StatusOK
responseBytes = []byte(`{"ok":"accepted"}`)
responseContentType = applicationJSON
responseContentLength = len(responseBytes)
} else if strings.Contains(req.URL.String(), ".well-known/webfinger") {
responseCode, responseBytes, responseContentType, responseContentLength = WebfingerResponse(req)
} else if note, ok := mockHTTPClient.testRemoteStatuses[req.URL.String()]; ok {
// the request is for a note that we have stored
noteI, err := streams.Serialize(note)
if err != nil {
panic(err)
}
noteJSON, err := json.Marshal(noteI)
if err != nil {
panic(err)
}
responseCode = http.StatusOK
responseBytes = noteJSON
responseContentType = applicationActivityJSON
responseContentLength = len(noteJSON)
} else if person, ok := mockHTTPClient.testRemotePeople[req.URL.String()]; ok {
// the request is for a person that we have stored
personI, err := streams.Serialize(person)
if err != nil {
panic(err)
}
personJSON, err := json.Marshal(personI)
if err != nil {
panic(err)
}
responseCode = http.StatusOK
responseBytes = personJSON
responseContentType = applicationActivityJSON
responseContentLength = len(personJSON)
} else if group, ok := mockHTTPClient.testRemoteGroups[req.URL.String()]; ok {
// the request is for a person that we have stored
groupI, err := streams.Serialize(group)
if err != nil {
panic(err)
}
groupJSON, err := json.Marshal(groupI)
if err != nil {
panic(err)
}
responseCode = http.StatusOK
responseBytes = groupJSON
responseContentType = applicationActivityJSON
responseContentLength = len(groupJSON)
} else if service, ok := mockHTTPClient.testRemoteServices[req.URL.String()]; ok {
serviceI, err := streams.Serialize(service)
if err != nil {
panic(err)
}
serviceJSON, err := json.Marshal(serviceI)
if err != nil {
panic(err)
}
responseCode = http.StatusOK
responseBytes = serviceJSON
responseContentType = applicationActivityJSON
responseContentLength = len(serviceJSON)
} else if attachment, ok := mockHTTPClient.testRemoteAttachments[req.URL.String()]; ok {
responseCode = http.StatusOK
responseBytes = attachment.Data
responseContentType = attachment.ContentType
responseContentLength = len(attachment.Data)
}
[chore] use our own logging implementation (#716) * first commit Signed-off-by: kim <grufwub@gmail.com> * replace logging with our own log library Signed-off-by: kim <grufwub@gmail.com> * fix imports Signed-off-by: kim <grufwub@gmail.com> * fix log imports Signed-off-by: kim <grufwub@gmail.com> * add license text Signed-off-by: kim <grufwub@gmail.com> * fix package import cycle between config and log package Signed-off-by: kim <grufwub@gmail.com> * fix empty kv.Fields{} being passed to WithFields() Signed-off-by: kim <grufwub@gmail.com> * fix uses of log.WithFields() with whitespace issues and empty slices Signed-off-by: kim <grufwub@gmail.com> * *linter related grumbling* Signed-off-by: kim <grufwub@gmail.com> * gofmt the codebase! also fix more log.WithFields() formatting issues Signed-off-by: kim <grufwub@gmail.com> * update testrig code to match new changes Signed-off-by: kim <grufwub@gmail.com> * fix error wrapping in non fmt.Errorf function Signed-off-by: kim <grufwub@gmail.com> * add benchmarking of log.Caller() vs non-cached Signed-off-by: kim <grufwub@gmail.com> * fix syslog tests, add standard build tags to test runner to ensure consistency Signed-off-by: kim <grufwub@gmail.com> * make syslog tests more robust Signed-off-by: kim <grufwub@gmail.com> * fix caller depth arithmatic (is that how you spell it?) Signed-off-by: kim <grufwub@gmail.com> * update to use unkeyed fields in kv.Field{} instances Signed-off-by: kim <grufwub@gmail.com> * update go-kv library Signed-off-by: kim <grufwub@gmail.com> * update libraries list Signed-off-by: kim <grufwub@gmail.com> * fuck you linter get nerfed Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 08:47:55 +00:00
log.Debugf("returning response %s", string(responseBytes))
reader := bytes.NewReader(responseBytes)
readCloser := io.NopCloser(reader)
return &http.Response{
StatusCode: responseCode,
Body: readCloser,
ContentLength: int64(responseContentLength),
Header: http.Header{
"content-type": {responseContentType},
},
}, nil
}
return mockHTTPClient
}
func (m *MockHTTPClient) Do(req *http.Request) (*http.Response, error) {
return m.do(req)
}
func WebfingerResponse(req *http.Request) (responseCode int, responseBytes []byte, responseContentType string, responseContentLength int) {
var wfr *apimodel.WellKnownResponse
switch req.URL.String() {
case "https://unknown-instance.com/.well-known/webfinger?resource=acct:some_group@unknown-instance.com":
wfr = &apimodel.WellKnownResponse{
Subject: "acct:some_group@unknown-instance.com",
Links: []apimodel.Link{
{
Rel: "self",
Type: applicationActivityJSON,
Href: "https://unknown-instance.com/groups/some_group",
},
},
}
case "https://owncast.example.org/.well-known/webfinger?resource=acct:rgh@owncast.example.org":
wfr = &apimodel.WellKnownResponse{
Subject: "acct:rgh@example.org",
Links: []apimodel.Link{
{
Rel: "self",
Type: applicationActivityJSON,
Href: "https://owncast.example.org/federation/user/rgh",
},
},
}
case "https://unknown-instance.com/.well-known/webfinger?resource=acct:brand_new_person@unknown-instance.com":
wfr = &apimodel.WellKnownResponse{
Subject: "acct:brand_new_person@unknown-instance.com",
Links: []apimodel.Link{
{
Rel: "self",
Type: applicationActivityJSON,
Href: "https://unknown-instance.com/users/brand_new_person",
},
},
}
case "https://turnip.farm/.well-known/webfinger?resource=acct:turniplover6969@turnip.farm":
wfr = &apimodel.WellKnownResponse{
Subject: "acct:turniplover6969@turnip.farm",
Links: []apimodel.Link{
{
Rel: "self",
Type: applicationActivityJSON,
Href: "https://turnip.farm/users/turniplover6969",
},
},
}
case "https://fossbros-anonymous.io/.well-known/webfinger?resource=acct:foss_satan@fossbros-anonymous.io":
wfr = &apimodel.WellKnownResponse{
Subject: "acct:foss_satan@fossbros-anonymous.io",
Links: []apimodel.Link{
{
Rel: "self",
Type: applicationActivityJSON,
Href: "https://fossbros-anonymous.io/users/foss_satan",
},
},
}
case "https://example.org/.well-known/webfinger?resource=acct:some_user@example.org":
wfr = &apimodel.WellKnownResponse{
Subject: "acct:some_user@example.org",
Links: []apimodel.Link{
{
Rel: "self",
Type: applicationActivityJSON,
Href: "https://example.org/users/some_user",
},
},
}
}
if wfr == nil {
[chore] use our own logging implementation (#716) * first commit Signed-off-by: kim <grufwub@gmail.com> * replace logging with our own log library Signed-off-by: kim <grufwub@gmail.com> * fix imports Signed-off-by: kim <grufwub@gmail.com> * fix log imports Signed-off-by: kim <grufwub@gmail.com> * add license text Signed-off-by: kim <grufwub@gmail.com> * fix package import cycle between config and log package Signed-off-by: kim <grufwub@gmail.com> * fix empty kv.Fields{} being passed to WithFields() Signed-off-by: kim <grufwub@gmail.com> * fix uses of log.WithFields() with whitespace issues and empty slices Signed-off-by: kim <grufwub@gmail.com> * *linter related grumbling* Signed-off-by: kim <grufwub@gmail.com> * gofmt the codebase! also fix more log.WithFields() formatting issues Signed-off-by: kim <grufwub@gmail.com> * update testrig code to match new changes Signed-off-by: kim <grufwub@gmail.com> * fix error wrapping in non fmt.Errorf function Signed-off-by: kim <grufwub@gmail.com> * add benchmarking of log.Caller() vs non-cached Signed-off-by: kim <grufwub@gmail.com> * fix syslog tests, add standard build tags to test runner to ensure consistency Signed-off-by: kim <grufwub@gmail.com> * make syslog tests more robust Signed-off-by: kim <grufwub@gmail.com> * fix caller depth arithmatic (is that how you spell it?) Signed-off-by: kim <grufwub@gmail.com> * update to use unkeyed fields in kv.Field{} instances Signed-off-by: kim <grufwub@gmail.com> * update go-kv library Signed-off-by: kim <grufwub@gmail.com> * update libraries list Signed-off-by: kim <grufwub@gmail.com> * fuck you linter get nerfed Signed-off-by: kim <grufwub@gmail.com> Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 08:47:55 +00:00
log.Debugf("webfinger response not available for %s", req.URL)
responseCode = http.StatusNotFound
responseBytes = []byte(`{"error":"not found"}`)
responseContentType = applicationJSON
responseContentLength = len(responseBytes)
return
}
wfrJSON, err := json.Marshal(wfr)
if err != nil {
panic(err)
}
responseCode = http.StatusOK
responseBytes = wfrJSON
responseContentType = applicationJSON
responseContentLength = len(wfrJSON)
return
}