Commit graph

11 commits

Author SHA1 Message Date
Daenney
70398891b8
[chore] Move request validation earlier in client (#1531)
This moves checking if the request is valid as early as possible in the
chain. This should ensure that for an invalid request we never bother
acquiring the wait queue and taking up a spot in it.
2023-02-19 11:01:15 +00:00
Daenney
68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi
0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
Daniele Sluijters
746f3fa4e6
Additional IP range validations (#1152)
* [bugfix] Ensure requests happen over TCP

It's possible for the network to be udp4 or udp6. This is rather
unlikely to occur, but since we're given the network anyway as part of
the Sanitize function getting called we might as well check for it.

* [chore] Align reserved v6 blocks to IANA registry

* [chore] Add test for ValidateIP

The net and netip packages diverge in that net.ParseIP will consider an
IPv4-mapped address to be an IPv4 address and as such it would get
caught by the IPv4Reserved list. However, netip considers it an IPv6
address, so we need to ensure the mapped range is in IPv6Reserved.

* [chore] Align reserved v4 blocks to IANA registry

This includes a number of tests for /32's explicitly called out in the
registry to ensure we always consider those invalid.
2022-11-26 11:09:55 +00:00
tobi
c9d893fec1
[feature/performance] Fail fast when doing remote transport calls inside incoming request contexts (#1119)
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts

* [chore] Reduce outgoing request timeout to 15s

* log error messages when fastfailing

* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry

* add fast-fail context key test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-23 21:40:07 +00:00
kim
0e57246083
[feature] various worker / request queue improvements (#995)
* greatly simplify httpclient request queuing

Signed-off-by: kim <grufwub@gmail.com>

* improved request queue mutex logic

Signed-off-by: kim <grufwub@gmail.com>

* use improved hashmap library

Signed-off-by: kim <grufwub@gmail.com>

* add warn logging when request queues are full

Signed-off-by: kim <grufwub@gmail.com>

* improve worker pool prefix var naming

Signed-off-by: kim <grufwub@gmail.com>

* improved worker pool error logging

Signed-off-by: kim <grufwub@gmail.com>

* move error message into separate field

Signed-off-by: kim <grufwub@gmail.com>

* remove old log statement

Signed-off-by: kim <grufwub@gmail.com>

* don't export worker message, it gets very spammy :')

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-08 10:35:24 +01:00
kim
a156188b3e
[chore] update dependencies, bump to Go 1.19.1 (#826)
* update dependencies, bump Go version to 1.19

* bump test image Go version

* update golangci-lint

* update gotosocial-drone-build

* sign

* linting, go fmt

* update swagger docs

* update swagger docs

* whitespace

* update contributing.md

* fuckin whoopsie doopsie

* linterino, linteroni

* fix followrequest test not starting processor

* fix other api/client tests not starting processor

* fix remaining tests where processor not started

* bump go-runners version

* don't check last-webfingered-at, processor may have updated this

* update swagger command

* update bun to latest version

* fix embed to work the same as before with new bun

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-09-28 18:30:40 +01:00
tobi
969c194fcd
[bugfix] Relax outgoing http request queue (#760)
* add request queue trace logging

* fix  misleading wording

* implement request slots per host per method

* undo formatting change (?)

* remove gratuitous trace logging

* rename done -> release
avoids confusion with ctx.Done
2022-08-27 12:00:19 +02:00
tobi
ab03318b7a
[chore] move dialer inside new (#715)
* move dialer inside new, use default resolver

* instantiate resolver
2022-07-18 11:25:26 +02:00
tobi
1cdc163276
[performance] Don't retry/backoff invalid http requests that will never succeed (#609)
* add httpguts (ew)

* add ValidateRequest err wrapping logic

* don't retry on unrecoverable errors

* i am very clever
2022-05-26 13:38:41 +02:00
kim
223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00