Commit graph

357 commits

Author SHA1 Message Date
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
tobi fd8a724e77
[chore] Bump go swagger (#2871)
* bump go swagger version

* bump swagger version
2024-04-26 11:31:10 +02:00
dependabot[bot] 62788aa116
[chore]: Bump codeberg.org/gruf/go-mutexes from 1.4.0 to 1.4.1 (#2860)
Bumps codeberg.org/gruf/go-mutexes from 1.4.0 to 1.4.1.

---
updated-dependencies:
- dependency-name: codeberg.org/gruf/go-mutexes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-22 13:43:45 +01:00
dependabot[bot] a57dd15a8e
[chore]: Bump github.com/miekg/dns from 1.1.58 to 1.1.59 (#2861) 2024-04-22 09:11:53 +00:00
kim 12a7eba01f
[chore] bump modernc.org/sqlite to v1.29.8 (with our concurrency workaround) (#2855) 2024-04-22 11:02:11 +02:00
dependabot[bot] 0db9e34b69
[chore]: Bump github.com/KimMachineGun/automemlimit from 0.5.0 to 0.6.0 (#2859) 2024-04-22 08:59:30 +00:00
kim b3f2d44143
bump to modernc.org/sqlite v1.29.7 (#2850) 2024-04-17 17:10:51 +01:00
kim 1018cde107
[chore] bump bun library versions (#2837) 2024-04-15 12:01:20 +02:00
dependabot[bot] 6bb43f3f9b
[chore]: Bump golang.org/x/net from 0.23.0 to 0.24.0 (#2834) 2024-04-15 08:44:29 +00:00
dependabot[bot] 66e4510bf1
[chore]: Bump golang.org/x/crypto from 0.21.0 to 0.22.0 (#2835) 2024-04-15 08:34:49 +00:00
dependabot[bot] c097745c38
[chore]: Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc (#2818)
Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) from 1.24.0 to 1.25.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.24.0...v1.25.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-11 11:46:18 +02:00
kim db2dcc3455
[chore] update go-structr => v0.6.2 (fixes nested field ptr following) (#2822)
* update go-structr => v0.6.1 (fixes nested field ptr following)

* bump to v0.6.2
2024-04-11 11:46:08 +02:00
kim a483bd9e38
[performance] massively improved ActivityPub delivery worker efficiency (#2812)
* add delivery worker type that pulls from queue to httpclient package

* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller

* hook up queue deletion logic

* support deleting queued http requests by target ID

* don't index APRequest by hostname in the queue

* use gorun

* use the original context's values when wrapping msg type as delivery{}

* actually log in the AP delivery worker ...

* add uncommitted changes

* use errors.AsV2()

* use errorsv2.AsV2()

* finish adding some code comments, add bad host handling to delivery workers

* slightly tweak deliveryworkerpool API, use advanced sender multiplier

* remove PopCtx() method, let others instead rely on Wait()

* shuffle things around to move delivery stuff into transport/ subpkg

* remove dead code

* formatting

* validate request before queueing for delivery

* finish adding code comments, fix up backoff code

* finish adding more code comments

* clamp minimum no. senders to 1

* add start/stop logging to delivery worker, some slight changes

* remove double logging

* use worker ptrs

* expose the embedded log fields in httpclient.Request{}

* ensure request context values are preserved when updating ctx

* add delivery worker tests

* fix linter issues

* ensure delivery worker gets inited in testrig

* fix tests to delivering messages to check worker delivery queue

* update error type to use ptr instead of value receiver

* fix test calling Workers{}.Start() instead of testrig.StartWorkers()

* update docs for advanced-sender-multiplier

* update to the latest activity library version

* add comment about not using httptest.Server{}
2024-04-11 11:45:35 +02:00
dependabot[bot] 15733cddb2
[chore]: Bump github.com/yuin/goldmark from 1.7.0 to 1.7.1 (#2819)
Bumps [github.com/yuin/goldmark](https://github.com/yuin/goldmark) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/yuin/goldmark/releases)
- [Commits](https://github.com/yuin/goldmark/compare/v1.7.0...v1.7.1)

---
updated-dependencies:
- dependency-name: github.com/yuin/goldmark
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-10 12:04:27 +01:00
dependabot[bot] 4d16c3adaf
[chore]: Bump golang.org/x/oauth2 from 0.18.0 to 0.19.0 (#2816)
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.18.0 to 0.19.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.18.0...v0.19.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-10 12:04:11 +01:00
kim 85bc140b58
[bugfix] temporarily replace modernc.org/sqlite-v1.29.5 with gitlab.com/NyaaaWhatsUpDoc/sqlite-v1.29.5-concurrency-workaround (#2811) 2024-04-05 11:58:28 +02:00
kim adf345f1ec
[chore] bump go structr cache version -> v0.6.0 (#2773)
* update go-structr library -> v0.6.0, add necessary wrapping types + code changes to support these changes

* update readme with go-structr package changes

* improved wrapping of the SliceCache type

* add code comments for the cache wrapper types

* remove test.out 😇

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2024-04-02 12:03:40 +02:00
dependabot[bot] 29031d1e27
[chore]: Bump github.com/gin-contrib/sessions from 0.0.5 to 1.0.0 (#2782) 2024-03-25 11:00:36 +00:00
dependabot[bot] a24936040c
[chore]: Bump github.com/coreos/go-oidc/v3 from 3.9.0 to 3.10.0 (#2779) 2024-03-25 10:58:34 +00:00
dependabot[bot] 5f43419a87
[chore]: Bump github.com/gin-contrib/cors from 1.7.0 to 1.7.1 (#2778) 2024-03-25 10:43:08 +00:00
dependabot[bot] 40ee470536
[chore]: Bump github.com/gin-contrib/gzip from 0.0.6 to 1.0.0 (#2781) 2024-03-25 10:31:19 +00:00
dependabot[bot] 967558947a
[chore]: Bump modernc.org/sqlite from 1.29.4 to 1.29.5 (#2768)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.29.4 to 1.29.5.
- [Commits](https://gitlab.com/cznic/sqlite/commits/v1.29.5)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-21 11:14:24 +00:00
dependabot[bot] 51e8879779
[chore]: Bump github.com/tdewolff/minify/v2 from 2.20.18 to 2.20.19 (#2767) 2024-03-18 10:05:26 +00:00
dependabot[bot] 8570793dfb
[chore]: Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 (#2766) 2024-03-18 09:56:42 +00:00
dependabot[bot] fae7627a1b
[chore]: Bump github.com/technologize/otel-go-contrib (#2765) 2024-03-18 09:54:32 +00:00
kim c27049ad15
[chore] bump modernc.org/sqlite v1.28.0 -> v1.29.4 (#2749) 2024-03-12 18:21:59 +01:00
Daenney 5e871e81a8
[chore] Update usage of OTEL libraries (#2725)
* otel to 1.24
* prometheus exporter to 0.46
* bunotel to 1.1.17

Also:
* Use schemaless URL for metrics
* Add software version to tracing schema
2024-03-11 15:34:34 +01:00
dependabot[bot] 8e88ee8d9c
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.67 to 7.0.69 (#2748) 2024-03-11 10:51:13 +00:00
dependabot[bot] 5a56f4f8fb
[chore]: Bump github.com/go-jose/go-jose/v3 from 3.0.1 to 3.0.3 (#2733) 2024-03-11 10:36:13 +00:00
dependabot[bot] d115f9ebc4
[chore]: Bump github.com/jackc/pgx/v5 from 5.5.3 to 5.5.5 (#2747) 2024-03-11 10:13:33 +00:00
dependabot[bot] e24efcac8b
[chore]: Bump github.com/gin-contrib/cors from 1.5.0 to 1.7.0 (#2745) 2024-03-11 10:12:06 +00:00
tobi ebdee5aed8
[chore] Downgrade sqlite v1.29.2 -> v1.28.0 (#2736)
* [chore] Downgrade sqlite v1.29.2 -> v1.29.0

* go down to v1.28.0
2024-03-08 11:45:15 +01:00
Vyr Cossont fc3741365c
[bugfix] Fix Swagger spec and add test script (#2698)
* Add Swagger spec test script

* Fix Swagger spec errors not related to statuses with polls

* Add API tests that post a status with a poll

* Fix creating a status with a poll from form params

* Fix Swagger spec errors related to statuses with polls (this is the last error)

* Fix Swagger spec warnings not related to unused definitions

* Suppress a duplicate list update params definition that was somehow causing wrong param names

* Add Swagger test to CI

- updates Drone config
- vendorizes go-swagger
- fixes a file extension issue that caused the test script to generate JSON instead of YAML with the vendorized version

* Put `Sample: ` on its own line everywhere

* Remove unused id param from emojiCategoriesGet

* Add 5 more pairs of profile fields to account update API Swagger

* Remove Swagger prefix from dummy fields

It makes the generated code look weird

* Manually annotate params for statusCreate operation

* Fix all remaining Swagger spec warnings

- Change some models into operation parameters
- Ignore models that already correspond to manually documented operation parameters but can't be trivially changed (those with file fields)

* Documented that creating a status with scheduled_at isn't implemented yet

* sign drone.yml

* Fix filter API Swagger errors

* fixup! Fix filter API Swagger errors

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-03-06 18:05:45 +01:00
dependabot[bot] 7bc536d1f7
[chore]: Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 (#2714)
* [chore]: Bump github.com/stretchr/testify from 1.8.4 to 1.9.0

Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.4 to 1.9.0.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.4...v1.9.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* cast the exxpected test values as float32 to bypass (possible) testify suite regression

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kim <grufwub@gmail.com>
2024-03-04 15:05:24 +00:00
dependabot[bot] c7845c70d6
[chore]: Bump golang.org/x/crypto from 0.19.0 to 0.20.0 (#2716) 2024-03-04 09:42:28 +00:00
dependabot[bot] adb4cdcf6c
[chore]: Bump modernc.org/sqlite from 1.28.0 to 1.29.2 (#2718) 2024-03-04 09:42:11 +00:00
dependabot[bot] 87e8cca3ae
[chore]: Bump github.com/tdewolff/minify/v2 from 2.20.17 to 2.20.18 (#2689)
Bumps [github.com/tdewolff/minify/v2](https://github.com/tdewolff/minify) from 2.20.17 to 2.20.18.
- [Release notes](https://github.com/tdewolff/minify/releases)
- [Commits](https://github.com/tdewolff/minify/compare/v2.20.17...v2.20.18)

---
updated-dependencies:
- dependency-name: github.com/tdewolff/minify/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-26 10:32:02 +00:00
kim 1d51e3c8d6
[bugfix] 2643 bug search for account url doesnt always work when redirected (#2673)
* update activity library so dereferencer returns full response and checks *final* link to allow for redirects

* temporarily add bodged fixed library

* remove unused code

* update getAccountFeatured() to use dereferenceCollectionPage()

* make sure to release map

* perform a 2nd decode to ensure reader is empty after primary decode

* add comment explaining choice of using Decode() instead of Unmarshal()

* update embedded activity library to latest matching https://github.com/superseriousbusiness/activity/pull/21

* add checks to look for changed URI and re-check database if redirected

* update max iteration count to 512, add checks during dereferenceAncestors() for indirect URLs

* remove doubled-up code

* fix use of status instead of current

* use URIs for checking equality for security

* use the latest known URI for boost_of_uri in case original was an indirect

* add dereferenceCollection() function for dereferenceAccountFeatured()

* pull in latest github.com/superseriousbusiness/activity version (and remove the bodge!!)

* fix typo in code comments

* update decodeType() to accept a readcloser and handle body closing

* switch to checking using BoostOfID and add note why not using BoostOfURI

* ensure InReplyTo gets unset when deleting status parent in case currently stubbed

* add tests for Collection and CollectionPage iterators
2024-02-23 16:24:40 +01:00
dependabot[bot] e2ebcbb516
[chore]: Bump github.com/jackc/pgx/v5 from 5.5.2 to 5.5.3 (#2664)
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) from 5.5.2 to 5.5.3.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v5.5.2...v5.5.3)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-19 11:20:15 +01:00
dependabot[bot] 2db115fa36
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.66 to 7.0.67 (#2662)
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.66 to 7.0.67.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.66...v7.0.67)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-19 09:36:50 +00:00
dependabot[bot] 30f6e6c7fe
[chore]: Bump github.com/tdewolff/minify/v2 from 2.20.16 to 2.20.17 (#2661) 2024-02-19 07:23:08 +00:00
dependabot[bot] db835f4810
[chore]: Bump github.com/tdewolff/minify/v2 from 2.20.14 to 2.20.16 (#2631)
Bumps [github.com/tdewolff/minify/v2](https://github.com/tdewolff/minify) from 2.20.14 to 2.20.16.
- [Release notes](https://github.com/tdewolff/minify/releases)
- [Commits](https://github.com/tdewolff/minify/compare/v2.20.14...v2.20.16)

---
updated-dependencies:
- dependency-name: github.com/tdewolff/minify/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-12 10:59:43 +00:00
dependabot[bot] db45e65e89
[chore]: Bump golang.org/x/oauth2 from 0.16.0 to 0.17.0 (#2629) 2024-02-12 10:04:32 +00:00
dependabot[bot] d0e674b3da
[chore]: Bump golang.org/x/crypto from 0.18.0 to 0.19.0 (#2632) 2024-02-12 09:56:45 +00:00
dependabot[bot] 3cc51d5072
[chore]: Bump github.com/miekg/dns from 1.1.57 to 1.1.58 (#2606) 2024-02-05 14:09:50 +00:00
dependabot[bot] d59cba487f
[chore]: Bump github.com/google/uuid from 1.5.0 to 1.6.0 (#2604) 2024-02-05 09:52:30 +00:00
dependabot[bot] a482cb59f2
[chore]: Bump github.com/yuin/goldmark from 1.6.0 to 1.7.0 (#2603) 2024-02-05 09:50:35 +00:00
tobi 8692cc1f36
[chore] bump activity dependency -> v1.6.0-gts (#2599) 2024-02-02 14:06:09 +01:00
tobi b614d33c40
[feature] Try HTTP signature validation with and without query params for incoming requests (#2591)
* [feature] Verify signatures both with + without query params

* Bump to tagged version
2024-01-31 14:15:28 +00:00
kim 81198fa2d0
update go-structr v0.2.0 => v0.3.0 to fix possible hash collision issues (#2586) 2024-01-29 15:13:53 +00:00