Commit graph

1575 commits

Author SHA1 Message Date
kim 5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
kim 9eff0d46e4
[feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
dependabot[bot] 81fe59dadc
[chore]: Bump github.com/microcosm-cc/bluemonday from 1.0.24 to 1.0.25 (#2021) 2023-07-24 10:24:03 +00:00
dependabot[bot] b05f6c8f56
[chore]: Bump github.com/yuin/goldmark from 1.5.4 to 1.5.5 (#2023) 2023-07-24 10:21:14 +00:00
dependabot[bot] c9f3539b5a
[chore]: Bump codeberg.org/gruf/go-cache/v3 from 3.4.1 to 3.4.3 (#2022) 2023-07-24 10:20:11 +00:00
dependabot[bot] 8b71567f9e
[chore]: Bump github.com/jackc/pgconn from 1.14.0 to 1.14.1 (#2024) 2023-07-24 10:14:16 +00:00
dependabot[bot] b7891bb462
[chore]: Bump go.uber.org/automaxprocs from 1.5.2 to 1.5.3 (#2020) 2023-07-24 10:13:31 +00:00
Daenney 994d5e80d9
[chore] Disable the right webfinger test (#2018)
We don't talk about 8868889 / #1983.
2023-07-23 16:33:16 +02:00
tobi 5a29a031ad
[chore] Admin CLI + new account creation refactoring (#2008)
* set maxPasswordLength to 72 bytes, rename validate function

* refactor NewSignup

* refactor admin account CLI commands

* refactor oidc create user

* refactor processor create

* tweak password change, check old != new password
2023-07-23 12:33:17 +02:00
Daenney f8f0312042
[feature] Report Masto version in /api/v1/instance (#1977) 2023-07-21 18:49:13 +01:00
dependabot[bot] 43a2753ef0
[chore]: Bump modernc.org/sqlite from 1.23.1 to 1.24.0 (#1993)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.23.1 to 1.24.0.
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.23.1...v1.24.0)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-21 16:43:56 +02:00
Daenney da8282081c
[feature] Lower remote media cache config duration (#2007)
The old default of 30d can lead to a lot of media getting cached and
significant disk usage, even on small or single person instances. A lot
of deployments decrease this value, to 15 or even less. This is less of
an issue when using object storage, but for local storage which is the
more popular deployment option running out of disk space is unpleasant.

With GoToSocial's aim to fit in small places, this changes the default
to a much more conservative 7 days. In all likelihood people aren't
scrolling that far back in their timeline so this change shouldn't
result in any issue. Existing deployments will only be affected by
this change if the admin hasn't already configured this value, or didn't
bootstrap from the example configuration.
2023-07-21 16:22:59 +02:00
dependabot[bot] fa57c699fe
[chore]: Bump github.com/jackc/pgx/v5 from 5.4.1 to 5.4.2 (#1991) 2023-07-21 14:22:20 +00:00
dependabot[bot] 83139989b5
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.59 to 7.0.60 (#1992) 2023-07-21 14:22:17 +00:00
dependabot[bot] d53449ccb4
[chore]: Bump github.com/go-playground/form/v4 from 4.2.0 to 4.2.1 (#1994) 2023-07-21 14:22:13 +00:00
dependabot[bot] 8ac3a3f688
Bump word-wrap from 1.2.3 to 1.2.4 in /web/source (#2001) 2023-07-21 14:21:50 +00:00
tobi 89ee9d5004
[bugfix] Return all accounts when list accounts limit <= 0 (#2014) 2023-07-21 13:56:38 +01:00
tobi d6fa74e5dc
[bugfix] Change maximumPasswordLength to 72 bytes (#2012) 2023-07-21 10:29:18 +01:00
mirabilos 95e2024c2a
[docs] Apache setup for Caching assets and media (#2005) (#2005)
Also change the nginx fileserver expiry, after discussion, to
one week, to match.
2023-07-20 18:48:52 +02:00
kim f4319740ab
[bugfix] more robust list timeline invalidation (#1995) 2023-07-18 09:43:17 +01:00
tobi 346ecabd07
[bugfix] Add missing continue statement in prepareXBetweenIDs (#1996) 2023-07-18 09:39:16 +01:00
tobi 12b6cdcd8c
[bugfix] Set Vary header correctly on cache-control (#1988)
* [bugfix] Set Vary header correctly on cache-control

* Prefer activitypub types on AP endpoints

* use immutable on file server, vary by range

* vary auth on Accept
2023-07-13 21:27:25 +02:00
Daenney 88688899aa
[chore] Skip webfinger test on CI (#1983) 2023-07-12 16:09:49 +01:00
tobi 1951e6c840
[bugfix] Update account Update logic (#1984) 2023-07-12 12:20:15 +01:00
firescry 8d92b2479f
[bugfix] Align default values in the configuration file with the code (#1971)
* Set default value of SMTPFrom to empty string

This parameter should contain proper e-mail address (to be provided by user during configuration).

* Update default values in example/config.yaml

Default values and related comments in example/config.yaml are aligned
with values defined in internal/config/defaults.go.
Small improvements to foramting of config.yaml file.

* Add default value for AdvancedThrottlingRetryAfter to internal/config/defaults.go

AdvancedThrottlingRetryAfter was introduced in 70739d3 (superseriousbusiness/gotosocial#1466).

* Update config.yaml snippets in documentation
2023-07-12 10:28:41 +02:00
dependabot[bot] f5d6503aad
Bump semver from 6.3.0 to 6.3.1 in /web/source (#1981)
Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v6.3.0...v6.3.1)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-12 10:27:31 +02:00
MaeIsBad 0d267fd598
[bugfix] Properly handle range > content-length (#1979)
This makes the serveFileRange function return the entire file
if suffix-range is larger than content-length in compliance with RFC9110

Co-authored-by: mae <git@badat.dev>
2023-07-12 09:51:51 +02:00
tobi ca5492b65f
[bugfix] Tidy up rss feed serving; don't error on empty feed (#1970)
* [bugfix] Tidy up rss feed serving; don't error on empty feed

* fall back to account creation time as rss feed update time

* return feed early when account has no eligible statuses
2023-07-10 17:05:59 +02:00
dependabot[bot] a29b5affc8
[chore]: Bump golang.org/x/oauth2 from 0.9.0 to 0.10.0 (#1975)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 12:58:06 +01:00
Daenney f0dad439f6
[feature] Use Read/Write lock for config (#1969) 2023-07-10 12:56:14 +01:00
dependabot[bot] 6de5ca46f8
[chore]: Bump golang.org/x/net from 0.11.0 to 0.12.0 (#1973)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 12:43:32 +01:00
dependabot[bot] 901770f5c7
[chore]: Bump golang.org/x/image from 0.8.0 to 0.9.0 (#1972)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 12:31:09 +01:00
Daenney 65c89709bc
[chore] Fix password typos (#1966)
Fixes #1963
2023-07-09 18:25:37 +02:00
Daenney b0f0c8b822
[docs] Activate the edit feature (#1967)
We already set edit_uri, but we forgot to enable the feature flag. By
turning it on each page now gets an edit button on the right hand side
of the page header. This'll open the edit view in GitHub.
2023-07-09 18:25:14 +02:00
tobi 6dbb8ba7aa
[bugfix] Fix delete follow req instead of follow (#1962) 2023-07-08 23:12:06 +02:00
tobi f40bb02f31
[bugfix] Delete mutual follow (requests) when receiving block from remote (#1960)
* [bugfix] Delete mutual follow (requests) on block

* fix test
2023-07-08 16:43:12 +02:00
Daenney 747ea584bd
[docs] Clarify how to add a page (#1959)
* Add instructions on how to make a new page show up in the sidebar / navigation
* Explain how to create a virtual env to install the dependencies in if you're not using Conda
2023-07-08 15:44:58 +02:00
Daenney 672386a1b9
[bugfix] Don't overflow on very wide status'es (#1956)
If someone makes a post with a long, uninterrupted piece of text in a
code snippet, we would stretch the column to fit it, resulting in the
UI going a bit whacky.

By setting min-width: 0% this fixes it, and we now automatically get a
scrollbar on overflow instead.

Fixes: #1952
2023-07-08 09:54:26 +02:00
tobi 2a99df0588
[feature] enable + document explicit IP dialer allowing/denying (#1950)
* [feature] enable + document explicit IP dialer allowing/denying

* lord have mercy

* allee jonge

* shortcut check ipv6 prefixes

* comment

* separate httpclient_test, export Sanitizer
2023-07-07 16:17:39 +02:00
tobi ac564c1862
[bugfix] Reorder web view logic, other small fixes (#1954) 2023-07-07 14:58:53 +02:00
Daenney 9ff4c20374
[docs] Rework backups a bit (#1942)
* [docs] Rework backups a bit

This changes the existing backup documentation to:

* Push a bit harder on people to perform backups, it's not really just a
  nice to have
* Removes the language about migrating to/from GoToSocial and a
  different ActivityPub server since that's really not supported
* Adds a section about using backup software and provides an example on
  how to do this using Borgmatic

* [docs] Remove too much info in db section

* [docs] Add docs on how to backup local media

This adds documentation pointing people at the media list-local command
in order to determine what media files they need to include as part of
their backups.

Provides a Python script that people can use to transform the media
listing from the CLI into Borg patterns. It also includes a Borgmatic
config.yaml in the repository so people can easily fetch it instead of
copy-pasting from the docs.

* [bugfix] Ensure we emit an absolute path prefix

It works either way, as a pattern like data/files/<ID> would match a
file on /data/files/<ID>. But it would potentially also match any path
that happens to include data/files/<ID> but not rooted at the
storage-local-base-path.

* [docs] Add more links to media list CLI reference
2023-07-07 11:45:42 +02:00
Daenney 81f33c3b9f
[feature] Add media list command (#1943)
* [feature] Add media list command

This is an attempt to help alleviate #1776. Using admin media list
--local the full path to each local media file will be printed, with a
newline. The output of this should be feadable into backup tools in
order to allow to backup local media too. Together with the database
this should allow to fully recover from the loss of an instance.

The list command also gets a --remote flag for symmetry. In the case
of --remote we print the RemoteURL instead, the location the asset can
be retrieved from.

To get all media, you can run with --local and --remote.

* [bugfix] Fix the test failures

* [feature] Reimplement list media as top commands

This changes the implementation of admin media list --<variant> to two
separate top-level commands, list-local and list-remote.

The implementation now iterates over over the database in batches of 200
in order to avoid loading all media metadata into memory.

* [feature] Implement ListMedia with filter callback

This does away with the somewhat odd iterator-like structure we had
before and does away with most of the loop duplication in list-local and
list-remote. Instead they call GetAllMediaPaths with a filter func to
select the media they want. That's accumulated into a slice and
eventually returned.

* [bugfix] Simplify remote filter

Since we don't append the empty string anywhere, the remote filter can
be limited to returning RemoteURL, as that'll be an empty string for
local media.

* [docs] Add media list commands to CLI reference

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2023-07-07 11:35:05 +02:00
tobi e70bf8a6c8
[chore/bugfix] Domain block tidying up, Implement first pass of 207 Multi-Status (#1886)
* [chore/refactor] update domain block processing

* expose domain block import errors a lil better

* move/remove unused query keys
2023-07-07 11:34:12 +02:00
tobi d9c69f6ce0
[chore/performance] Remove remaining 'whereEmptyOrNull' funcs (#1946) 2023-07-05 12:34:37 +02:00
tobi 3d16962173
[chore/bugfix] Break Websockets logic into smaller read/write functions, don't log expected errors (#1932)
* [chore/bugfix] Break Websockets logic into smaller read/write functions, don't log expected errors

* tweak

* tidy up, use control message
2023-07-04 12:55:10 +02:00
Daenney ba0bc06b8c
[feature] Add instance stats to /about (#1936)
When you configure the landing-page-user redirect, you lose access to
the one page that displays server stats. This adds the same stats as we
have on / to /about to bring those back.
2023-07-04 11:33:07 +02:00
dependabot[bot] 1218f97e01
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.58 to 7.0.59 (#1941)
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.58 to 7.0.59.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.58...v7.0.59)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 11:22:40 +02:00
Daenney 2a40c81f10
[bugfix] Try to fix the webfinger test, again (#1931)
For some reason we hit the case in CI where the
TestFingerWithHostMetaCacheStrategy seems to experience some time
dilation. It's possible this is a genuine bug, but I can't for the life
of me reproduce it locally, even after having run this test thousands of
times (-count=1000 when invoking go test etc.)

This changes the test to explicitly stop the webfinger cache, set TTL
and Sweep frequency to something well beyond the lifetime of the cache
during the test and then starts the cache again. Hopefully that does it,
because the other option that remains is that for some reason
timekeeping in CI/Docker is not as precise as when running the test on a
host.
2023-06-27 18:32:03 +02:00
tobi d98b6318ac
[bugfix] Use gtserror package for WrongType errs (#1930)
* [bugfix] Use gtserror package for WrongType errs

* test
2023-06-27 11:37:42 +02:00
dependabot[bot] e3e0f673cc
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.56 to 7.0.58 (#1928) 2023-06-26 20:45:49 +00:00