The idea behind a streaming CSV export was to reduce the amount of
memory used, by avoiding building the entire CSV file in memory before
sending it to the client. However, it didn't work out this way in
practice: the query objects that were created to represent each line
caused Postgres to generate a very large (~200MB on bookwyrm.social)
temp file, not to mention the memory being used by the Query object
likely being similar to, if not larger than that used by the finalized
CSV row.
While we should in the long term run our CSV exports as a Celery task,
this change should allow CSV exports to work on large servers without
causing disk-space problems.
Fixes: #2157
get_data can return exceptions other than ConnectorException, and when
it does, we want to simply not show the update section, rather than
crashing.
Related: #2717
Since we don't use the results of our Celery tasks (all of them return
None implicitly), it's prudent to set the ignore_result flag, for a
potential performance improvement. See the Celery docs for details [1].
We could do this with the global CELERY_IGNORE_RESULT setting, but it
offers more flexibility if we want to use task results in the future to
set it on a per-task basis.
[1]: https://docs.celeryq.dev/en/stable/userguide/tasks.html#ignore-results-you-don-t-want
/bin/bash, while common, is not part of the unix standard, and does not
exist on some operating systems (such as NixOS). /usr/bin/env, on the
other hand, is standardized, and thus should exist on all systems.
The existing polling code had a few problems:
* It started the timer for a new request when the first request was
sent, rather than when a response was received.
* It increased the delay regardless of whether the response was a
success or a failure.
This commit changes it to a more standard exponential backoff system,
where it starts with a 5 minute ± 30 second delay, and uses that same
delay until it hits an error, at which point the delay is increased by
10%. Once it receives a successful response again, the delay is reset to
the default.
I suspect this should be nicer on the server, since it avoids the
initial sending of many requests. After about half an hour of leaving
the page open, the request rate for this new code will be higher than
that of the old code, so it's possible that this may cause problems, but
I think that a five-minute request frequency should be pretty reasonable.
We should store hashtags case-sensitive, but ensures that an existing
hashtag with different case are found and re-used. for example,
an existing #BookWyrm hashtag will be found and used even if the
status content is using #bookwyrm.
Since the status content already contains rendered HTML when we receive an
ActivityPub inbox message it contains links to the mentioned hashtags on the
originating instance.
To fix this on the receiving instance we need to post-process the status content
after successfully storing the status and its many-to-many fields (the one we're
is the `mention_hashtags`). Post-processing means that we run a regex against the
content to find the anchor tags linking to the originating hashtag and replace the
`href` attribute with the URL to the hashtag page on the receiving (local) instance.
This ensures that when an existing hashtag comes in through ActivityPub federation,
it correctly finds the local one, instead of creating duplicate hashtags.
This adds a new storage backend for Azure BlobStorage, I'm running this patch in Azure Kubernetes
hosting all my assets in Azure BlobStorage and it's been stable enough I thought I should commit
this back upstream.
This was suggested on Matrix a while ago but I only found the time now to move forward with it.
Signed-off-by: André Jaenisch <andre.jaenisch@posteo.de>
Expand TOTP validity window
This changes the default window to allow 2 codes (60 seconds) on either side. Admins can change this by setting a different `TWO_FACTOR_LOGIN_VALIDITY_WINDOW` value in `.env`
