Commit graph

8234 commits

Author SHA1 Message Date
Mouse Reeve
ab1f08b39d
Merge pull request #2761 from bookwyrm-social/dependabot/pip/redis-4.5.3
Bump redis from 3.4.1 to 4.5.3
2023-03-27 18:58:40 -07:00
dependabot[bot]
e9526ecaea
Bump redis from 3.4.1 to 4.5.3
Bumps [redis](https://github.com/redis/redis-py) from 3.4.1 to 4.5.3.
- [Release notes](https://github.com/redis/redis-py/releases)
- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES)
- [Commits](https://github.com/redis/redis-py/compare/3.4.1...v4.5.3)

---
updated-dependencies:
- dependency-name: redis
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-27 21:57:11 +00:00
Mouse Reeve
b2801d44a5
Merge pull request #2752 from lostfictions/patch-1
Add message about shelf privacy in user settings
2023-03-27 06:21:33 -07:00
Mouse Reeve
30a3096b25
Merge pull request #2749 from bookwyrm-social/redirects
Add helper to refer views back to http referers safely
2023-03-27 06:20:26 -07:00
s
a70417a07f
Add message about shelf privacy in user settings 2023-03-20 22:10:50 -04:00
Mouse Reeve
e9d08e7424
Merge pull request #2744 from WesleyAC/opentelemetry-console-exporter
Add OpenTelemetry console exporter option
2023-03-20 14:26:24 -07:00
Mouse Reeve
89bb541bef
Merge pull request #2704 from tebriel/azure-support
Enable Azure BlobStorage as an alternative to AWS S3
2023-03-20 14:12:57 -07:00
Mouse Reeve
6f1e80be60
Merge pull request #2741 from WesleyAC/optimize-csv-export-query
Optimize CSV export query
2023-03-20 14:11:46 -07:00
Mouse Reeve
018cadf0cc
Merge pull request #2746 from WesleyAC/env-var-improvements
Env var improvements
2023-03-20 14:04:11 -07:00
Mouse Reeve
0cebc4388c Adds unit tests for referer helper 2023-03-20 12:47:21 -07:00
Mouse Reeve
219f78755f Fixes args in list view redirect 2023-03-20 11:15:27 -07:00
Mouse Reeve
4e9fad2091 Updates views that currently use the validator to use the redirect
helper
2023-03-20 10:27:52 -07:00
Mouse Reeve
2de115fc1e Add helper to refer views back to http referers safely
In most cases, we want to return back to where we came from after
performing an action. It's not safe to return to an arbitrary referer,
so this streamlines using the util validator to verify the redirect and
fall back on regular redirect params if the referer is outside our
domain.
2023-03-20 10:25:38 -07:00
Wesley Aptekar-Cassels
63ab3abe67 Add setting for OpenTelemetry console exporter
This is primarily useful for debugging changes to the OpenTelemetry
setup.
2023-03-18 15:47:30 -04:00
Wesley Aptekar-Cassels
fea3b67a55 Raise error if using default SECRET_KEY in production 2023-03-18 15:44:20 -04:00
Wesley Aptekar-Cassels
5bb5d120f0 Set default FLOWER_PORT 2023-03-18 15:35:41 -04:00
Wesley Aptekar-Cassels
b3d3de5374 Use typecasting env functions
Many of these environment variables were probably not actually usable,
since they would be strings if set in the env file. Using the
typecasting functions fixes this, and generally shows the intention of
the code more clearly.
2023-03-18 15:35:08 -04:00
Wesley Aptekar-Cassels
7bb4e21a36 Bump opentelemetry versions
This isn't actually required for anything, just good practice to keep
things up to date.
2023-03-16 13:57:19 -04:00
Wesley Aptekar-Cassels
60fee54da9 Optimize CSV export query
Splitting this into five separate queries avoids the large join that
prevents us from using indexes, and requires materializing to disk.

Fixes: #2157 (hopefully)
2023-03-13 15:45:21 -04:00
Mouse Reeve
ded3f469ef
Merge pull request #2738 from bookwyrm-social/update-version
Update version number and js cachebuster
2023-03-13 08:16:24 -07:00
Mouse Reeve
177131f53f
Merge pull request #2739 from bookwyrm-social/locales
Updates locales
2023-03-13 08:16:12 -07:00
Mouse Reeve
ae164ee6e1 Updates locales 2023-03-13 07:56:25 -07:00
Mouse Reeve
9c1aaadab3 Update verison number and js cachebuster 2023-03-13 07:52:28 -07:00
Mouse Reeve
a73a461867
Merge pull request #2737 from WesleyAC/fix-500-page-css
Use context processor for 500 page
2023-03-13 07:43:29 -07:00
Wesley Aptekar-Cassels
0b9e4d617e Use context processor for 500 page
By default, Django doesn't run any context processors for server errors,
to make the error path as simple as possible. However, this has the
downside that our template does not load correctly. To fix this, I added
a custom 500 error handler, which will run the context processor.

Fixes: #2736
2023-03-13 03:47:23 -04:00
Mouse Reeve
cca20f4834
Merge pull request #2735 from bookwyrm-social/migration
Adds merge migration
2023-03-12 17:34:17 -07:00
Mouse Reeve
7ffe5b9440 Adds merge migration 2023-03-12 16:43:06 -07:00
Mouse Reeve
12af5992a3
Merge pull request #2524 from chdorner/feature/tag-support
Initial hashtag support
2023-03-12 16:37:39 -07:00
Mouse Reeve
48889ee6c4
Merge pull request #2695 from chdorner/book-edit-form-validation-notification
Show notification banner on top of form when book failed to update
2023-03-12 16:33:34 -07:00
Mouse Reeve
2e7eb0f3ce
Merge pull request #2702 from Ryuno-Ki/lazyload-images
Add attributes to images to hint async load
2023-03-12 16:31:27 -07:00
Mouse Reeve
d253a61f02
Merge pull request #2708 from WesleyAC/portable-hashbangs
Use more portable hashbang for dev scripts.
2023-03-12 16:29:10 -07:00
Mouse Reeve
863ec1602a
Merge pull request #2710 from WesleyAC/celery-env-vars
Add env vars for celery concurrency and time limit
2023-03-12 16:27:02 -07:00
Mouse Reeve
6345beb90d
Merge pull request #2714 from WesleyAC/celery-ignore-results
Ignore Celery task results
2023-03-12 16:26:20 -07:00
Mouse Reeve
84b8a5c433
Merge pull request #2713 from WesleyAC/buffer-csv-export
Change CSV export to buffer instead of streaming
2023-03-12 16:17:53 -07:00
Mouse Reeve
d17190fae3
Merge pull request #2718 from WesleyAC/broaden-dashboard-http-except
Broaden except section for HTTP request in dashboard
2023-03-12 16:10:01 -07:00
Mouse Reeve
600340771a
Merge pull request #2723 from WesleyAC/get-audience-perf
Improve `HomeStream.get_audience` performance
2023-03-12 16:08:54 -07:00
Mouse Reeve
352ba972c5
Merge pull request #2724 from WesleyAC/fix-bw-dev-dbshell
Fix dbshell command
2023-03-12 15:45:45 -07:00
Mouse Reeve
c28d523e6f
Merge branch 'main' into get-audience-perf 2023-03-12 15:40:53 -07:00
Mouse Reeve
efe3cb9461
Merge pull request #2726 from WesleyAC/optimize-add-remove-book-statuses-task
Optimize add/remove book statuses task queries
2023-03-12 15:36:27 -07:00
Wesley Aptekar-Cassels
2a5f722f6e Optimize add/remove book statuses task queries
The queries as they previously existed required joining together 12
different tables, which is extremely expensive. Splitting it into four
queries means that the individual queries can effectively use the
indexes we have, and should be very fast no matter how many statuses are
in the database.

Removing the .distinct() call is fine, since we're adding them to a set
in Redis anyways, which will take care of the duplicates.

It's a bit ugly that we now make four separate calls to Redis (this
might result in things being slightly slower in cases where there are an
extremely small number of statuses), but doing things differently would
result in significantly more surgery to the existing code, so I've opted
to avoid that for the moment.

Fixes: #2725
2023-03-09 15:26:03 -05:00
Wesley Aptekar-Cassels
cc610372ca Fix dbshell command
dbshell needs to be run in a already-running container, thus exec rather
than run is the correct docker-compose command.
2023-03-09 02:02:56 -05:00
Wesley Aptekar-Cassels
56243f6529 Optimize HomeStream.get_audience
This splits HomeStream.get_audience into two separate database queries,
in order to more effectively take advantage of the indexes we have.
Combining the user ID query and the user following query means that
Postgres isn't able to use the index we have on the userfollows table.

The query planner claims that the userfollows query should be about 20
times faster than it was previously, and the id query should take a
negligible amount of time, since it's selecting a single item by primary
key.

We don't need to worry about duplicates, since there is a constraint
preventing a user from following themself.

Fixes: #2720
2023-03-09 00:50:24 -05:00
Wesley Aptekar-Cassels
23698dafe5 Change get_audience to return list of user IDs
This will make it simpler to implement various optimizations.
2023-03-09 00:50:24 -05:00
Wesley Aptekar-Cassels
41e14bdfaf Change unread_by_status_type_id to take user ID
Same reason as in prior commit.
2023-03-09 00:50:24 -05:00
Wesley Aptekar-Cassels
653e8ee81b Change unread_id to take user ID
Same reason as described in the prior commit.
2023-03-09 00:50:24 -05:00
Wesley Aptekar-Cassels
5446869c38 Change stream_id to take user ID
Anywhere we have a user object, we can easily get the user ID in the
caller, and this will allow us more flexibility in the future to
implement optimizations that involve knowing a user ID without querying
the database for the user object.
2023-03-09 00:50:16 -05:00
Mouse Reeve
e4edef03c5
Merge pull request #2721 from verymilan/verymilan-patch-1
fix typo in systemd example
2023-03-08 18:39:05 -08:00
Wesley Aptekar-Cassels
50a81bdfdd Change CSV export to buffer instead of streaming
The idea behind a streaming CSV export was to reduce the amount of
memory used, by avoiding building the entire CSV file in memory before
sending it to the client. However, it didn't work out this way in
practice: the query objects that were created to represent each line
caused Postgres to generate a very large (~200MB on bookwyrm.social)
temp file, not to mention the memory being used by the Query object
likely being similar to, if not larger than that used by the finalized
CSV row.

While we should in the long term run our CSV exports as a Celery task,
this change should allow CSV exports to work on large servers without
causing disk-space problems.

Fixes: #2157
2023-03-08 21:37:56 -05:00
Mouse Reeve
5c109a2566
Merge branch 'main' into celery-env-vars 2023-03-08 18:37:03 -08:00
Mouse Reeve
2f737efeff
Merge pull request #2709 from WesleyAC/improve-polling-backoff
Improve polling algorithm
2023-03-08 18:36:19 -08:00