This is an enjoyable book with great characters.
", + "to": ["https://example.net/user/library_lurker/followers"], + "cc": [], + "replies": { + "id": "https://example.net/user/library_lurker/review/2/replies", + "type": "OrderedCollection", + "totalItems": 0, + "first": "https://example.net/user/library_lurker/review/2/replies?page=1", + "last": "https://example.net/user/library_lurker/review/2/replies?page=1", + "@context": "https://www.w3.org/ns/activitystreams" + }, + "summary": "Spoilers ahead!", + "tag": [], + "attachment": [], + "sensitive": true, + "inReplyToBook": "https://example.net/book/1", + "name": "What a cracking read", + "rating": 4.5, + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +##### Comment + +A `Comment` on a book mentions a book and has a message body, reading status, and progress indicator. + +Example: + +```json +{ + "id": "https://example.net/user/library_lurker/comment/9", + "type": "Comment", + "published": "2023-06-30T21:43:46.013132+00:00", + "attributedTo": "https://example.net/user/library_lurker", + "content": "This is a very enjoyable book so far.
", + "to": ["https://example.net/user/library_lurker/followers"], + "cc": [], + "replies": { + "id": "https://example.net/user/library_lurker/comment/9/replies", + "type": "OrderedCollection", + "totalItems": 0, + "first": "https://example.net/user/library_lurker/comment/9/replies?page=1", + "last": "https://example.net/user/library_lurker/comment/9/replies?page=1", + "@context": "https://www.w3.org/ns/activitystreams" + }, + "summary": "Spoilers ahead!", + "tag": [], + "attachment": [], + "sensitive": true, + "inReplyToBook": "https://example.net/book/1", + "readingStatus": "reading", + "progress": 25, + "progressMode": "PG", + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +##### Quotation + +A quotation (aka "quote") has a message body, an excerpt from a book including position as a page number or percentage indicator, and mentions a book. + +Example: + +```json +{ + "id": "https://example.net/user/mouse/quotation/13", + "url": "https://example.net/user/mouse/quotation/13", + "inReplyTo": null, + "published": "2020-05-10T02:38:31.150343+00:00", + "attributedTo": "https://example.net/user/mouse", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "cc": [ + "https://example.net/user/mouse/followers" + ], + "sensitive": false, + "content": "I really like this quote", + "type": "Quotation", + "replies": { + "id": "https://example.net/user/mouse/quotation/13/replies", + "type": "Collection", + "first": { + "type": "CollectionPage", + "next": "https://example.net/user/mouse/quotation/13/replies?only_other_accounts=true&page=true", + "partOf": "https://example.net/user/mouse/quotation/13/replies", + "items": [] + } + }, + "inReplyToBook": "https://example.net/book/1", + "quote": "To be or not to be, that is the question.", + "position": 50, + "positionMode": "PCT", + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +### Custom Objects + +##### Work +A particular book, a "work" in the [FRBR](https://en.wikipedia.org/wiki/Functional_Requirements_for_Bibliographic_Records) sense. + +Example: + +```json +{ + "id": "https://bookwyrm.social/book/5988", + "type": "Work", + "authors": [ + "https://bookwyrm.social/author/417" + ], + "first_published_date": null, + "published_date": null, + "title": "Piranesi", + "sort_title": null, + "subtitle": null, + "description": "**From the *New York Times* bestselling author of *Jonathan Strange & Mr. Norrell*, an intoxicating, hypnotic new novel set in a dreamlike alternative reality.", + "languages": [], + "series": null, + "series_number": null, + "subjects": [ + "English literature" + ], + "subject_places": [], + "openlibrary_key": "OL20893680W", + "librarything_key": null, + "goodreads_key": null, + "attachment": [ + { + "url": "https://bookwyrm.social/images/covers/10226290-M.jpg", + "type": "Image" + } + ], + "lccn": null, + "editions": [ + "https://bookwyrm.social/book/5989" + ], + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +##### Edition +A particular _manifestation_ of a Work, in the [FRBR](https://en.wikipedia.org/wiki/Functional_Requirements_for_Bibliographic_Records) sense. + +Example: + +```json +{ + "id": "https://bookwyrm.social/book/5989", + "lastEditedBy": "https://example.net/users/rat", + "type": "Edition", + "authors": [ + "https://bookwyrm.social/author/417" + ], + "first_published_date": null, + "published_date": "2020-09-15T00:00:00+00:00", + "title": "Piranesi", + "sort_title": null, + "subtitle": null, + "description": "Piranesi's house is no ordinary building; its rooms are infinite, its corridors endless, its walls are lined with thousands upon thousands of statues, each one different from all the others.", + "languages": [ + "English" + ], + "series": null, + "series_number": null, + "subjects": [], + "subject_places": [], + "openlibrary_key": "OL29486417M", + "librarything_key": null, + "goodreads_key": null, + "isfdb": null, + "attachment": [ + { + "url": "https://bookwyrm.social/images/covers/50202953._SX318_.jpg", + "type": "Image" + } + ], + "isbn_10": "1526622424", + "isbn_13": "9781526622426", + "oclc_number": null, + "asin": null, + "pages": 272, + "physical_format": null, + "publishers": [ + "Bloomsbury Publishing Plc" + ], + "work": "https://bookwyrm.social/book/5988", + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +#### Shelf + +A user's book collection. By default, every user has a `to-read`, `reading`, `read`, and `stopped-reading` shelf which are used to track reading progress. Users may create an unlimited number of additional shelves with their own ids. + +Example + +```json +{ + "id": "https://example.net/user/avid_reader/books/extraspecialbooks-5", + "type": "Shelf", + "totalItems": 0, + "first": "https://example.net/user/avid_reader/books/extraspecialbooks-5?page=1", + "last": "https://example.net/user/avid_reader/books/extraspecialbooks-5?page=1", + "name": "Extra special books", + "owner": "https://example.net/user/avid_reader", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "cc": [ + "https://example.net/user/avid_reader/followers" + ], + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +#### List + +A collection of books that may have items contributed by users other than the one who created the list. + +Example: + +```json +{ + "id": "https://example.net/list/1", + "type": "BookList", + "totalItems": 0, + "first": "https://example.net/list/1?page=1", + "last": "https://example.net/list/1?page=1", + "name": "My cool list", + "owner": "https://example.net/user/avid_reader", + "to": [ + "https://www.w3.org/ns/activitystreams#Public" + ], + "cc": [ + "https://example.net/user/avid_reader/followers" + ], + "summary": "A list of books I like.", + "curation": "curated", + "@context": "https://www.w3.org/ns/activitystreams" +} +``` + +#### Activities + +- `Create`: Adds a shelf or list to the database. +- `Delete`: Removes a shelf or list. +- `Add`: Adds a book to a shelf or list. +- `Remove`: Removes a book from a shelf or list. + +## Alternative Serialization +Because BookWyrm uses custom object types that aren't listed in [the standard ActivityStreams Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary), some statuses are transformed into standard types when sent to or viewed by non-BookWyrm services. `Review`s are converted into `Article`s, and `Comment`s and `Quotation`s are converted into `Note`s, with a link to the book and the cover image attached. + +In future this may be done with [JSON-LD type arrays](https://www.w3.org/TR/json-ld/#specifying-the-type) instead. + +## Other extensions + +### Webfinger + +Bookwyrm uses the [Webfinger](https://datatracker.ietf.org/doc/html/rfc7033) standard to identify and disambiguate fediverse actors. The [Webfinger documentation on the Mastodon project](https://docs.joinmastodon.org/spec/webfinger/) provides a good overview of how Webfinger is used. + +### HTTP Signatures + +Bookwyrm uses and requires HTTP signatures for all `POST` requests. `GET` requests are not signed by default, but if Bookwyrm receives a `403` response to a `GET` it will re-send the request, signed by the default server user. This usually will have a user id of `https://example.net/user/bookwyrm.instance.actor` + +#### publicKey id + +In older versions of Bookwyrm the `publicKey.id` was incorrectly listed in request headers as `https://example.net/user/username#main-key`. As of v0.6.3 the id is now listed correctly, as `https://example.net/user/username/#main-key`. In most ActivityPub implementations this will make no difference as the URL will usually resolve to the same place. + +### NodeInfo + +Bookwyrm uses the [NodeInfo](http://nodeinfo.diaspora.software/) standard to provide statistics and version information for each instance. + +## Further Documentation + +See [docs.joinbookwyrm.com/](https://docs.joinbookwyrm.com/) for more documentation. \ No newline at end of file diff --git a/README.md b/README.md index 558d42d45..f8b2eb1f6 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ Keep track of what books you've read, and what books you'd like to read in the f Federation allows you to interact with users on other instances and services, and also shares metadata about books and authors, which collaboratively builds a decentralized database of books. ### Privacy and moderation -Users and administrators can control who can see thier posts and what other instances to federate with. +Users and administrators can control who can see their posts and what other instances to federate with. ## Tech Stack Web backend diff --git a/bookwyrm/activitypub/__init__.py b/bookwyrm/activitypub/__init__.py index bfb22fa32..2697620f0 100644 --- a/bookwyrm/activitypub/__init__.py +++ b/bookwyrm/activitypub/__init__.py @@ -3,8 +3,12 @@ import inspect import sys from .base_activity import ActivityEncoder, Signature, naive_parse -from .base_activity import Link, Mention -from .base_activity import ActivitySerializerError, resolve_remote_id +from .base_activity import Link, Mention, Hashtag +from .base_activity import ( + ActivitySerializerError, + resolve_remote_id, + get_representative, +) from .image import Document, Image from .note import Note, GeneratedNote, Article, Comment, Quotation from .note import Review, Rating diff --git a/bookwyrm/activitypub/base_activity.py b/bookwyrm/activitypub/base_activity.py index e942c9aeb..05e7d8a05 100644 --- a/bookwyrm/activitypub/base_activity.py +++ b/bookwyrm/activitypub/base_activity.py @@ -1,16 +1,27 @@ """ basics for an activitypub serializer """ +from __future__ import annotations from dataclasses import dataclass, fields, MISSING from json import JSONEncoder import logging +from typing import Optional, Union, TypeVar, overload, Any + +import requests from django.apps import apps from django.db import IntegrityError, transaction +from django.utils.http import http_date +from bookwyrm import models from bookwyrm.connectors import ConnectorException, get_data -from bookwyrm.tasks import app, MEDIUM +from bookwyrm.models import base_model +from bookwyrm.signatures import make_signature +from bookwyrm.settings import DOMAIN, INSTANCE_ACTOR_USERNAME +from bookwyrm.tasks import app, MISC logger = logging.getLogger(__name__) +TBookWyrmModel = TypeVar("TBookWyrmModel", bound=base_model.BookWyrmModel) + class ActivitySerializerError(ValueError): """routine problems serializing activitypub json""" @@ -60,7 +71,13 @@ class ActivityObject: id: str type: str - def __init__(self, activity_objects=None, **kwargs): + def __init__( + self, + activity_objects: Optional[ + dict[str, Union[str, list[str], ActivityObject, base_model.BookWyrmModel]] + ] = None, + **kwargs: Any, + ): """this lets you pass in an object with fields that aren't in the dataclass, which it ignores. Any field in the dataclass is required or has a default value""" @@ -95,16 +112,34 @@ class ActivityObject: # pylint: disable=too-many-locals,too-many-branches,too-many-arguments def to_model( - self, model=None, instance=None, allow_create=True, save=True, overwrite=True - ): - """convert from an activity to a model instance""" + self, + model: Optional[type[TBookWyrmModel]] = None, + instance: Optional[TBookWyrmModel] = None, + allow_create: bool = True, + save: bool = True, + overwrite: bool = True, + allow_external_connections: bool = True, + ) -> Optional[TBookWyrmModel]: + """convert from an activity to a model instance. Args: + model: the django model that this object is being converted to + (will guess if not known) + instance: an existing database entry that is going to be updated by + this activity + allow_create: whether a new object should be created if there is no + existing object is provided or found matching the remote_id + save: store in the database if true, return an unsaved model obj if false + overwrite: replace fields in the database with this activity if true, + only update blank fields if false + allow_external_connections: look up missing data if true, + throw an exception if false and an external connection is needed + """ model = model or get_model_from_type(self.type) # only reject statuses if we're potentially creating them if ( allow_create and hasattr(model, "ignore_activity") - and model.ignore_activity(self) + and model.ignore_activity(self, allow_external_connections) ): return None @@ -122,7 +157,10 @@ class ActivityObject: for field in instance.simple_fields: try: changed = field.set_field_from_activity( - instance, self, overwrite=overwrite + instance, + self, + overwrite=overwrite, + allow_external_connections=allow_external_connections, ) if changed: update_fields.append(field.name) @@ -133,7 +171,11 @@ class ActivityObject: # too early and jank up users for field in instance.image_fields: changed = field.set_field_from_activity( - instance, self, save=save, overwrite=overwrite + instance, + self, + save=save, + overwrite=overwrite, + allow_external_connections=allow_external_connections, ) if changed: update_fields.append(field.name) @@ -156,8 +198,12 @@ class ActivityObject: # add many to many fields, which have to be set post-save for field in instance.many_to_many_fields: - # mention books/users, for example - field.set_field_from_activity(instance, self) + # mention books/users/hashtags, for example + field.set_field_from_activity( + instance, + self, + allow_external_connections=allow_external_connections, + ) # reversed relationships in the models for ( @@ -194,6 +240,11 @@ class ActivityObject: try: if issubclass(type(v), ActivityObject): data[k] = v.serialize() + elif isinstance(v, list): + data[k] = [ + e.serialize() if issubclass(type(e), ActivityObject) else e + for e in v + ] except TypeError: pass data = {k: v for (k, v) in data.items() if v is not None and k not in omit} @@ -202,7 +253,7 @@ class ActivityObject: return data -@app.task(queue=MEDIUM) +@app.task(queue=MISC) @transaction.atomic def set_related_field( model_name, origin_model_name, related_field_name, related_remote_id, data @@ -241,10 +292,10 @@ def set_related_field( def get_model_from_type(activity_type): """given the activity, what type of model""" - models = apps.get_models() + activity_models = apps.get_models() model = [ m - for m in models + for m in activity_models if hasattr(m, "activity_serializer") and hasattr(m.activity_serializer, "type") and m.activity_serializer.type == activity_type @@ -256,10 +307,48 @@ def get_model_from_type(activity_type): return model[0] +# pylint: disable=too-many-arguments +@overload def resolve_remote_id( - remote_id, model=None, refresh=False, save=True, get_activity=False -): - """take a remote_id and return an instance, creating if necessary""" + remote_id: str, + model: type[TBookWyrmModel], + refresh: bool = False, + save: bool = True, + get_activity: bool = False, + allow_external_connections: bool = True, +) -> TBookWyrmModel: + ... + + +# pylint: disable=too-many-arguments +@overload +def resolve_remote_id( + remote_id: str, + model: Optional[str] = None, + refresh: bool = False, + save: bool = True, + get_activity: bool = False, + allow_external_connections: bool = True, +) -> base_model.BookWyrmModel: + ... + + +# pylint: disable=too-many-arguments +def resolve_remote_id( + remote_id: str, + model: Optional[Union[str, type[base_model.BookWyrmModel]]] = None, + refresh: bool = False, + save: bool = True, + get_activity: bool = False, + allow_external_connections: bool = True, +) -> base_model.BookWyrmModel: + """take a remote_id and return an instance, creating if necessary. Args: + remote_id: the unique url for looking up the object in the db or by http + model: a string or object representing the model that corresponds to the object + save: whether to return an unsaved database entry or a saved one + get_activity: whether to return the activitypub object or the model object + allow_external_connections: whether to make http connections + """ if model: # a bonus check we can do if we already know the model if isinstance(model, str): model = apps.get_model(f"bookwyrm.{model}", require_ready=True) @@ -267,13 +356,26 @@ def resolve_remote_id( if result and not refresh: return result if not get_activity else result.to_activity_dataclass() + # The above block will return the object if it already exists in the database. + # If it doesn't, an external connection would be needed, so check if that's cool + if not allow_external_connections: + raise ActivitySerializerError( + "Unable to serialize object without making external HTTP requests" + ) + # load the data and create the object try: data = get_data(remote_id) - except ConnectorException: - logger.exception("Could not connect to host for remote_id: %s", remote_id) + except ConnectionError: + logger.info("Could not connect to host for remote_id: %s", remote_id) return None - + except requests.HTTPError as e: + if (e.response is not None) and e.response.status_code == 401: + # This most likely means it's a mastodon with secure fetch enabled. + data = get_activitypub_data(remote_id) + else: + logger.info("Could not connect to host for remote_id: %s", remote_id) + return None # determine the model implicitly, if not provided # or if it's a model with subclasses like Status, check again if not model or hasattr(model.objects, "select_subclasses"): @@ -292,6 +394,52 @@ def resolve_remote_id( return item.to_model(model=model, instance=result, save=save) +def get_representative(): + """Get or create an actor representing the instance + to sign requests to 'secure mastodon' servers""" + username = f"{INSTANCE_ACTOR_USERNAME}@{DOMAIN}" + email = "bookwyrm@localhost" + try: + user = models.User.objects.get(username=username) + except models.User.DoesNotExist: + user = models.User.objects.create_user( + username=username, + email=email, + local=True, + localname=INSTANCE_ACTOR_USERNAME, + ) + return user + + +def get_activitypub_data(url): + """wrapper for request.get""" + now = http_date() + sender = get_representative() + if not sender.key_pair.private_key: + # this shouldn't happen. it would be bad if it happened. + raise ValueError("No private key found for sender") + try: + resp = requests.get( + url, + headers={ + # pylint: disable=line-too-long + "Accept": 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"', + "Date": now, + "Signature": make_signature("get", sender, url, now), + }, + ) + except requests.RequestException: + raise ConnectorException() + if not resp.ok: + resp.raise_for_status() + try: + data = resp.json() + except ValueError: + raise ConnectorException() + + return data + + @dataclass(init=False) class Link(ActivityObject): """for tagging a book in a status""" @@ -306,7 +454,9 @@ class Link(ActivityObject): def serialize(self, **kwargs): """remove fields""" - omit = ("id", "type", "@context") + omit = ("id", "@context") + if self.type == "Link": + omit += ("type",) return super().serialize(omit=omit) @@ -315,3 +465,10 @@ class Mention(Link): """a subtype of Link for mentioning an actor""" type: str = "Mention" + + +@dataclass(init=False) +class Hashtag(Link): + """a subtype of Link for mentioning a hashtag""" + + type: str = "Hashtag" diff --git a/bookwyrm/activitypub/book.py b/bookwyrm/activitypub/book.py index e6a01b359..5db0dc3ac 100644 --- a/bookwyrm/activitypub/book.py +++ b/bookwyrm/activitypub/book.py @@ -1,6 +1,6 @@ """ book and author data """ from dataclasses import dataclass, field -from typing import List +from typing import Optional from .base_activity import ActivityObject from .image import Document @@ -11,17 +11,19 @@ from .image import Document class BookData(ActivityObject): """shared fields for all book data and authors""" - openlibraryKey: str = None - inventaireId: str = None - librarythingKey: str = None - goodreadsKey: str = None - bnfId: str = None - viaf: str = None - wikidata: str = None - asin: str = None - lastEditedBy: str = None - links: List[str] = field(default_factory=lambda: []) - fileLinks: List[str] = field(default_factory=lambda: []) + openlibraryKey: Optional[str] = None + inventaireId: Optional[str] = None + librarythingKey: Optional[str] = None + goodreadsKey: Optional[str] = None + bnfId: Optional[str] = None + viaf: Optional[str] = None + wikidata: Optional[str] = None + asin: Optional[str] = None + aasin: Optional[str] = None + isfdb: Optional[str] = None + lastEditedBy: Optional[str] = None + links: list[str] = field(default_factory=list) + fileLinks: list[str] = field(default_factory=list) # pylint: disable=invalid-name @@ -33,17 +35,17 @@ class Book(BookData): sortTitle: str = None subtitle: str = None description: str = "" - languages: List[str] = field(default_factory=lambda: []) + languages: list[str] = field(default_factory=list) series: str = "" seriesNumber: str = "" - subjects: List[str] = field(default_factory=lambda: []) - subjectPlaces: List[str] = field(default_factory=lambda: []) + subjects: list[str] = field(default_factory=list) + subjectPlaces: list[str] = field(default_factory=list) - authors: List[str] = field(default_factory=lambda: []) + authors: list[str] = field(default_factory=list) firstPublishedDate: str = "" publishedDate: str = "" - cover: Document = None + cover: Optional[Document] = None type: str = "Book" @@ -56,10 +58,10 @@ class Edition(Book): isbn10: str = "" isbn13: str = "" oclcNumber: str = "" - pages: int = None + pages: Optional[int] = None physicalFormat: str = "" physicalFormatDetail: str = "" - publishers: List[str] = field(default_factory=lambda: []) + publishers: list[str] = field(default_factory=list) editionRank: int = 0 type: str = "Edition" @@ -71,7 +73,7 @@ class Work(Book): """work instance of a book object""" lccn: str = "" - editions: List[str] = field(default_factory=lambda: []) + editions: list[str] = field(default_factory=list) type: str = "Work" @@ -81,12 +83,13 @@ class Author(BookData): """author of a book""" name: str - isni: str = None - viafId: str = None - gutenbergId: str = None - born: str = None - died: str = None - aliases: List[str] = field(default_factory=lambda: []) + isni: Optional[str] = None + viafId: Optional[str] = None + gutenbergId: Optional[str] = None + born: Optional[str] = None + died: Optional[str] = None + aliases: list[str] = field(default_factory=list) bio: str = "" wikipediaLink: str = "" type: str = "Author" + website: str = "" diff --git a/bookwyrm/activitypub/note.py b/bookwyrm/activitypub/note.py index eb18b8b8a..6a081058c 100644 --- a/bookwyrm/activitypub/note.py +++ b/bookwyrm/activitypub/note.py @@ -1,9 +1,12 @@ """ note serializer and children thereof """ from dataclasses import dataclass, field from typing import Dict, List -from django.apps import apps +import re -from .base_activity import ActivityObject, Link +from django.apps import apps +from django.db import IntegrityError, transaction + +from .base_activity import ActivityObject, ActivitySerializerError, Link from .image import Document @@ -38,6 +41,47 @@ class Note(ActivityObject): updated: str = None type: str = "Note" + # pylint: disable=too-many-arguments + def to_model( + self, + model=None, + instance=None, + allow_create=True, + save=True, + overwrite=True, + allow_external_connections=True, + ): + instance = super().to_model( + model, instance, allow_create, save, overwrite, allow_external_connections + ) + + if instance is None: + return instance + + # Replace links to hashtags in content with local URLs + changed_content = False + for hashtag in instance.mention_hashtags.all(): + updated_content = re.sub( + rf'({hashtag.name})', + rf"\1{hashtag.remote_id}\2", + instance.content, + flags=re.IGNORECASE, + ) + if instance.content != updated_content: + instance.content = updated_content + changed_content = True + + if not save or not changed_content: + return instance + + with transaction.atomic(): + try: + instance.save(broadcast=False, update_fields=["content"]) + except IntegrityError as e: + raise ActivitySerializerError(e) + + return instance + @dataclass(init=False) class Article(Note): diff --git a/bookwyrm/activitypub/verbs.py b/bookwyrm/activitypub/verbs.py index 36898bc7e..4b7514b5a 100644 --- a/bookwyrm/activitypub/verbs.py +++ b/bookwyrm/activitypub/verbs.py @@ -14,12 +14,12 @@ class Verb(ActivityObject): actor: str object: ActivityObject - def action(self): + def action(self, allow_external_connections=True): """usually we just want to update and save""" # self.object may return None if the object is invalid in an expected way # ie, Question type if self.object: - self.object.to_model() + self.object.to_model(allow_external_connections=allow_external_connections) # pylint: disable=invalid-name @@ -42,7 +42,7 @@ class Delete(Verb): cc: List[str] = field(default_factory=lambda: []) type: str = "Delete" - def action(self): + def action(self, allow_external_connections=True): """find and delete the activity object""" if not self.object: return @@ -52,7 +52,11 @@ class Delete(Verb): model = apps.get_model("bookwyrm.User") obj = model.find_existing_by_remote_id(self.object) else: - obj = self.object.to_model(save=False, allow_create=False) + obj = self.object.to_model( + save=False, + allow_create=False, + allow_external_connections=allow_external_connections, + ) if obj: obj.delete() @@ -67,11 +71,13 @@ class Update(Verb): to: List[str] type: str = "Update" - def action(self): + def action(self, allow_external_connections=True): """update a model instance from the dataclass""" if not self.object: return - self.object.to_model(allow_create=False) + self.object.to_model( + allow_create=False, allow_external_connections=allow_external_connections + ) @dataclass(init=False) @@ -80,10 +86,10 @@ class Undo(Verb): type: str = "Undo" - def action(self): + def action(self, allow_external_connections=True): """find and remove the activity object""" if isinstance(self.object, str): - # it may be that sometihng should be done with these, but idk what + # it may be that something should be done with these, but idk what # this seems just to be coming from pleroma return @@ -92,13 +98,28 @@ class Undo(Verb): model = None if self.object.type == "Follow": model = apps.get_model("bookwyrm.UserFollows") - obj = self.object.to_model(model=model, save=False, allow_create=False) + obj = self.object.to_model( + model=model, + save=False, + allow_create=False, + allow_external_connections=allow_external_connections, + ) if not obj: - # this could be a folloq request not a follow proper + # this could be a follow request not a follow proper model = apps.get_model("bookwyrm.UserFollowRequest") - obj = self.object.to_model(model=model, save=False, allow_create=False) + obj = self.object.to_model( + model=model, + save=False, + allow_create=False, + allow_external_connections=allow_external_connections, + ) else: - obj = self.object.to_model(model=model, save=False, allow_create=False) + obj = self.object.to_model( + model=model, + save=False, + allow_create=False, + allow_external_connections=allow_external_connections, + ) if not obj: # if we don't have the object, we can't undo it. happens a lot with boosts return @@ -112,9 +133,9 @@ class Follow(Verb): object: str type: str = "Follow" - def action(self): + def action(self, allow_external_connections=True): """relationship save""" - self.to_model() + self.to_model(allow_external_connections=allow_external_connections) @dataclass(init=False) @@ -124,9 +145,9 @@ class Block(Verb): object: str type: str = "Block" - def action(self): + def action(self, allow_external_connections=True): """relationship save""" - self.to_model() + self.to_model(allow_external_connections=allow_external_connections) @dataclass(init=False) @@ -136,7 +157,7 @@ class Accept(Verb): object: Follow type: str = "Accept" - def action(self): + def action(self, allow_external_connections=True): """accept a request""" obj = self.object.to_model(save=False, allow_create=True) obj.accept() @@ -149,7 +170,7 @@ class Reject(Verb): object: Follow type: str = "Reject" - def action(self): + def action(self, allow_external_connections=True): """reject a follow request""" obj = self.object.to_model(save=False, allow_create=False) obj.reject() @@ -163,7 +184,7 @@ class Add(Verb): object: CollectionItem type: str = "Add" - def action(self): + def action(self, allow_external_connections=True): """figure out the target to assign the item to a collection""" target = resolve_remote_id(self.target) item = self.object.to_model(save=False) @@ -177,7 +198,7 @@ class Remove(Add): type: str = "Remove" - def action(self): + def action(self, allow_external_connections=True): """find and remove the activity object""" obj = self.object.to_model(save=False, allow_create=False) if obj: @@ -191,9 +212,9 @@ class Like(Verb): object: str type: str = "Like" - def action(self): + def action(self, allow_external_connections=True): """like""" - self.to_model() + self.to_model(allow_external_connections=allow_external_connections) # pylint: disable=invalid-name @@ -207,6 +228,6 @@ class Announce(Verb): object: str type: str = "Announce" - def action(self): + def action(self, allow_external_connections=True): """boost""" - self.to_model() + self.to_model(allow_external_connections=allow_external_connections) diff --git a/bookwyrm/activitystreams.py b/bookwyrm/activitystreams.py index f8312f063..42f99e209 100644 --- a/bookwyrm/activitystreams.py +++ b/bookwyrm/activitystreams.py @@ -4,27 +4,32 @@ from django.dispatch import receiver from django.db import transaction from django.db.models import signals, Q from django.utils import timezone +from opentelemetry import trace from bookwyrm import models from bookwyrm.redis_store import RedisStore, r -from bookwyrm.tasks import app, LOW, MEDIUM, HIGH +from bookwyrm.tasks import app, STREAMS, IMPORT_TRIGGERED +from bookwyrm.telemetry import open_telemetry + + +tracer = open_telemetry.tracer() class ActivityStream(RedisStore): """a category of activity stream (like home, local, books)""" - def stream_id(self, user): + def stream_id(self, user_id): """the redis key for this user's instance of this stream""" - return f"{user.id}-{self.key}" + return f"{user_id}-{self.key}" - def unread_id(self, user): + def unread_id(self, user_id): """the redis key for this user's unread count for this stream""" - stream_id = self.stream_id(user) + stream_id = self.stream_id(user_id) return f"{stream_id}-unread" - def unread_by_status_type_id(self, user): + def unread_by_status_type_id(self, user_id): """the redis key for this user's unread count for this stream""" - stream_id = self.stream_id(user) + stream_id = self.stream_id(user_id) return f"{stream_id}-unread-by-type" def get_rank(self, obj): # pylint: disable=no-self-use @@ -33,16 +38,19 @@ class ActivityStream(RedisStore): def add_status(self, status, increment_unread=False): """add a status to users' feeds""" + audience = self.get_audience(status) # the pipeline contains all the add-to-stream activities - pipeline = self.add_object_to_related_stores(status, execute=False) + pipeline = self.add_object_to_stores( + status, self.get_stores_for_users(audience), execute=False + ) if increment_unread: - for user in self.get_audience(status): + for user_id in audience: # add to the unread status count - pipeline.incr(self.unread_id(user)) + pipeline.incr(self.unread_id(user_id)) # add to the unread status count for status type pipeline.hincrby( - self.unread_by_status_type_id(user), get_status_type(status), 1 + self.unread_by_status_type_id(user_id), get_status_type(status), 1 ) # and go! @@ -52,21 +60,21 @@ class ActivityStream(RedisStore): """add a user's statuses to another user's feed""" # only add the statuses that the viewer should be able to see (ie, not dms) statuses = models.Status.privacy_filter(viewer).filter(user=user) - self.bulk_add_objects_to_store(statuses, self.stream_id(viewer)) + self.bulk_add_objects_to_store(statuses, self.stream_id(viewer.id)) def remove_user_statuses(self, viewer, user): """remove a user's status from another user's feed""" # remove all so that followers only statuses are removed statuses = user.status_set.all() - self.bulk_remove_objects_from_store(statuses, self.stream_id(viewer)) + self.bulk_remove_objects_from_store(statuses, self.stream_id(viewer.id)) def get_activity_stream(self, user): """load the statuses to be displayed""" # clear unreads for this feed - r.set(self.unread_id(user), 0) - r.delete(self.unread_by_status_type_id(user)) + r.set(self.unread_id(user.id), 0) + r.delete(self.unread_by_status_type_id(user.id)) - statuses = self.get_store(self.stream_id(user)) + statuses = self.get_store(self.stream_id(user.id)) return ( models.Status.objects.select_subclasses() .filter(id__in=statuses) @@ -83,11 +91,11 @@ class ActivityStream(RedisStore): def get_unread_count(self, user): """get the unread status count for this user's feed""" - return int(r.get(self.unread_id(user)) or 0) + return int(r.get(self.unread_id(user.id)) or 0) def get_unread_count_by_status_type(self, user): """get the unread status count for this user's feed's status types""" - status_types = r.hgetall(self.unread_by_status_type_id(user)) + status_types = r.hgetall(self.unread_by_status_type_id(user.id)) return { str(key.decode("utf-8")): int(value) or 0 for key, value in status_types.items() @@ -95,13 +103,20 @@ class ActivityStream(RedisStore): def populate_streams(self, user): """go from zero to a timeline""" - self.populate_store(self.stream_id(user)) + self.populate_store(self.stream_id(user.id)) - def get_audience(self, status): # pylint: disable=no-self-use - """given a status, what users should see it""" - # direct messages don't appeard in feeds, direct comments/reviews/etc do + @tracer.start_as_current_span("ActivityStream._get_audience") + def _get_audience(self, status): # pylint: disable=no-self-use + """given a status, what users should see it, excluding the author""" + trace.get_current_span().set_attribute("status_type", status.status_type) + trace.get_current_span().set_attribute("status_privacy", status.privacy) + trace.get_current_span().set_attribute( + "status_reply_parent_privacy", + status.reply_parent.privacy if status.reply_parent else status.privacy, + ) + # direct messages don't appear in feeds, direct comments/reviews/etc do if status.privacy == "direct" and status.status_type == "Note": - return [] + return models.User.objects.none() # everybody who could plausibly see this status audience = models.User.objects.filter( @@ -114,15 +129,13 @@ class ActivityStream(RedisStore): # only visible to the poster and mentioned users if status.privacy == "direct": audience = audience.filter( - Q(id=status.user.id) # if the user is the post's author - | Q(id__in=status.mention_users.all()) # if the user is mentioned + Q(id__in=status.mention_users.all()) # if the user is mentioned ) # don't show replies to statuses the user can't see elif status.reply_parent and status.reply_parent.privacy == "followers": audience = audience.filter( - Q(id=status.user.id) # if the user is the post's author - | Q(id=status.reply_parent.user.id) # if the user is the OG author + Q(id=status.reply_parent.user.id) # if the user is the OG author | ( Q(following=status.user) & Q(following=status.reply_parent.user) ) # if the user is following both authors @@ -131,13 +144,23 @@ class ActivityStream(RedisStore): # only visible to the poster's followers and tagged users elif status.privacy == "followers": audience = audience.filter( - Q(id=status.user.id) # if the user is the post's author - | Q(following=status.user) # if the user is following the author + Q(following=status.user) # if the user is following the author ) return audience.distinct() - def get_stores_for_object(self, obj): - return [self.stream_id(u) for u in self.get_audience(obj)] + @tracer.start_as_current_span("ActivityStream.get_audience") + def get_audience(self, status): + """given a status, what users should see it""" + trace.get_current_span().set_attribute("stream_id", self.key) + audience = self._get_audience(status).values_list("id", flat=True) + status_author = models.User.objects.filter( + is_active=True, local=True, id=status.user.id + ).values_list("id", flat=True) + return list(set(list(audience) + list(status_author))) + + def get_stores_for_users(self, user_ids): + """convert a list of user ids into redis store ids""" + return [self.stream_id(user_id) for user_id in user_ids] def get_statuses_for_user(self, user): # pylint: disable=no-self-use """given a user, what statuses should they see on this stream""" @@ -156,14 +179,19 @@ class HomeStream(ActivityStream): key = "home" + @tracer.start_as_current_span("HomeStream.get_audience") def get_audience(self, status): - audience = super().get_audience(status) + trace.get_current_span().set_attribute("stream_id", self.key) + audience = super()._get_audience(status) if not audience: return [] - return audience.filter( - Q(id=status.user.id) # if the user is the post's author - | Q(following=status.user) # if the user is following the author - ).distinct() + # if the user is following the author + audience = audience.filter(following=status.user).values_list("id", flat=True) + # if the user is the post's author + status_author = models.User.objects.filter( + is_active=True, local=True, id=status.user.id + ).values_list("id", flat=True) + return list(set(list(audience) + list(status_author))) def get_statuses_for_user(self, user): return models.Status.privacy_filter( @@ -202,8 +230,20 @@ class BooksStream(ActivityStream): key = "books" - def get_audience(self, status): + def _get_audience(self, status): """anyone with the mentioned book on their shelves""" + work = ( + status.book.parent_work + if hasattr(status, "book") + else status.mention_books.first().parent_work + ) + + audience = super()._get_audience(status) + if not audience: + return models.User.objects.none() + return audience.filter(shelfbook__book__parent_work=work).distinct() + + def get_audience(self, status): # only show public statuses on the books feed, # and only statuses that mention books if status.privacy != "public" or not ( @@ -211,16 +251,7 @@ class BooksStream(ActivityStream): ): return [] - work = ( - status.book.parent_work - if hasattr(status, "book") - else status.mention_books.first().parent_work - ) - - audience = super().get_audience(status) - if not audience: - return [] - return audience.filter(shelfbook__book__parent_work=work).distinct() + return super().get_audience(status) def get_statuses_for_user(self, user): """any public status that mentions the user's books""" @@ -244,38 +275,38 @@ class BooksStream(ActivityStream): def add_book_statuses(self, user, book): """add statuses about a book to a user's feed""" work = book.parent_work - statuses = ( - models.Status.privacy_filter( - user, - privacy_levels=["public"], - ) - .filter( - Q(comment__book__parent_work=work) - | Q(quotation__book__parent_work=work) - | Q(review__book__parent_work=work) - | Q(mention_books__parent_work=work) - ) - .distinct() + statuses = models.Status.privacy_filter( + user, + privacy_levels=["public"], ) - self.bulk_add_objects_to_store(statuses, self.stream_id(user)) + + book_comments = statuses.filter(Q(comment__book__parent_work=work)) + book_quotations = statuses.filter(Q(quotation__book__parent_work=work)) + book_reviews = statuses.filter(Q(review__book__parent_work=work)) + book_mentions = statuses.filter(Q(mention_books__parent_work=work)) + + self.bulk_add_objects_to_store(book_comments, self.stream_id(user.id)) + self.bulk_add_objects_to_store(book_quotations, self.stream_id(user.id)) + self.bulk_add_objects_to_store(book_reviews, self.stream_id(user.id)) + self.bulk_add_objects_to_store(book_mentions, self.stream_id(user.id)) def remove_book_statuses(self, user, book): """add statuses about a book to a user's feed""" work = book.parent_work - statuses = ( - models.Status.privacy_filter( - user, - privacy_levels=["public"], - ) - .filter( - Q(comment__book__parent_work=work) - | Q(quotation__book__parent_work=work) - | Q(review__book__parent_work=work) - | Q(mention_books__parent_work=work) - ) - .distinct() + statuses = models.Status.privacy_filter( + user, + privacy_levels=["public"], ) - self.bulk_remove_objects_from_store(statuses, self.stream_id(user)) + + book_comments = statuses.filter(Q(comment__book__parent_work=work)) + book_quotations = statuses.filter(Q(quotation__book__parent_work=work)) + book_reviews = statuses.filter(Q(review__book__parent_work=work)) + book_mentions = statuses.filter(Q(mention_books__parent_work=work)) + + self.bulk_remove_objects_from_store(book_comments, self.stream_id(user.id)) + self.bulk_remove_objects_from_store(book_quotations, self.stream_id(user.id)) + self.bulk_remove_objects_from_store(book_reviews, self.stream_id(user.id)) + self.bulk_remove_objects_from_store(book_mentions, self.stream_id(user.id)) # determine which streams are enabled in settings.py @@ -298,6 +329,11 @@ def add_status_on_create(sender, instance, created, *args, **kwargs): remove_status_task.delay(instance.id) return + # We don't want to create multiple add_status_tasks for each status, and because + # the transactions are atomic, on_commit won't run until the status is ready to add. + if not created: + return + # when creating new things, gotta wait on the transaction transaction.on_commit( lambda: add_status_on_create_command(sender, instance, created) @@ -306,13 +342,21 @@ def add_status_on_create(sender, instance, created, *args, **kwargs): def add_status_on_create_command(sender, instance, created): """runs this code only after the database commit completes""" - priority = HIGH + # boosts trigger 'saves" twice, so don't bother duplicating the task + if sender == models.Boost and not created: + return + + priority = STREAMS # check if this is an old status, de-prioritize if so # (this will happen if federation is very slow, or, more expectedly, on csv import) if instance.published_date < timezone.now() - timedelta( days=1 ) or instance.created_date < instance.published_date - timedelta(days=1): - priority = LOW + # a backdated status from a local user is an import, don't add it + if instance.user.local: + return + # an out of date remote status is a low priority but should be added + priority = IMPORT_TRIGGERED add_status_task.apply_async( args=(instance.id,), @@ -456,7 +500,7 @@ def remove_statuses_on_unshelve(sender, instance, *args, **kwargs): # ---- TASKS -@app.task(queue=LOW) +@app.task(queue=STREAMS) def add_book_statuses_task(user_id, book_id): """add statuses related to a book on shelve""" user = models.User.objects.get(id=user_id) @@ -464,7 +508,7 @@ def add_book_statuses_task(user_id, book_id): BooksStream().add_book_statuses(user, book) -@app.task(queue=LOW) +@app.task(queue=STREAMS) def remove_book_statuses_task(user_id, book_id): """remove statuses about a book from a user's books feed""" user = models.User.objects.get(id=user_id) @@ -472,7 +516,7 @@ def remove_book_statuses_task(user_id, book_id): BooksStream().remove_book_statuses(user, book) -@app.task(queue=MEDIUM) +@app.task(queue=STREAMS) def populate_stream_task(stream, user_id): """background task for populating an empty activitystream""" user = models.User.objects.get(id=user_id) @@ -480,7 +524,7 @@ def populate_stream_task(stream, user_id): stream.populate_streams(user) -@app.task(queue=MEDIUM) +@app.task(queue=STREAMS) def remove_status_task(status_ids): """remove a status from any stream it might be in""" # this can take an id or a list of ids @@ -490,10 +534,12 @@ def remove_status_task(status_ids): for stream in streams.values(): for status in statuses: - stream.remove_object_from_related_stores(status) + stream.remove_object_from_stores( + status, stream.get_stores_for_users(stream.get_audience(status)) + ) -@app.task(queue=HIGH) +@app.task(queue=STREAMS) def add_status_task(status_id, increment_unread=False): """add a status to any stream it should be in""" status = models.Status.objects.select_subclasses().get(id=status_id) @@ -505,7 +551,7 @@ def add_status_task(status_id, increment_unread=False): stream.add_status(status, increment_unread=increment_unread) -@app.task(queue=MEDIUM) +@app.task(queue=STREAMS) def remove_user_statuses_task(viewer_id, user_id, stream_list=None): """remove all statuses by a user from a viewer's stream""" stream_list = [streams[s] for s in stream_list] if stream_list else streams.values() @@ -515,7 +561,7 @@ def remove_user_statuses_task(viewer_id, user_id, stream_list=None): stream.remove_user_statuses(viewer, user) -@app.task(queue=MEDIUM) +@app.task(queue=STREAMS) def add_user_statuses_task(viewer_id, user_id, stream_list=None): """add all statuses by a user to a viewer's stream""" stream_list = [streams[s] for s in stream_list] if stream_list else streams.values() @@ -525,7 +571,7 @@ def add_user_statuses_task(viewer_id, user_id, stream_list=None): stream.add_user_statuses(viewer, user) -@app.task(queue=MEDIUM) +@app.task(queue=STREAMS) def handle_boost_task(boost_id): """remove the original post and other, earlier boosts""" instance = models.Status.objects.get(id=boost_id) @@ -539,10 +585,10 @@ def handle_boost_task(boost_id): for stream in streams.values(): # people who should see the boost (not people who see the original status) - audience = stream.get_stores_for_object(instance) - stream.remove_object_from_related_stores(boosted, stores=audience) + audience = stream.get_stores_for_users(stream.get_audience(instance)) + stream.remove_object_from_stores(boosted, audience) for status in old_versions: - stream.remove_object_from_related_stores(status, stores=audience) + stream.remove_object_from_stores(status, audience) def get_status_type(status): diff --git a/bookwyrm/apps.py b/bookwyrm/apps.py index 786f86e1c..b0c3e3fa4 100644 --- a/bookwyrm/apps.py +++ b/bookwyrm/apps.py @@ -35,11 +35,12 @@ class BookwyrmConfig(AppConfig): # pylint: disable=no-self-use def ready(self): """set up OTLP and preview image files, if desired""" - if settings.OTEL_EXPORTER_OTLP_ENDPOINT: + if settings.OTEL_EXPORTER_OTLP_ENDPOINT or settings.OTEL_EXPORTER_CONSOLE: # pylint: disable=import-outside-toplevel from bookwyrm.telemetry import open_telemetry open_telemetry.instrumentDjango() + open_telemetry.instrumentPostgres() if settings.ENABLE_PREVIEW_IMAGES and settings.FONTS: # Download any fonts that we don't have yet diff --git a/bookwyrm/book_search.py b/bookwyrm/book_search.py index 803641cb7..ceb228f40 100644 --- a/bookwyrm/book_search.py +++ b/bookwyrm/book_search.py @@ -1,24 +1,62 @@ """ using a bookwyrm instance as a source of book data """ +from __future__ import annotations from dataclasses import asdict, dataclass from functools import reduce import operator +from typing import Optional, Union, Any, Literal, overload from django.contrib.postgres.search import SearchRank, SearchQuery -from django.db.models import OuterRef, Subquery, F, Q +from django.db.models import F, Q +from django.db.models.query import QuerySet from bookwyrm import models from bookwyrm import connectors from bookwyrm.settings import MEDIA_FULL_URL +@overload +def search( + query: str, + *, + min_confidence: float = 0, + filters: Optional[list[Any]] = None, + return_first: Literal[False], +) -> QuerySet[models.Edition]: + ... + + +@overload +def search( + query: str, + *, + min_confidence: float = 0, + filters: Optional[list[Any]] = None, + return_first: Literal[True], +) -> Optional[models.Edition]: + ... + + # pylint: disable=arguments-differ -def search(query, min_confidence=0, filters=None, return_first=False): +def search( + query: str, + *, + min_confidence: float = 0, + filters: Optional[list[Any]] = None, + return_first: bool = False, +) -> Union[Optional[models.Edition], QuerySet[models.Edition]]: """search your local database""" filters = filters or [] if not query: - return [] - # first, try searching unqiue identifiers - results = search_identifiers(query, *filters, return_first=return_first) + return None if return_first else [] + query = query.strip() + + results = None + # first, try searching unique identifiers + # unique identifiers never have spaces, title/author usually do + if not " " in query: + results = search_identifiers(query, *filters, return_first=return_first) + + # if there were no identifier results... if not results: # then try searching title/author results = search_title_author( @@ -35,24 +73,10 @@ def isbn_search(query): # If the ISBN has only 9 characters, prepend missing zero query = query.strip().upper().rjust(10, "0") filters = [{f: query} for f in ["isbn_10", "isbn_13"]] - results = models.Edition.objects.filter( + return models.Edition.objects.filter( reduce(operator.or_, (Q(**f) for f in filters)) ).distinct() - # when there are multiple editions of the same work, pick the default. - # it would be odd for this to happen. - - default_editions = models.Edition.objects.filter( - parent_work=OuterRef("parent_work") - ).order_by("-edition_rank") - results = ( - results.annotate(default_id=Subquery(default_editions.values("id")[:1])).filter( - default_id=F("id") - ) - or results - ) - return results - def format_search_result(search_result): """convert a book object into a search result object""" @@ -73,7 +97,9 @@ def format_search_result(search_result): ).json() -def search_identifiers(query, *filters, return_first=False): +def search_identifiers( + query, *filters, return_first=False +) -> Union[Optional[models.Edition], QuerySet[models.Edition]]: """tries remote_id, isbn; defined as dedupe fields on the model""" if connectors.maybe_isbn(query): # Oh did you think the 'S' in ISBN stood for 'standard'? @@ -88,28 +114,15 @@ def search_identifiers(query, *filters, return_first=False): results = models.Edition.objects.filter( *filters, reduce(operator.or_, (Q(**f) for f in or_filters)) ).distinct() - if results.count() <= 1: - if return_first: - return results.first() - return results - # when there are multiple editions of the same work, pick the default. - # it would be odd for this to happen. - default_editions = models.Edition.objects.filter( - parent_work=OuterRef("parent_work") - ).order_by("-edition_rank") - results = ( - results.annotate(default_id=Subquery(default_editions.values("id")[:1])).filter( - default_id=F("id") - ) - or results - ) if return_first: return results.first() return results -def search_title_author(query, min_confidence, *filters, return_first=False): +def search_title_author( + query, min_confidence, *filters, return_first=False +) -> QuerySet[models.Edition]: """searches for title and author""" query = SearchQuery(query, config="simple") | SearchQuery(query, config="english") results = ( @@ -120,19 +133,16 @@ def search_title_author(query, min_confidence, *filters, return_first=False): ) # when there are multiple editions of the same work, pick the closest - editions_of_work = results.values("parent_work__id").values_list("parent_work__id") + editions_of_work = results.values_list("parent_work__id", flat=True).distinct() # filter out multiple editions of the same work list_results = [] - for work_id in set(editions_of_work): - editions = results.filter(parent_work=work_id) - default = editions.order_by("-edition_rank").first() - default_rank = default.rank if default else 0 - # if mutliple books have the top rank, pick the default edition - if default_rank == editions.first().rank: - result = default - else: - result = editions.first() + for work_id in set(editions_of_work[:30]): + result = ( + results.filter(parent_work=work_id) + .order_by("-rank", "-edition_rank") + .first() + ) if return_first: return result @@ -147,11 +157,11 @@ class SearchResult: title: str key: str connector: object - view_link: str = None - author: str = None - year: str = None - cover: str = None - confidence: int = 1 + view_link: Optional[str] = None + author: Optional[str] = None + year: Optional[str] = None + cover: Optional[str] = None + confidence: float = 1.0 def __repr__(self): # pylint: disable=consider-using-f-string diff --git a/bookwyrm/connectors/abstract_connector.py b/bookwyrm/connectors/abstract_connector.py index c1ee7fe78..8b6dcb885 100644 --- a/bookwyrm/connectors/abstract_connector.py +++ b/bookwyrm/connectors/abstract_connector.py @@ -1,44 +1,55 @@ """ functionality outline for a book data connector """ +from __future__ import annotations from abc import ABC, abstractmethod +from typing import Optional, TypedDict, Any, Callable, Union, Iterator +from urllib.parse import quote_plus import imghdr import logging import re +import asyncio +import requests +from requests.exceptions import RequestException +import aiohttp from django.core.files.base import ContentFile from django.db import transaction -import requests -from requests.exceptions import RequestException from bookwyrm import activitypub, models, settings +from bookwyrm.settings import USER_AGENT from .connector_manager import load_more_data, ConnectorException, raise_not_valid_url from .format_mappings import format_mappings - +from ..book_search import SearchResult logger = logging.getLogger(__name__) +JsonDict = dict[str, Any] + + +class ConnectorResults(TypedDict): + """TypedDict for results returned by connector""" + + connector: AbstractMinimalConnector + results: list[SearchResult] + class AbstractMinimalConnector(ABC): """just the bare bones, for other bookwyrm instances""" - def __init__(self, identifier): + def __init__(self, identifier: str): # load connector settings info = models.Connector.objects.get(identifier=identifier) self.connector = info # the things in the connector model to copy over - self_fields = [ - "base_url", - "books_url", - "covers_url", - "search_url", - "isbn_search_url", - "name", - "identifier", - ] - for field in self_fields: - setattr(self, field, getattr(info, field)) + self.base_url = info.base_url + self.books_url = info.books_url + self.covers_url = info.covers_url + self.search_url = info.search_url + self.isbn_search_url = info.isbn_search_url + self.name = info.name + self.identifier = info.identifier - def get_search_url(self, query): + def get_search_url(self, query: str) -> str: """format the query url""" # Check if the query resembles an ISBN if maybe_isbn(query) and self.isbn_search_url and self.isbn_search_url != "": @@ -48,44 +59,93 @@ class AbstractMinimalConnector(ABC): return f"{self.isbn_search_url}{normalized_query}" # NOTE: previously, we tried searching isbn and if that produces no results, # searched as free text. This, instead, only searches isbn if it's isbn-y - return f"{self.search_url}{query}" + return f"{self.search_url}{quote_plus(query)}" - def process_search_response(self, query, data, min_confidence): - """Format the search results based on the formt of the query""" + def process_search_response( + self, query: str, data: Any, min_confidence: float + ) -> list[SearchResult]: + """Format the search results based on the format of the query""" if maybe_isbn(query): return list(self.parse_isbn_search_data(data))[:10] return list(self.parse_search_data(data, min_confidence))[:10] + async def get_results( + self, + session: aiohttp.ClientSession, + url: str, + min_confidence: float, + query: str, + ) -> Optional[ConnectorResults]: + """try this specific connector""" + # pylint: disable=line-too-long + headers = { + "Accept": ( + 'application/json, application/activity+json, application/ld+json; profile="https://www.w3.org/ns/activitystreams"; charset=utf-8' + ), + "User-Agent": USER_AGENT, + } + params = {"min_confidence": min_confidence} + try: + async with session.get(url, headers=headers, params=params) as response: + if not response.ok: + logger.info("Unable to connect to %s: %s", url, response.reason) + return None + + try: + raw_data = await response.json() + except aiohttp.client_exceptions.ContentTypeError as err: + logger.exception(err) + return None + + return ConnectorResults( + connector=self, + results=self.process_search_response( + query, raw_data, min_confidence + ), + ) + except asyncio.TimeoutError: + logger.info("Connection timed out for url: %s", url) + except aiohttp.ClientError as err: + logger.info(err) + return None + @abstractmethod - def get_or_create_book(self, remote_id): + def get_or_create_book(self, remote_id: str) -> Optional[models.Book]: """pull up a book record by whatever means possible""" @abstractmethod - def parse_search_data(self, data, min_confidence): + def parse_search_data( + self, data: Any, min_confidence: float + ) -> Iterator[SearchResult]: """turn the result json from a search into a list""" @abstractmethod - def parse_isbn_search_data(self, data): + def parse_isbn_search_data(self, data: Any) -> Iterator[SearchResult]: """turn the result json from a search into a list""" class AbstractConnector(AbstractMinimalConnector): """generic book data connector""" - def __init__(self, identifier): + generated_remote_link_field = "" + + def __init__(self, identifier: str): super().__init__(identifier) # fields we want to look for in book data to copy over # title we handle separately. - self.book_mappings = [] + self.book_mappings: list[Mapping] = [] + self.author_mappings: list[Mapping] = [] - def get_or_create_book(self, remote_id): + def get_or_create_book(self, remote_id: str) -> Optional[models.Book]: """translate arbitrary json into an Activitypub dataclass""" # first, check if we have the origin_id saved existing = models.Edition.find_existing_by_remote_id( remote_id ) or models.Work.find_existing_by_remote_id(remote_id) if existing: - if hasattr(existing, "default_edition"): + if hasattr(existing, "default_edition") and isinstance( + existing.default_edition, models.Edition + ): return existing.default_edition return existing @@ -117,6 +177,9 @@ class AbstractConnector(AbstractMinimalConnector): ) # this will dedupe automatically work = work_activity.to_model(model=models.Work, overwrite=False) + if not work: + return None + for author in self.get_authors_from_data(work_data): work.authors.add(author) @@ -124,12 +187,21 @@ class AbstractConnector(AbstractMinimalConnector): load_more_data.delay(self.connector.id, work.id) return edition - def get_book_data(self, remote_id): # pylint: disable=no-self-use + def get_book_data(self, remote_id: str) -> JsonDict: # pylint: disable=no-self-use """this allows connectors to override the default behavior""" return get_data(remote_id) - def create_edition_from_data(self, work, edition_data, instance=None): + def create_edition_from_data( + self, + work: models.Work, + edition_data: Union[str, JsonDict], + instance: Optional[models.Edition] = None, + ) -> Optional[models.Edition]: """if we already have the work, we're ready""" + if isinstance(edition_data, str): + # We don't expect a string here + return None + mapped_data = dict_from_mappings(edition_data, self.book_mappings) mapped_data["work"] = work.remote_id edition_activity = activitypub.Edition(**mapped_data) @@ -137,6 +209,9 @@ class AbstractConnector(AbstractMinimalConnector): model=models.Edition, overwrite=False, instance=instance ) + if not edition: + return None + # if we're updating an existing instance, we don't need to load authors if instance: return edition @@ -153,7 +228,9 @@ class AbstractConnector(AbstractMinimalConnector): return edition - def get_or_create_author(self, remote_id, instance=None): + def get_or_create_author( + self, remote_id: str, instance: Optional[models.Author] = None + ) -> Optional[models.Author]: """load that author""" if not instance: existing = models.Author.find_existing_by_remote_id(remote_id) @@ -173,46 +250,51 @@ class AbstractConnector(AbstractMinimalConnector): model=models.Author, overwrite=False, instance=instance ) - def get_remote_id_from_model(self, obj): + def get_remote_id_from_model(self, obj: models.BookDataModel) -> Optional[str]: """given the data stored, how can we look this up""" - return getattr(obj, getattr(self, "generated_remote_link_field")) + remote_id: Optional[str] = getattr(obj, self.generated_remote_link_field) + return remote_id - def update_author_from_remote(self, obj): + def update_author_from_remote(self, obj: models.Author) -> Optional[models.Author]: """load the remote data from this connector and add it to an existing author""" remote_id = self.get_remote_id_from_model(obj) + if not remote_id: + return None return self.get_or_create_author(remote_id, instance=obj) - def update_book_from_remote(self, obj): + def update_book_from_remote(self, obj: models.Edition) -> Optional[models.Edition]: """load the remote data from this connector and add it to an existing book""" remote_id = self.get_remote_id_from_model(obj) + if not remote_id: + return None data = self.get_book_data(remote_id) return self.create_edition_from_data(obj.parent_work, data, instance=obj) @abstractmethod - def is_work_data(self, data): + def is_work_data(self, data: JsonDict) -> bool: """differentiate works and editions""" @abstractmethod - def get_edition_from_work_data(self, data): + def get_edition_from_work_data(self, data: JsonDict) -> JsonDict: """every work needs at least one edition""" @abstractmethod - def get_work_from_edition_data(self, data): + def get_work_from_edition_data(self, data: JsonDict) -> JsonDict: """every edition needs a work""" @abstractmethod - def get_authors_from_data(self, data): + def get_authors_from_data(self, data: JsonDict) -> Iterator[models.Author]: """load author data""" @abstractmethod - def expand_book_data(self, book): + def expand_book_data(self, book: models.Book) -> None: """get more info on a book""" -def dict_from_mappings(data, mappings): +def dict_from_mappings(data: JsonDict, mappings: list[Mapping]) -> JsonDict: """create a dict in Activitypub format, using mappings supplies by the subclass""" - result = {} + result: JsonDict = {} for mapping in mappings: # sometimes there are multiple mappings for one field, don't # overwrite earlier writes in that case @@ -222,7 +304,11 @@ def dict_from_mappings(data, mappings): return result -def get_data(url, params=None, timeout=10): +def get_data( + url: str, + params: Optional[dict[str, str]] = None, + timeout: int = settings.QUERY_TIMEOUT, +) -> JsonDict: """wrapper for request.get""" # check if the url is blocked raise_not_valid_url(url) @@ -244,17 +330,26 @@ def get_data(url, params=None, timeout=10): raise ConnectorException(err) if not resp.ok: - raise ConnectorException() + if resp.status_code == 401: + # this is probably an AUTHORIZED_FETCH issue + resp.raise_for_status() + else: + raise ConnectorException() try: data = resp.json() except ValueError as err: logger.info(err) raise ConnectorException(err) + if not isinstance(data, dict): + raise ConnectorException("Unexpected data format") + return data -def get_image(url, timeout=10): +def get_image( + url: str, timeout: int = 10 +) -> Union[tuple[ContentFile[bytes], str], tuple[None, None]]: """wrapper for requesting an image""" raise_not_valid_url(url) try: @@ -284,14 +379,19 @@ def get_image(url, timeout=10): class Mapping: """associate a local database field with a field in an external dataset""" - def __init__(self, local_field, remote_field=None, formatter=None): + def __init__( + self, + local_field: str, + remote_field: Optional[str] = None, + formatter: Optional[Callable[[Any], Any]] = None, + ): noop = lambda x: x self.local_field = local_field self.remote_field = remote_field or local_field self.formatter = formatter or noop - def get_value(self, data): + def get_value(self, data: JsonDict) -> Optional[Any]: """pull a field from incoming json and return the formatted version""" value = data.get(self.remote_field) if not value: @@ -302,7 +402,7 @@ class Mapping: return None -def infer_physical_format(format_text): +def infer_physical_format(format_text: str) -> Optional[str]: """try to figure out what the standardized format is from the free value""" format_text = format_text.lower() if format_text in format_mappings: @@ -315,8 +415,8 @@ def infer_physical_format(format_text): return matches[0] -def unique_physical_format(format_text): - """only store the format if it isn't diretly in the format mappings""" +def unique_physical_format(format_text: str) -> Optional[str]: + """only store the format if it isn't directly in the format mappings""" format_text = format_text.lower() if format_text in format_mappings: # try a direct match, so saving this would be redundant @@ -324,7 +424,7 @@ def unique_physical_format(format_text): return format_text -def maybe_isbn(query): +def maybe_isbn(query: str) -> bool: """check if a query looks like an isbn""" isbn = re.sub(r"[\W_]", "", query) # removes filler characters # ISBNs must be numeric except an ISBN10 checkdigit can be 'X' diff --git a/bookwyrm/connectors/bookwyrm_connector.py b/bookwyrm/connectors/bookwyrm_connector.py index e07a0b281..4064f4b4c 100644 --- a/bookwyrm/connectors/bookwyrm_connector.py +++ b/bookwyrm/connectors/bookwyrm_connector.py @@ -1,4 +1,7 @@ """ using another bookwyrm instance as a source of book data """ +from __future__ import annotations +from typing import Any, Iterator + from bookwyrm import activitypub, models from bookwyrm.book_search import SearchResult from .abstract_connector import AbstractMinimalConnector @@ -7,15 +10,19 @@ from .abstract_connector import AbstractMinimalConnector class Connector(AbstractMinimalConnector): """this is basically just for search""" - def get_or_create_book(self, remote_id): + def get_or_create_book(self, remote_id: str) -> models.Edition: return activitypub.resolve_remote_id(remote_id, model=models.Edition) - def parse_search_data(self, data, min_confidence): + def parse_search_data( + self, data: list[dict[str, Any]], min_confidence: float + ) -> Iterator[SearchResult]: for search_result in data: search_result["connector"] = self yield SearchResult(**search_result) - def parse_isbn_search_data(self, data): + def parse_isbn_search_data( + self, data: list[dict[str, Any]] + ) -> Iterator[SearchResult]: for search_result in data: search_result["connector"] = self yield SearchResult(**search_result) diff --git a/bookwyrm/connectors/connector_manager.py b/bookwyrm/connectors/connector_manager.py index 9a6f834af..444a626ba 100644 --- a/bookwyrm/connectors/connector_manager.py +++ b/bookwyrm/connectors/connector_manager.py @@ -1,8 +1,11 @@ """ interface with whatever connectors the app has """ +from __future__ import annotations import asyncio import importlib import ipaddress import logging +from asyncio import Future +from typing import Iterator, Any, Optional, Union, overload, Literal from urllib.parse import urlparse import aiohttp @@ -12,8 +15,10 @@ from django.db.models import signals from requests import HTTPError from bookwyrm import book_search, models -from bookwyrm.settings import SEARCH_TIMEOUT, USER_AGENT -from bookwyrm.tasks import app, LOW +from bookwyrm.book_search import SearchResult +from bookwyrm.connectors import abstract_connector +from bookwyrm.settings import SEARCH_TIMEOUT +from bookwyrm.tasks import app, CONNECTORS logger = logging.getLogger(__name__) @@ -22,61 +27,46 @@ class ConnectorException(HTTPError): """when the connector can't do what was asked""" -async def get_results(session, url, min_confidence, query, connector): - """try this specific connector""" - # pylint: disable=line-too-long - headers = { - "Accept": ( - 'application/json, application/activity+json, application/ld+json; profile="https://www.w3.org/ns/activitystreams"; charset=utf-8' - ), - "User-Agent": USER_AGENT, - } - params = {"min_confidence": min_confidence} - try: - async with session.get(url, headers=headers, params=params) as response: - if not response.ok: - logger.info("Unable to connect to %s: %s", url, response.reason) - return - - try: - raw_data = await response.json() - except aiohttp.client_exceptions.ContentTypeError as err: - logger.exception(err) - return - - return { - "connector": connector, - "results": connector.process_search_response( - query, raw_data, min_confidence - ), - } - except asyncio.TimeoutError: - logger.info("Connection timed out for url: %s", url) - except aiohttp.ClientError as err: - logger.info(err) - - -async def async_connector_search(query, items, min_confidence): +async def async_connector_search( + query: str, + items: list[tuple[str, abstract_connector.AbstractConnector]], + min_confidence: float, +) -> list[Optional[abstract_connector.ConnectorResults]]: """Try a number of requests simultaneously""" timeout = aiohttp.ClientTimeout(total=SEARCH_TIMEOUT) async with aiohttp.ClientSession(timeout=timeout) as session: - tasks = [] + tasks: list[Future[Optional[abstract_connector.ConnectorResults]]] = [] for url, connector in items: tasks.append( asyncio.ensure_future( - get_results(session, url, min_confidence, query, connector) + connector.get_results(session, url, min_confidence, query) ) ) results = await asyncio.gather(*tasks) - return results + return list(results) -def search(query, min_confidence=0.1, return_first=False): - """find books based on arbitary keywords""" +@overload +def search( + query: str, *, min_confidence: float = 0.1, return_first: Literal[False] +) -> list[abstract_connector.ConnectorResults]: + ... + + +@overload +def search( + query: str, *, min_confidence: float = 0.1, return_first: Literal[True] +) -> Optional[SearchResult]: + ... + + +def search( + query: str, *, min_confidence: float = 0.1, return_first: bool = False +) -> Union[list[abstract_connector.ConnectorResults], Optional[SearchResult]]: + """find books based on arbitrary keywords""" if not query: - return [] - results = [] + return None if return_first else [] items = [] for connector in get_connectors(): @@ -91,8 +81,12 @@ def search(query, min_confidence=0.1, return_first=False): items.append((url, connector)) # load as many results as we can - results = asyncio.run(async_connector_search(query, items, min_confidence)) - results = [r for r in results if r] + # failed requests will return None, so filter those out + results = [ + r + for r in asyncio.run(async_connector_search(query, items, min_confidence)) + if r + ] if return_first: # find the best result from all the responses and return that @@ -100,11 +94,12 @@ def search(query, min_confidence=0.1, return_first=False): all_results = sorted(all_results, key=lambda r: r.confidence, reverse=True) return all_results[0] if all_results else None - # failed requests will return None, so filter those out return results -def first_search_result(query, min_confidence=0.1): +def first_search_result( + query: str, min_confidence: float = 0.1 +) -> Union[models.Edition, SearchResult, None]: """search until you find a result that fits""" # try local search first result = book_search.search(query, min_confidence=min_confidence, return_first=True) @@ -114,13 +109,13 @@ def first_search_result(query, min_confidence=0.1): return search(query, min_confidence=min_confidence, return_first=True) or None -def get_connectors(): +def get_connectors() -> Iterator[abstract_connector.AbstractConnector]: """load all connectors""" for info in models.Connector.objects.filter(active=True).order_by("priority").all(): yield load_connector(info) -def get_or_create_connector(remote_id): +def get_or_create_connector(remote_id: str) -> abstract_connector.AbstractConnector: """get the connector related to the object's server""" url = urlparse(remote_id) identifier = url.netloc @@ -143,8 +138,8 @@ def get_or_create_connector(remote_id): return load_connector(connector_info) -@app.task(queue=LOW) -def load_more_data(connector_id, book_id): +@app.task(queue=CONNECTORS) +def load_more_data(connector_id: str, book_id: str) -> None: """background the work of getting all 10,000 editions of LoTR""" connector_info = models.Connector.objects.get(id=connector_id) connector = load_connector(connector_info) @@ -152,8 +147,10 @@ def load_more_data(connector_id, book_id): connector.expand_book_data(book) -@app.task(queue=LOW) -def create_edition_task(connector_id, work_id, data): +@app.task(queue=CONNECTORS) +def create_edition_task( + connector_id: int, work_id: int, data: Union[str, abstract_connector.JsonDict] +) -> None: """separate task for each of the 10,000 editions of LoTR""" connector_info = models.Connector.objects.get(id=connector_id) connector = load_connector(connector_info) @@ -161,23 +158,31 @@ def create_edition_task(connector_id, work_id, data): connector.create_edition_from_data(work, data) -def load_connector(connector_info): +def load_connector( + connector_info: models.Connector, +) -> abstract_connector.AbstractConnector: """instantiate the connector class""" connector = importlib.import_module( f"bookwyrm.connectors.{connector_info.connector_file}" ) - return connector.Connector(connector_info.identifier) + return connector.Connector(connector_info.identifier) # type: ignore[no-any-return] @receiver(signals.post_save, sender="bookwyrm.FederatedServer") # pylint: disable=unused-argument -def create_connector(sender, instance, created, *args, **kwargs): +def create_connector( + sender: Any, + instance: models.FederatedServer, + created: Any, + *args: Any, + **kwargs: Any, +) -> None: """create a connector to an external bookwyrm server""" if instance.application_type == "bookwyrm": get_or_create_connector(f"https://{instance.server_name}") -def raise_not_valid_url(url): +def raise_not_valid_url(url: str) -> None: """do some basic reality checks on the url""" parsed = urlparse(url) if not parsed.scheme in ["http", "https"]: diff --git a/bookwyrm/connectors/inventaire.py b/bookwyrm/connectors/inventaire.py index df9b2e43a..c08bcdee1 100644 --- a/bookwyrm/connectors/inventaire.py +++ b/bookwyrm/connectors/inventaire.py @@ -1,9 +1,10 @@ """ inventaire data connector """ import re +from typing import Any, Union, Optional, Iterator, Iterable from bookwyrm import models from bookwyrm.book_search import SearchResult -from .abstract_connector import AbstractConnector, Mapping +from .abstract_connector import AbstractConnector, Mapping, JsonDict from .abstract_connector import get_data from .connector_manager import ConnectorException, create_edition_task @@ -13,7 +14,7 @@ class Connector(AbstractConnector): generated_remote_link_field = "inventaire_id" - def __init__(self, identifier): + def __init__(self, identifier: str): super().__init__(identifier) get_first = lambda a: a[0] @@ -60,13 +61,13 @@ class Connector(AbstractConnector): Mapping("died", remote_field="wdt:P570", formatter=get_first), ] + shared_mappings - def get_remote_id(self, value): + def get_remote_id(self, value: str) -> str: """convert an id/uri into a url""" return f"{self.books_url}?action=by-uris&uris={value}" - def get_book_data(self, remote_id): + def get_book_data(self, remote_id: str) -> JsonDict: data = get_data(remote_id) - extracted = list(data.get("entities").values()) + extracted = list(data.get("entities", {}).values()) try: data = extracted[0] except (KeyError, IndexError): @@ -74,10 +75,16 @@ class Connector(AbstractConnector): # flatten the data so that images, uri, and claims are on the same level return { **data.get("claims", {}), - **{k: data.get(k) for k in ["uri", "image", "labels", "sitelinks", "type"]}, + **{ + k: data.get(k) + for k in ["uri", "image", "labels", "sitelinks", "type"] + if k in data + }, } - def parse_search_data(self, data, min_confidence): + def parse_search_data( + self, data: JsonDict, min_confidence: float + ) -> Iterator[SearchResult]: for search_result in data.get("results", []): images = search_result.get("image") cover = f"{self.covers_url}/img/entities/{images[0]}" if images else None @@ -96,8 +103,8 @@ class Connector(AbstractConnector): connector=self, ) - def parse_isbn_search_data(self, data): - """got some daaaata""" + def parse_isbn_search_data(self, data: JsonDict) -> Iterator[SearchResult]: + """got some data""" results = data.get("entities") if not results: return @@ -114,35 +121,44 @@ class Connector(AbstractConnector): connector=self, ) - def is_work_data(self, data): + def is_work_data(self, data: JsonDict) -> bool: return data.get("type") == "work" - def load_edition_data(self, work_uri): + def load_edition_data(self, work_uri: str) -> JsonDict: """get a list of editions for a work""" # pylint: disable=line-too-long url = f"{self.books_url}?action=reverse-claims&property=wdt:P629&value={work_uri}&sort=true" return get_data(url) - def get_edition_from_work_data(self, data): - data = self.load_edition_data(data.get("uri")) + def get_edition_from_work_data(self, data: JsonDict) -> JsonDict: + work_uri = data.get("uri") + if not work_uri: + raise ConnectorException("Invalid URI") + data = self.load_edition_data(work_uri) try: uri = data.get("uris", [])[0] except IndexError: raise ConnectorException("Invalid book data") return self.get_book_data(self.get_remote_id(uri)) - def get_work_from_edition_data(self, data): - uri = data.get("wdt:P629", [None])[0] + def get_work_from_edition_data(self, data: JsonDict) -> JsonDict: + try: + uri = data.get("wdt:P629", [])[0] + except IndexError: + raise ConnectorException("Invalid book data") + if not uri: raise ConnectorException("Invalid book data") return self.get_book_data(self.get_remote_id(uri)) - def get_authors_from_data(self, data): + def get_authors_from_data(self, data: JsonDict) -> Iterator[models.Author]: authors = data.get("wdt:P50", []) for author in authors: - yield self.get_or_create_author(self.get_remote_id(author)) + model = self.get_or_create_author(self.get_remote_id(author)) + if model: + yield model - def expand_book_data(self, book): + def expand_book_data(self, book: models.Book) -> None: work = book # go from the edition to the work, if necessary if isinstance(book, models.Edition): @@ -154,36 +170,45 @@ class Connector(AbstractConnector): # who knows, man return - for edition_uri in edition_options.get("uris"): + for edition_uri in edition_options.get("uris", []): remote_id = self.get_remote_id(edition_uri) create_edition_task.delay(self.connector.id, work.id, remote_id) - def create_edition_from_data(self, work, edition_data, instance=None): + def create_edition_from_data( + self, + work: models.Work, + edition_data: Union[str, JsonDict], + instance: Optional[models.Edition] = None, + ) -> Optional[models.Edition]: """pass in the url as data and then call the version in abstract connector""" if isinstance(edition_data, str): try: edition_data = self.get_book_data(edition_data) except ConnectorException: # who, indeed, knows - return - super().create_edition_from_data(work, edition_data, instance=instance) + return None + return super().create_edition_from_data(work, edition_data, instance=instance) - def get_cover_url(self, cover_blob, *_): + def get_cover_url( + self, cover_blob: Union[list[JsonDict], JsonDict], *_: Any + ) -> Optional[str]: """format the relative cover url into an absolute one: {"url": "/img/entities/e794783f01b9d4f897a1ea9820b96e00d346994f"} """ # covers may or may not be a list - if isinstance(cover_blob, list) and len(cover_blob) > 0: + if isinstance(cover_blob, list): + if len(cover_blob) == 0: + return None cover_blob = cover_blob[0] cover_id = cover_blob.get("url") - if not cover_id: + if not isinstance(cover_id, str): return None # cover may or may not be an absolute url already if re.match(r"^http", cover_id): return cover_id return f"{self.covers_url}{cover_id}" - def resolve_keys(self, keys): + def resolve_keys(self, keys: Iterable[str]) -> list[str]: """cool, it's "wd:Q3156592" now what the heck does that mean""" results = [] for uri in keys: @@ -191,10 +216,10 @@ class Connector(AbstractConnector): data = self.get_book_data(self.get_remote_id(uri)) except ConnectorException: continue - results.append(get_language_code(data.get("labels"))) + results.append(get_language_code(data.get("labels", {}))) return results - def get_description(self, links): + def get_description(self, links: JsonDict) -> str: """grab an extracted excerpt from wikipedia""" link = links.get("enwiki") if not link: @@ -204,15 +229,15 @@ class Connector(AbstractConnector): data = get_data(url) except ConnectorException: return "" - return data.get("extract") + return data.get("extract", "") - def get_remote_id_from_model(self, obj): + def get_remote_id_from_model(self, obj: models.BookDataModel) -> str: """use get_remote_id to figure out the link from a model obj""" remote_id_value = obj.inventaire_id return self.get_remote_id(remote_id_value) -def get_language_code(options, code="en"): +def get_language_code(options: JsonDict, code: str = "en") -> Any: """when there are a bunch of translation but we need a single field""" result = options.get(code) if result: diff --git a/bookwyrm/connectors/openlibrary.py b/bookwyrm/connectors/openlibrary.py index 0fd786660..4dc6d6ac1 100644 --- a/bookwyrm/connectors/openlibrary.py +++ b/bookwyrm/connectors/openlibrary.py @@ -1,9 +1,13 @@ """ openlibrary data connector """ import re +from typing import Any, Optional, Union, Iterator, Iterable + +from markdown import markdown from bookwyrm import models from bookwyrm.book_search import SearchResult -from .abstract_connector import AbstractConnector, Mapping +from bookwyrm.utils.sanitizer import clean +from .abstract_connector import AbstractConnector, Mapping, JsonDict from .abstract_connector import get_data, infer_physical_format, unique_physical_format from .connector_manager import ConnectorException, create_edition_task from .openlibrary_languages import languages @@ -14,7 +18,7 @@ class Connector(AbstractConnector): generated_remote_link_field = "openlibrary_link" - def __init__(self, identifier): + def __init__(self, identifier: str): super().__init__(identifier) get_first = lambda a, *args: a[0] @@ -94,14 +98,14 @@ class Connector(AbstractConnector): Mapping("inventaire_id", remote_field="links", formatter=get_inventaire_id), ] - def get_book_data(self, remote_id): + def get_book_data(self, remote_id: str) -> JsonDict: data = get_data(remote_id) if data.get("type", {}).get("key") == "/type/redirect": - remote_id = self.base_url + data.get("location") + remote_id = self.base_url + data.get("location", "") return get_data(remote_id) return data - def get_remote_id_from_data(self, data): + def get_remote_id_from_data(self, data: JsonDict) -> str: """format a url from an openlibrary id field""" try: key = data["key"] @@ -109,10 +113,10 @@ class Connector(AbstractConnector): raise ConnectorException("Invalid book data") return f"{self.books_url}{key}" - def is_work_data(self, data): + def is_work_data(self, data: JsonDict) -> bool: return bool(re.match(r"^[\/\w]+OL\d+W$", data["key"])) - def get_edition_from_work_data(self, data): + def get_edition_from_work_data(self, data: JsonDict) -> JsonDict: try: key = data["key"] except KeyError: @@ -124,7 +128,7 @@ class Connector(AbstractConnector): raise ConnectorException("No editions for work") return edition - def get_work_from_edition_data(self, data): + def get_work_from_edition_data(self, data: JsonDict) -> JsonDict: try: key = data["works"][0]["key"] except (IndexError, KeyError): @@ -132,7 +136,7 @@ class Connector(AbstractConnector): url = f"{self.books_url}{key}" return self.get_book_data(url) - def get_authors_from_data(self, data): + def get_authors_from_data(self, data: JsonDict) -> Iterator[models.Author]: """parse author json and load or create authors""" for author_blob in data.get("authors", []): author_blob = author_blob.get("author", author_blob) @@ -144,7 +148,7 @@ class Connector(AbstractConnector): continue yield author - def get_cover_url(self, cover_blob, size="L"): + def get_cover_url(self, cover_blob: list[str], size: str = "L") -> Optional[str]: """ask openlibrary for the cover""" if not cover_blob: return None @@ -152,8 +156,10 @@ class Connector(AbstractConnector): image_name = f"{cover_id}-{size}.jpg" return f"{self.covers_url}/b/id/{image_name}" - def parse_search_data(self, data, min_confidence): - for idx, search_result in enumerate(data.get("docs")): + def parse_search_data( + self, data: JsonDict, min_confidence: float + ) -> Iterator[SearchResult]: + for idx, search_result in enumerate(data.get("docs", [])): # build the remote id from the openlibrary key key = self.books_url + search_result["key"] author = search_result.get("author_name") or ["Unknown"] @@ -174,7 +180,7 @@ class Connector(AbstractConnector): confidence=confidence, ) - def parse_isbn_search_data(self, data): + def parse_isbn_search_data(self, data: JsonDict) -> Iterator[SearchResult]: for search_result in list(data.values()): # build the remote id from the openlibrary key key = self.books_url + search_result["key"] @@ -188,12 +194,12 @@ class Connector(AbstractConnector): year=search_result.get("publish_date"), ) - def load_edition_data(self, olkey): + def load_edition_data(self, olkey: str) -> JsonDict: """query openlibrary for editions of a work""" url = f"{self.books_url}/works/{olkey}/editions" return self.get_book_data(url) - def expand_book_data(self, book): + def expand_book_data(self, book: models.Book) -> None: work = book # go from the edition to the work, if necessary if isinstance(book, models.Edition): @@ -206,14 +212,14 @@ class Connector(AbstractConnector): # who knows, man return - for edition_data in edition_options.get("entries"): + for edition_data in edition_options.get("entries", []): # does this edition have ANY interesting data? if ignore_edition(edition_data): continue create_edition_task.delay(self.connector.id, work.id, edition_data) -def ignore_edition(edition_data): +def ignore_edition(edition_data: JsonDict) -> bool: """don't load a million editions that have no metadata""" # an isbn, we love to see it if edition_data.get("isbn_13") or edition_data.get("isbn_10"): @@ -232,19 +238,30 @@ def ignore_edition(edition_data): return True -def get_description(description_blob): +def get_description(description_blob: Union[JsonDict, str]) -> str: """descriptions can be a string or a dict""" if isinstance(description_blob, dict): - return description_blob.get("value") - return description_blob + description = markdown(description_blob.get("value", "")) + else: + description = markdown(description_blob) + + if ( + description.startswith("") + and description.endswith("
") + and description.count("") == 1 + ): + # If there is just one
tag and it is around the text remove it + return description[len("
") : -len("
")].strip() + + return clean(description) -def get_openlibrary_key(key): +def get_openlibrary_key(key: str) -> str: """convert /books/OL27320736M into OL27320736M""" return key.split("/")[-1] -def get_languages(language_blob): +def get_languages(language_blob: Iterable[JsonDict]) -> list[Optional[str]]: """/language/eng -> English""" langs = [] for lang in language_blob: @@ -252,14 +269,14 @@ def get_languages(language_blob): return langs -def get_dict_field(blob, field_name): +def get_dict_field(blob: Optional[JsonDict], field_name: str) -> Optional[Any]: """extract the isni from the remote id data for the author""" if not blob or not isinstance(blob, dict): return None return blob.get(field_name) -def get_wikipedia_link(links): +def get_wikipedia_link(links: list[Any]) -> Optional[str]: """extract wikipedia links""" if not isinstance(links, list): return None @@ -272,7 +289,7 @@ def get_wikipedia_link(links): return None -def get_inventaire_id(links): +def get_inventaire_id(links: list[Any]) -> Optional[str]: """extract and format inventaire ids""" if not isinstance(links, list): return None @@ -282,11 +299,13 @@ def get_inventaire_id(links): continue if link.get("title") == "inventaire.io": iv_link = link.get("url") + if not isinstance(iv_link, str): + return None return iv_link.split("/")[-1] return None -def pick_default_edition(options): +def pick_default_edition(options: list[JsonDict]) -> Optional[JsonDict]: """favor physical copies with covers in english""" if not options: return None diff --git a/bookwyrm/emailing.py b/bookwyrm/emailing.py index e767d5374..5e08ebba1 100644 --- a/bookwyrm/emailing.py +++ b/bookwyrm/emailing.py @@ -3,7 +3,7 @@ from django.core.mail import EmailMultiAlternatives from django.template.loader import get_template from bookwyrm import models, settings -from bookwyrm.tasks import app, HIGH +from bookwyrm.tasks import app, EMAIL from bookwyrm.settings import DOMAIN @@ -18,6 +18,12 @@ def email_data(): } +def test_email(user): + """Just an admin checking if emails are sending""" + data = email_data() + send_email(user.email, *format_email("test", data)) + + def email_confirmation_email(user): """newly registered users confirm email address""" data = email_data() @@ -38,7 +44,7 @@ def password_reset_email(reset_code): data = email_data() data["reset_link"] = reset_code.link data["user"] = reset_code.user.display_name - send_email.delay(reset_code.user.email, *format_email("password_reset", data)) + send_email(reset_code.user.email, *format_email("password_reset", data)) def moderation_report_email(report): @@ -48,6 +54,7 @@ def moderation_report_email(report): if report.user: data["reportee"] = report.user.localname or report.user.username data["report_link"] = report.remote_id + data["link_domain"] = report.links.exists() for admin in models.User.objects.filter( groups__name__in=["admin", "moderator"] @@ -68,7 +75,7 @@ def format_email(email_name, data): return (subject, html_content, text_content) -@app.task(queue=HIGH) +@app.task(queue=EMAIL) def send_email(recipient, subject, html_content, text_content): """use a task to send the email""" email = EmailMultiAlternatives( diff --git a/bookwyrm/forms/admin.py b/bookwyrm/forms/admin.py index ae15e011b..72f50ccb8 100644 --- a/bookwyrm/forms/admin.py +++ b/bookwyrm/forms/admin.py @@ -15,7 +15,7 @@ from .custom_form import CustomForm, StyledForm # pylint: disable=missing-class-docstring class ExpiryWidget(widgets.Select): def value_from_datadict(self, data, files, name): - """human-readable exiration time buckets""" + """human-readable expiration time buckets""" selected_string = super().value_from_datadict(data, files, name) if selected_string == "day": @@ -55,11 +55,46 @@ class CreateInviteForm(CustomForm): class SiteForm(CustomForm): class Meta: model = models.SiteSettings - exclude = ["admin_code", "install_mode"] + fields = [ + "name", + "instance_tagline", + "instance_description", + "instance_short_description", + "default_theme", + "code_of_conduct", + "privacy_policy", + "impressum", + "show_impressum", + "logo", + "logo_small", + "favicon", + "support_link", + "support_title", + "admin_email", + "footer_item", + ] widgets = { "instance_short_description": forms.TextInput( attrs={"aria-describedby": "desc_instance_short_description"} ), + } + + +class RegistrationForm(CustomForm): + class Meta: + model = models.SiteSettings + fields = [ + "allow_registration", + "allow_invite_requests", + "registration_closed_text", + "invite_request_text", + "invite_request_question", + "invite_question_text", + "require_confirm_email", + "default_user_auth_group", + ] + + widgets = { "require_confirm_email": forms.CheckboxInput( attrs={"aria-describedby": "desc_require_confirm_email"} ), @@ -69,6 +104,23 @@ class SiteForm(CustomForm): } +class RegistrationLimitedForm(CustomForm): + class Meta: + model = models.SiteSettings + fields = [ + "registration_closed_text", + "invite_request_text", + "invite_request_question", + "invite_question_text", + ] + + widgets = { + "invite_request_text": forms.Textarea( + attrs={"aria-describedby": "desc_invite_request_text"} + ), + } + + class ThemeForm(CustomForm): class Meta: model = models.Theme diff --git a/bookwyrm/forms/author.py b/bookwyrm/forms/author.py index ca59426de..5b54a07b5 100644 --- a/bookwyrm/forms/author.py +++ b/bookwyrm/forms/author.py @@ -15,12 +15,14 @@ class AuthorForm(CustomForm): "aliases", "bio", "wikipedia_link", + "website", "born", "died", "openlibrary_key", "inventaire_id", "librarything_key", "goodreads_key", + "isfdb", "isni", ] widgets = { @@ -30,10 +32,11 @@ class AuthorForm(CustomForm): "wikipedia_link": forms.TextInput( attrs={"aria-describedby": "desc_wikipedia_link"} ), + "website": forms.TextInput(attrs={"aria-describedby": "desc_website"}), "born": forms.SelectDateWidget(attrs={"aria-describedby": "desc_born"}), "died": forms.SelectDateWidget(attrs={"aria-describedby": "desc_died"}), - "oepnlibrary_key": forms.TextInput( - attrs={"aria-describedby": "desc_oepnlibrary_key"} + "openlibrary_key": forms.TextInput( + attrs={"aria-describedby": "desc_openlibrary_key"} ), "inventaire_id": forms.TextInput( attrs={"aria-describedby": "desc_inventaire_id"} diff --git a/bookwyrm/forms/books.py b/bookwyrm/forms/books.py index 9b3c84010..4885dc063 100644 --- a/bookwyrm/forms/books.py +++ b/bookwyrm/forms/books.py @@ -18,22 +18,37 @@ class CoverForm(CustomForm): class EditionForm(CustomForm): class Meta: model = models.Edition - exclude = [ - "remote_id", - "origin_id", - "created_date", - "updated_date", - "edition_rank", - "authors", - "parent_work", - "shelves", - "connector", - "search_vector", - "links", - "file_links", + fields = [ + "title", + "sort_title", + "subtitle", + "description", + "series", + "series_number", + "languages", + "subjects", + "publishers", + "first_published_date", + "published_date", + "cover", + "physical_format", + "physical_format_detail", + "pages", + "isbn_13", + "isbn_10", + "openlibrary_key", + "inventaire_id", + "goodreads_key", + "oclc_number", + "asin", + "aasin", + "isfdb", ] widgets = { "title": forms.TextInput(attrs={"aria-describedby": "desc_title"}), + "sort_title": forms.TextInput( + attrs={"aria-describedby": "desc_sort_title"} + ), "subtitle": forms.TextInput(attrs={"aria-describedby": "desc_subtitle"}), "description": forms.Textarea( attrs={"aria-describedby": "desc_description"} @@ -73,10 +88,15 @@ class EditionForm(CustomForm): "inventaire_id": forms.TextInput( attrs={"aria-describedby": "desc_inventaire_id"} ), + "goodreads_key": forms.TextInput( + attrs={"aria-describedby": "desc_goodreads_key"} + ), "oclc_number": forms.TextInput( attrs={"aria-describedby": "desc_oclc_number"} ), "ASIN": forms.TextInput(attrs={"aria-describedby": "desc_ASIN"}), + "AASIN": forms.TextInput(attrs={"aria-describedby": "desc_AASIN"}), + "isfdb": forms.TextInput(attrs={"aria-describedby": "desc_isfdb"}), } @@ -91,6 +111,7 @@ class EditionFromWorkForm(CustomForm): model = models.Work fields = [ "title", + "sort_title", "subtitle", "authors", "description", diff --git a/bookwyrm/forms/landing.py b/bookwyrm/forms/landing.py index bd9884bc3..1da4fc4f1 100644 --- a/bookwyrm/forms/landing.py +++ b/bookwyrm/forms/landing.py @@ -8,6 +8,7 @@ import pyotp from bookwyrm import models from bookwyrm.settings import DOMAIN +from bookwyrm.settings import TWO_FACTOR_LOGIN_VALIDITY_WINDOW from .custom_form import CustomForm @@ -108,7 +109,7 @@ class Confirm2FAForm(CustomForm): otp = self.data.get("otp") totp = pyotp.TOTP(self.instance.otp_secret) - if not totp.verify(otp): + if not totp.verify(otp, valid_window=TWO_FACTOR_LOGIN_VALIDITY_WINDOW): if self.instance.hotp_secret: # maybe it's a backup code? diff --git a/bookwyrm/forms/links.py b/bookwyrm/forms/links.py index de229bc2d..d2fd5f116 100644 --- a/bookwyrm/forms/links.py +++ b/bookwyrm/forms/links.py @@ -36,13 +36,16 @@ class FileLinkForm(CustomForm): "This domain is blocked. Please contact your administrator if you think this is an error." ), ) - elif models.FileLink.objects.filter( + if ( + not self.instance + and models.FileLink.objects.filter( url=url, book=book, filetype=filetype - ).exists(): - # pylint: disable=line-too-long - self.add_error( - "url", - _( - "This link with file type has already been added for this book. If it is not visible, the domain is still pending." - ), - ) + ).exists() + ): + # pylint: disable=line-too-long + self.add_error( + "url", + _( + "This link with file type has already been added for this book. If it is not visible, the domain is still pending." + ), + ) diff --git a/bookwyrm/forms/lists.py b/bookwyrm/forms/lists.py index 647db3bfe..f5008baa3 100644 --- a/bookwyrm/forms/lists.py +++ b/bookwyrm/forms/lists.py @@ -24,7 +24,7 @@ class SortListForm(forms.Form): sort_by = ChoiceField( choices=( ("order", _("List Order")), - ("title", _("Book Title")), + ("sort_title", _("Book Title")), ("rating", _("Rating")), ), label=_("Sort By"), diff --git a/bookwyrm/forms/status.py b/bookwyrm/forms/status.py index 0800166bf..b562595ee 100644 --- a/bookwyrm/forms/status.py +++ b/bookwyrm/forms/status.py @@ -53,6 +53,7 @@ class QuotationForm(CustomForm): "sensitive", "privacy", "position", + "endposition", "position_mode", ] diff --git a/bookwyrm/importers/calibre_import.py b/bookwyrm/importers/calibre_import.py index 5426e9333..5c22a539d 100644 --- a/bookwyrm/importers/calibre_import.py +++ b/bookwyrm/importers/calibre_import.py @@ -1,4 +1,6 @@ """ handle reading a csv from calibre """ +from typing import Any, Optional + from bookwyrm.models import Shelf from . import Importer @@ -9,7 +11,7 @@ class CalibreImporter(Importer): service = "Calibre" - def __init__(self, *args, **kwargs): + def __init__(self, *args: Any, **kwargs: Any): # Add timestamp to row_mappings_guesses for date_added to avoid # integrity error row_mappings_guesses = [] @@ -23,6 +25,6 @@ class CalibreImporter(Importer): self.row_mappings_guesses = row_mappings_guesses super().__init__(*args, **kwargs) - def get_shelf(self, normalized_row): + def get_shelf(self, normalized_row: dict[str, Optional[str]]) -> Optional[str]: # Calibre export does not indicate which shelf to use. Use a default one for now return Shelf.TO_READ diff --git a/bookwyrm/importers/importer.py b/bookwyrm/importers/importer.py index a3cfba198..5b3192fa5 100644 --- a/bookwyrm/importers/importer.py +++ b/bookwyrm/importers/importer.py @@ -1,7 +1,10 @@ """ handle reading a csv from an external service, defaults are from Goodreads """ import csv +from datetime import timedelta +from typing import Iterable, Optional + from django.utils import timezone -from bookwyrm.models import ImportJob, ImportItem +from bookwyrm.models import ImportJob, ImportItem, SiteSettings, User class Importer: @@ -16,8 +19,8 @@ class Importer: ("id", ["id", "book id"]), ("title", ["title"]), ("authors", ["author", "authors", "primary author"]), - ("isbn_10", ["isbn10", "isbn"]), - ("isbn_13", ["isbn13", "isbn", "isbns"]), + ("isbn_10", ["isbn10", "isbn", "isbn/uid"]), + ("isbn_13", ["isbn13", "isbn", "isbns", "isbn/uid"]), ("shelf", ["shelf", "exclusive shelf", "read status", "bookshelf"]), ("review_name", ["review name"]), ("review_body", ["my review", "review"]), @@ -33,26 +36,48 @@ class Importer: "reading": ["currently-reading", "reading", "currently reading"], } - def create_job(self, user, csv_file, include_reviews, privacy): + # pylint: disable=too-many-locals + def create_job( + self, user: User, csv_file: Iterable[str], include_reviews: bool, privacy: str + ) -> ImportJob: """check over a csv and creates a database entry for the job""" csv_reader = csv.DictReader(csv_file, delimiter=self.delimiter) - rows = enumerate(list(csv_reader)) + rows = list(csv_reader) + if len(rows) < 1: + raise ValueError("CSV file is empty") + + mappings = ( + self.create_row_mappings(list(fieldnames)) + if (fieldnames := csv_reader.fieldnames) + else {} + ) + job = ImportJob.objects.create( user=user, include_reviews=include_reviews, privacy=privacy, - mappings=self.create_row_mappings(csv_reader.fieldnames), + mappings=mappings, source=self.service, ) - for index, entry in rows: + enforce_limit, allowed_imports = self.get_import_limit(user) + if enforce_limit and allowed_imports <= 0: + job.complete_job() + return job + for index, entry in enumerate(rows): + if enforce_limit and index >= allowed_imports: + break self.create_item(job, index, entry) return job - def update_legacy_job(self, job): + def update_legacy_job(self, job: ImportJob) -> None: """patch up a job that was in the old format""" items = job.items - headers = list(items.first().data.keys()) + first_item = items.first() + if first_item is None: + return + + headers = list(first_item.data.keys()) job.mappings = self.create_row_mappings(headers) job.updated_date = timezone.now() job.save() @@ -63,24 +88,24 @@ class Importer: item.normalized_data = normalized item.save() - def create_row_mappings(self, headers): + def create_row_mappings(self, headers: list[str]) -> dict[str, Optional[str]]: """guess what the headers mean""" mappings = {} for (key, guesses) in self.row_mappings_guesses: - value = [h for h in headers if h.lower() in guesses] - value = value[0] if len(value) else None + values = [h for h in headers if h.lower() in guesses] + value = values[0] if len(values) else None if value: headers.remove(value) mappings[key] = value return mappings - def create_item(self, job, index, data): + def create_item(self, job: ImportJob, index: int, data: dict[str, str]) -> None: """creates and saves an import item""" normalized = self.normalize_row(data, job.mappings) normalized["shelf"] = self.get_shelf(normalized) ImportItem(job=job, index=index, data=data, normalized_data=normalized).save() - def get_shelf(self, normalized_row): + def get_shelf(self, normalized_row: dict[str, Optional[str]]) -> Optional[str]: """determine which shelf to use""" shelf_name = normalized_row.get("shelf") if not shelf_name: @@ -91,11 +116,35 @@ class Importer: ] return shelf[0] if shelf else None - def normalize_row(self, entry, mappings): # pylint: disable=no-self-use + # pylint: disable=no-self-use + def normalize_row( + self, entry: dict[str, str], mappings: dict[str, Optional[str]] + ) -> dict[str, Optional[str]]: """use the dataclass to create the formatted row of data""" - return {k: entry.get(v) for k, v in mappings.items()} + return {k: entry.get(v) if v else None for k, v in mappings.items()} - def create_retry_job(self, user, original_job, items): + # pylint: disable=no-self-use + def get_import_limit(self, user: User) -> tuple[int, int]: + """check if import limit is set and return how many imports are left""" + site_settings = SiteSettings.objects.get() + import_size_limit = site_settings.import_size_limit + import_limit_reset = site_settings.import_limit_reset + enforce_limit = import_size_limit and import_limit_reset + allowed_imports = 0 + + if enforce_limit: + time_range = timezone.now() - timedelta(days=import_limit_reset) + import_jobs = ImportJob.objects.filter( + user=user, created_date__gte=time_range + ) + # pylint: disable=consider-using-generator + imported_books = sum([job.successful_item_count for job in import_jobs]) + allowed_imports = import_size_limit - imported_books + return enforce_limit, allowed_imports + + def create_retry_job( + self, user: User, original_job: ImportJob, items: list[ImportItem] + ) -> ImportJob: """retry items that didn't import""" job = ImportJob.objects.create( user=user, @@ -106,7 +155,13 @@ class Importer: mappings=original_job.mappings, retry=True, ) - for item in items: + enforce_limit, allowed_imports = self.get_import_limit(user) + if enforce_limit and allowed_imports <= 0: + job.complete_job() + return job + for index, item in enumerate(items): + if enforce_limit and index >= allowed_imports: + break # this will re-normalize the raw data self.create_item(job, item.index, item.data) return job diff --git a/bookwyrm/importers/librarything_import.py b/bookwyrm/importers/librarything_import.py index c6833547d..145657ba0 100644 --- a/bookwyrm/importers/librarything_import.py +++ b/bookwyrm/importers/librarything_import.py @@ -1,11 +1,16 @@ """ handle reading a tsv from librarything """ import re +from typing import Optional from bookwyrm.models import Shelf from . import Importer +def _remove_brackets(value: Optional[str]) -> Optional[str]: + return re.sub(r"\[|\]", "", value) if value else None + + class LibrarythingImporter(Importer): """csv downloads from librarything""" @@ -13,16 +18,19 @@ class LibrarythingImporter(Importer): delimiter = "\t" encoding = "ISO-8859-1" - def normalize_row(self, entry, mappings): # pylint: disable=no-self-use + def normalize_row( + self, entry: dict[str, str], mappings: dict[str, Optional[str]] + ) -> dict[str, Optional[str]]: # pylint: disable=no-self-use """use the dataclass to create the formatted row of data""" - remove_brackets = lambda v: re.sub(r"\[|\]", "", v) if v else None - normalized = {k: remove_brackets(entry.get(v)) for k, v in mappings.items()} - isbn_13 = normalized.get("isbn_13") - isbn_13 = isbn_13.split(", ") if isbn_13 else [] - normalized["isbn_13"] = isbn_13[1] if len(isbn_13) > 0 else None + normalized = { + k: _remove_brackets(entry.get(v) if v else None) + for k, v in mappings.items() + } + isbn_13 = value.split(", ") if (value := normalized.get("isbn_13")) else [] + normalized["isbn_13"] = isbn_13[1] if len(isbn_13) > 1 else None return normalized - def get_shelf(self, normalized_row): + def get_shelf(self, normalized_row: dict[str, Optional[str]]) -> Optional[str]: if normalized_row["date_finished"]: return Shelf.READ_FINISHED if normalized_row["date_started"]: diff --git a/bookwyrm/importers/openlibrary_import.py b/bookwyrm/importers/openlibrary_import.py index ef1030609..6a954ed3c 100644 --- a/bookwyrm/importers/openlibrary_import.py +++ b/bookwyrm/importers/openlibrary_import.py @@ -1,4 +1,6 @@ """ handle reading a csv from openlibrary""" +from typing import Any + from . import Importer @@ -7,7 +9,7 @@ class OpenLibraryImporter(Importer): service = "OpenLibrary" - def __init__(self, *args, **kwargs): + def __init__(self, *args: Any, **kwargs: Any): self.row_mappings_guesses.append(("openlibrary_key", ["edition id"])) self.row_mappings_guesses.append(("openlibrary_work_key", ["work id"])) super().__init__(*args, **kwargs) diff --git a/bookwyrm/isbn/RangeMessage.xml b/bookwyrm/isbn/RangeMessage.xml new file mode 100644 index 000000000..619cf1ff7 --- /dev/null +++ b/bookwyrm/isbn/RangeMessage.xml @@ -0,0 +1,7904 @@ + + + + + + + + + + + + + + + +]> +(comment on ' - f'"{self.book.title}", page {self.progress})
' - ) - else: - return_value = ( - f'{self.content}(comment on ' - f'"{self.book.title}")
' - ) - return return_value + progress = self.progress or 0 + citation = ( + f'comment on ' + f"{self.book.title}" + ) + if self.progress_mode == "PG" and progress > 0: + citation += f", p. {progress}" + return f"{self.content}({citation})
" activity_serializer = activitypub.Comment @@ -327,6 +341,9 @@ class Quotation(BookStatus): position = models.IntegerField( validators=[MinValueValidator(0)], null=True, blank=True ) + endposition = models.IntegerField( + validators=[MinValueValidator(0)], null=True, blank=True + ) position_mode = models.CharField( max_length=3, choices=ProgressMode.choices, @@ -335,22 +352,24 @@ class Quotation(BookStatus): blank=True, ) + def _format_position(self) -> Optional[str]: + """serialize page position""" + beg = self.position + end = self.endposition or 0 + if self.position_mode != "PG" or not beg: + return None + return f"pp. {beg}-{end}" if end > beg else f"p. {beg}" + @property def pure_content(self): """indicate the book in question for mastodon (or w/e) users""" quote = re.sub(r"^", '
"', self.quote) quote = re.sub(r"
$", '"', quote) - if self.position_mode == "PG" and self.position and (self.position > 0): - return_value = ( - f'{quote}-- ' - f'"{self.book.title}", page {self.position}
{self.content}' - ) - else: - return_value = ( - f'{quote} {self.content}' - ) - return return_value + title, href = self.book.title, self.book.remote_id + citation = f'— {title}' + if position := self._format_position(): + citation += f", {position}" + return f"{quote}{citation}
{self.content}" activity_serializer = activitypub.Quotation @@ -399,7 +418,7 @@ class ReviewRating(Review): def save(self, *args, **kwargs): if not self.rating: raise ValueError("ReviewRating object must include a numerical rating") - return super().save(*args, **kwargs) + super().save(*args, **kwargs) @property def pure_content(self): diff --git a/bookwyrm/models/user.py b/bookwyrm/models/user.py index 5f7b00d87..6e0912aec 100644 --- a/bookwyrm/models/user.py +++ b/bookwyrm/models/user.py @@ -3,9 +3,9 @@ import re from urllib.parse import urlparse from django.apps import apps -from django.contrib.auth.models import AbstractUser, Group +from django.contrib.auth.models import AbstractUser from django.contrib.postgres.fields import ArrayField, CICharField -from django.core.exceptions import PermissionDenied +from django.core.exceptions import PermissionDenied, ObjectDoesNotExist from django.dispatch import receiver from django.db import models, transaction from django.utils import timezone @@ -20,7 +20,7 @@ from bookwyrm.models.status import Status from bookwyrm.preview_images import generate_user_preview_image_task from bookwyrm.settings import DOMAIN, ENABLE_PREVIEW_IMAGES, USE_HTTPS, LANGUAGES from bookwyrm.signatures import create_key_pair -from bookwyrm.tasks import app, LOW +from bookwyrm.tasks import app, MISC from bookwyrm.utils import regex from .activitypub_mixin import OrderedCollectionPageMixin, ActivitypubMixin from .base_model import BookWyrmModel, DeactivationReason, new_access_code @@ -244,9 +244,10 @@ class User(OrderedCollectionPageMixin, AbstractUser): def admins(cls): """Get a queryset of the admins for this instance""" return cls.objects.filter( - models.Q(user_permissions__name__in=["moderate_user", "moderate_post"]) - | models.Q(is_superuser=True) - ) + models.Q(groups__name__in=["moderator", "admin"]) + | models.Q(is_superuser=True), + is_active=True, + ).distinct() def update_active_date(self): """this user is here! they are doing things!""" @@ -338,7 +339,7 @@ class User(OrderedCollectionPageMixin, AbstractUser): # this is a new remote user, we need to set their remote server field if not self.local: super().save(*args, **kwargs) - transaction.on_commit(lambda: set_remote_server.delay(self.id)) + transaction.on_commit(lambda: set_remote_server(self.id)) return with transaction.atomic(): @@ -355,8 +356,14 @@ class User(OrderedCollectionPageMixin, AbstractUser): # make users editors by default try: - self.groups.add(Group.objects.get(name="editor")) - except Group.DoesNotExist: + group = ( + apps.get_model("bookwyrm.SiteSettings") + .objects.get() + .default_user_auth_group + ) + if group: + self.groups.add(group) + except ObjectDoesNotExist: # this should only happen in tests pass @@ -372,6 +379,7 @@ class User(OrderedCollectionPageMixin, AbstractUser): """We don't actually delete the database entry""" # pylint: disable=attribute-defined-outside-init self.is_active = False + self.avatar = "" # skip the logic in this class's save() super().save(*args, **kwargs) @@ -386,10 +394,15 @@ class User(OrderedCollectionPageMixin, AbstractUser): def reactivate(self): """Now you want to come back, huh?""" # pylint: disable=attribute-defined-outside-init + if not self.allow_reactivation: + return self.is_active = True self.deactivation_reason = None self.allow_reactivation = False - super().save(broadcast=False) + super().save( + broadcast=False, + update_fields=["deactivation_reason", "is_active", "allow_reactivation"], + ) @property def local_path(self): @@ -458,18 +471,30 @@ class KeyPair(ActivitypubMixin, BookWyrmModel): return super().save(*args, **kwargs) -@app.task(queue=LOW) -def set_remote_server(user_id): +@app.task(queue=MISC) +def set_remote_server(user_id, allow_external_connections=False): """figure out the user's remote server in the background""" user = User.objects.get(id=user_id) actor_parts = urlparse(user.remote_id) - user.federated_server = get_or_create_remote_server(actor_parts.netloc) + federated_server = get_or_create_remote_server( + actor_parts.netloc, allow_external_connections=allow_external_connections + ) + # if we were unable to find the server, we need to create a new entry for it + if not federated_server: + # and to do that, we will call this function asynchronously. + if not allow_external_connections: + set_remote_server.delay(user_id, allow_external_connections=True) + return + + user.federated_server = federated_server user.save(broadcast=False, update_fields=["federated_server"]) if user.bookwyrm_user and user.outbox: get_remote_reviews.delay(user.outbox) -def get_or_create_remote_server(domain, refresh=False): +def get_or_create_remote_server( + domain, allow_external_connections=False, refresh=False +): """get info on a remote server""" server = FederatedServer() try: @@ -479,6 +504,9 @@ def get_or_create_remote_server(domain, refresh=False): except FederatedServer.DoesNotExist: pass + if not allow_external_connections: + return None + try: data = get_data(f"https://{domain}/.well-known/nodeinfo") try: @@ -502,7 +530,7 @@ def get_or_create_remote_server(domain, refresh=False): return server -@app.task(queue=LOW) +@app.task(queue=MISC) def get_remote_reviews(outbox): """ingest reviews by a new remote bookwyrm user""" outbox_page = outbox + "?page=true&type=Review" @@ -521,6 +549,11 @@ def preview_image(instance, *args, **kwargs): """create preview images when user is updated""" if not ENABLE_PREVIEW_IMAGES: return + + # don't call the task for remote users + if not instance.local: + return + changed_fields = instance.field_tracker.changed() if len(changed_fields) > 0: diff --git a/bookwyrm/preview_images.py b/bookwyrm/preview_images.py index d20145cd3..aba372abc 100644 --- a/bookwyrm/preview_images.py +++ b/bookwyrm/preview_images.py @@ -16,7 +16,7 @@ from django.core.files.storage import default_storage from django.db.models import Avg from bookwyrm import models, settings -from bookwyrm.tasks import app, LOW +from bookwyrm.tasks import app, IMAGES logger = logging.getLogger(__name__) @@ -71,20 +71,29 @@ def get_wrapped_text(text, font, content_width): low = 0 high = len(text) + draw = ImageDraw.Draw(Image.new("RGB", (100, 100))) + try: # ideal length is determined via binary search while low < high: mid = math.floor(low + high) wrapped_text = textwrap.fill(text, width=mid) - width = font.getsize_multiline(wrapped_text)[0] + + left, top, right, bottom = draw.multiline_textbbox( + (0, 0), wrapped_text, font=font + ) + width = right - left + height = bottom - top + if width < content_width: low = mid else: high = mid - 1 except AttributeError: wrapped_text = text + height = 26 - return wrapped_text + return wrapped_text, height def generate_texts_layer(texts, content_width): @@ -100,47 +109,53 @@ def generate_texts_layer(texts, content_width): text_y = 0 if "text_zero" in texts and texts["text_zero"]: - # Text one (Book title) - text_zero = get_wrapped_text(texts["text_zero"], font_text_zero, content_width) + # Text zero (Site preview domain name) + text_zero, text_height = get_wrapped_text( + texts["text_zero"], font_text_zero, content_width + ) text_layer_draw.multiline_text( (0, text_y), text_zero, font=font_text_zero, fill=TEXT_COLOR ) try: - text_y = text_y + font_text_zero.getsize_multiline(text_zero)[1] + 16 + text_y = text_y + text_height + 16 except (AttributeError, IndexError): text_y = text_y + 26 if "text_one" in texts and texts["text_one"]: - # Text one (Book title) - text_one = get_wrapped_text(texts["text_one"], font_text_one, content_width) + # Text one (Book/Site title, User display name) + text_one, text_height = get_wrapped_text( + texts["text_one"], font_text_one, content_width + ) text_layer_draw.multiline_text( (0, text_y), text_one, font=font_text_one, fill=TEXT_COLOR ) try: - text_y = text_y + font_text_one.getsize_multiline(text_one)[1] + 16 + text_y = text_y + text_height + 16 except (AttributeError, IndexError): text_y = text_y + 26 if "text_two" in texts and texts["text_two"]: - # Text one (Book subtitle) - text_two = get_wrapped_text(texts["text_two"], font_text_two, content_width) + # Text two (Book subtitle) + text_two, text_height = get_wrapped_text( + texts["text_two"], font_text_two, content_width + ) text_layer_draw.multiline_text( (0, text_y), text_two, font=font_text_two, fill=TEXT_COLOR ) try: - text_y = text_y + font_text_one.getsize_multiline(text_two)[1] + 16 + text_y = text_y + text_height + 16 except (AttributeError, IndexError): text_y = text_y + 26 if "text_three" in texts and texts["text_three"]: - # Text three (Book authors) - text_three = get_wrapped_text( + # Text three (Book authors, Site tagline, User address) + text_three, _ = get_wrapped_text( texts["text_three"], font_text_three, content_width ) @@ -172,7 +187,7 @@ def generate_instance_layer(content_width): instance_text_x = 0 if logo_img: - logo_img.thumbnail((50, 50), Image.ANTIALIAS) + logo_img.thumbnail((50, 50), Image.Resampling.LANCZOS) instance_layer.paste(logo_img, (0, 0)) @@ -183,7 +198,7 @@ def generate_instance_layer(content_width): (instance_text_x, 10), site.name, font=font_instance, fill=TEXT_COLOR ) - line_width = 50 + 10 + font_instance.getsize(site.name)[0] + line_width = 50 + 10 + round(font_instance.getlength(site.name)) line_layer = Image.new( "RGBA", (line_width, 2), color=(*(ImageColor.getrgb(TEXT_COLOR)), 50) @@ -253,10 +268,12 @@ def generate_default_inner_img(): default_cover_draw = ImageDraw.Draw(default_cover) text = "no image :(" - text_dimensions = font_cover.getsize(text) + text_left, text_top, text_right, text_bottom = font_cover.getbbox(text) + text_width, text_height = text_right - text_left, text_bottom - text_top + text_coords = ( - math.floor((inner_img_width - text_dimensions[0]) / 2), - math.floor((inner_img_height - text_dimensions[1]) / 2), + math.floor((inner_img_width - text_width) / 2), + math.floor((inner_img_height - text_height) / 2), ) default_cover_draw.text(text_coords, text, font=font_cover, fill="white") @@ -273,7 +290,9 @@ def generate_preview_image( # Cover try: inner_img_layer = Image.open(picture) - inner_img_layer.thumbnail((inner_img_width, inner_img_height), Image.ANTIALIAS) + inner_img_layer.thumbnail( + (inner_img_width, inner_img_height), Image.Resampling.LANCZOS + ) color_thief = ColorThief(picture) dominant_color = color_thief.get_color(quality=1) except: # pylint: disable=bare-except @@ -401,7 +420,7 @@ def save_and_cleanup(image, instance=None): # pylint: disable=invalid-name -@app.task(queue=LOW) +@app.task(queue=IMAGES) def generate_site_preview_image_task(): """generate preview_image for the website""" if not settings.ENABLE_PREVIEW_IMAGES: @@ -426,7 +445,7 @@ def generate_site_preview_image_task(): # pylint: disable=invalid-name -@app.task(queue=LOW) +@app.task(queue=IMAGES) def generate_edition_preview_image_task(book_id): """generate preview_image for a book""" if not settings.ENABLE_PREVIEW_IMAGES: @@ -451,14 +470,17 @@ def generate_edition_preview_image_task(book_id): save_and_cleanup(image, instance=book) -@app.task(queue=LOW) +@app.task(queue=IMAGES) def generate_user_preview_image_task(user_id): - """generate preview_image for a book""" + """generate preview_image for a user""" if not settings.ENABLE_PREVIEW_IMAGES: return user = models.User.objects.get(id=user_id) + if not user.local: + return + texts = { "text_one": user.display_name, "text_three": f"@{user.localname}@{settings.DOMAIN}", @@ -472,3 +494,25 @@ def generate_user_preview_image_task(user_id): image = generate_preview_image(texts=texts, picture=avatar) save_and_cleanup(image, instance=user) + + +@app.task(queue=IMAGES) +def remove_user_preview_image_task(user_id): + """remove preview_image for a user""" + if not settings.ENABLE_PREVIEW_IMAGES: + return + + user = models.User.objects.get(id=user_id) + + try: + file_name = user.preview_image.name + except ValueError: + file_name = None + + # Delete image in model + user.preview_image.delete(save=False) + user.save(broadcast=False, update_fields=["preview_image"]) + + # Delete image file + if file_name and default_storage.exists(file_name): + default_storage.delete(file_name) diff --git a/bookwyrm/redis_store.py b/bookwyrm/redis_store.py index ae50db2ee..e188487aa 100644 --- a/bookwyrm/redis_store.py +++ b/bookwyrm/redis_store.py @@ -4,12 +4,7 @@ import redis from bookwyrm import settings -r = redis.Redis( - host=settings.REDIS_ACTIVITY_HOST, - port=settings.REDIS_ACTIVITY_PORT, - password=settings.REDIS_ACTIVITY_PASSWORD, - db=settings.REDIS_ACTIVITY_DB_INDEX, -) +r = redis.from_url(settings.REDIS_ACTIVITY_URL) class RedisStore(ABC): @@ -21,12 +16,12 @@ class RedisStore(ABC): """the object and rank""" return {obj.id: self.get_rank(obj)} - def add_object_to_related_stores(self, obj, execute=True): - """add an object to all suitable stores""" + def add_object_to_stores(self, obj, stores, execute=True): + """add an object to a given set of stores""" value = self.get_value(obj) # we want to do this as a bulk operation, hence "pipeline" pipeline = r.pipeline() - for store in self.get_stores_for_object(obj): + for store in stores: # add the status to the feed pipeline.zadd(store, value) # trim the store @@ -37,14 +32,14 @@ class RedisStore(ABC): # and go! return pipeline.execute() - def remove_object_from_related_stores(self, obj, stores=None): + # pylint: disable=no-self-use + def remove_object_from_stores(self, obj, stores): """remove an object from all stores""" - # if the stoers are provided, the object can just be an id + # if the stores are provided, the object can just be an id if stores and isinstance(obj, int): obj_id = obj else: obj_id = obj.id - stores = self.get_stores_for_object(obj) if stores is None else stores pipeline = r.pipeline() for store in stores: pipeline.zrem(store, -1, obj_id) @@ -87,10 +82,6 @@ class RedisStore(ABC): def get_objects_for_store(self, store): """a queryset of what should go in a store, used for populating it""" - @abstractmethod - def get_stores_for_object(self, obj): - """the stores that an object belongs in""" - @abstractmethod def get_rank(self, obj): """how to rank an object""" diff --git a/bookwyrm/settings.py b/bookwyrm/settings.py index de898609f..94ec761db 100644 --- a/bookwyrm/settings.py +++ b/bookwyrm/settings.py @@ -1,9 +1,12 @@ """ bookwyrm settings and configuration """ import os +from typing import AnyStr + from environs import Env import requests from django.utils.translation import gettext_lazy as _ +from django.core.exceptions import ImproperlyConfigured # pylint: disable=line-too-long @@ -11,22 +14,22 @@ from django.utils.translation import gettext_lazy as _ env = Env() env.read_env() DOMAIN = env("DOMAIN") -VERSION = "0.4.6" +VERSION = "0.6.6" RELEASE_API = env( "RELEASE_API", "https://api.github.com/repos/bookwyrm-social/bookwyrm/releases/latest", ) -PAGE_LENGTH = env("PAGE_LENGTH", 15) +PAGE_LENGTH = env.int("PAGE_LENGTH", 15) DEFAULT_LANGUAGE = env("DEFAULT_LANGUAGE", "English") -JS_CACHE = "e678183c" +JS_CACHE = "ac315a3b" # email EMAIL_BACKEND = env("EMAIL_BACKEND", "django.core.mail.backends.smtp.EmailBackend") EMAIL_HOST = env("EMAIL_HOST") -EMAIL_PORT = env("EMAIL_PORT", 587) +EMAIL_PORT = env.int("EMAIL_PORT", 587) EMAIL_HOST_USER = env("EMAIL_HOST_USER") EMAIL_HOST_PASSWORD = env("EMAIL_HOST_PASSWORD") EMAIL_USE_TLS = env.bool("EMAIL_USE_TLS", True) @@ -36,7 +39,7 @@ EMAIL_SENDER_DOMAIN = env("EMAIL_SENDER_DOMAIN", DOMAIN) EMAIL_SENDER = f"{EMAIL_SENDER_NAME}@{EMAIL_SENDER_DOMAIN}" # Build paths inside the project like this: os.path.join(BASE_DIR, ...) -BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +BASE_DIR: AnyStr = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) LOCALE_PATHS = [ os.path.join(BASE_DIR, "locale"), ] @@ -68,13 +71,15 @@ FONT_DIR = os.path.join(STATIC_ROOT, "fonts") # Quick-start development settings - unsuitable for production # See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/ -# SECURITY WARNING: keep the secret key used in production secret! -SECRET_KEY = env("SECRET_KEY") - # SECURITY WARNING: don't run with debug turned on in production! DEBUG = env.bool("DEBUG", True) USE_HTTPS = env.bool("USE_HTTPS", not DEBUG) +# SECURITY WARNING: keep the secret key used in production secret! +SECRET_KEY = env("SECRET_KEY") +if not DEBUG and SECRET_KEY == "7(2w1sedok=aznpq)ta1mc4i%4h=xx@hxwx*o57ctsuml0x%fr": + raise ImproperlyConfigured("You must change the SECRET_KEY env variable") + ALLOWED_HOSTS = env.list("ALLOWED_HOSTS", ["*"]) # Application definition @@ -101,6 +106,7 @@ MIDDLEWARE = [ "django.middleware.locale.LocaleMiddleware", "django.middleware.common.CommonMiddleware", "django.middleware.csrf.CsrfViewMiddleware", + "csp.middleware.CSPMiddleware", "django.contrib.auth.middleware.AuthenticationMiddleware", "bookwyrm.middleware.TimezoneMiddleware", "bookwyrm.middleware.IPBlocklistMiddleware", @@ -147,6 +153,9 @@ LOGGING = { "require_debug_true": { "()": "django.utils.log.RequireDebugTrue", }, + "ignore_missing_variable": { + "()": "bookwyrm.utils.log.IgnoreVariableDoesNotExist", + }, }, "handlers": { # Overrides the default handler to make it log to console @@ -154,6 +163,7 @@ LOGGING = { # console if DEBUG=False) "console": { "level": LOG_LEVEL, + "filters": ["ignore_missing_variable"], "class": "logging.StreamHandler", }, # This is copied as-is from the default logger, and is @@ -189,7 +199,8 @@ STATICFILES_FINDERS = [ ] SASS_PROCESSOR_INCLUDE_FILE_PATTERN = r"^.+\.[s]{0,1}(?:a|c)ss$" -SASS_PROCESSOR_ENABLED = True +# when debug is disabled, make sure to compile themes once with `./bw-dev compile_themes` +SASS_PROCESSOR_ENABLED = DEBUG # minify css is production but not dev if not DEBUG: @@ -199,11 +210,14 @@ WSGI_APPLICATION = "bookwyrm.wsgi.application" # redis/activity streams settings REDIS_ACTIVITY_HOST = env("REDIS_ACTIVITY_HOST", "localhost") -REDIS_ACTIVITY_PORT = env("REDIS_ACTIVITY_PORT", 6379) -REDIS_ACTIVITY_PASSWORD = env("REDIS_ACTIVITY_PASSWORD", None) -REDIS_ACTIVITY_DB_INDEX = env("REDIS_ACTIVITY_DB_INDEX", 0) - -MAX_STREAM_LENGTH = int(env("MAX_STREAM_LENGTH", 200)) +REDIS_ACTIVITY_PORT = env.int("REDIS_ACTIVITY_PORT", 6379) +REDIS_ACTIVITY_PASSWORD = requests.utils.quote(env("REDIS_ACTIVITY_PASSWORD", "")) +REDIS_ACTIVITY_DB_INDEX = env.int("REDIS_ACTIVITY_DB_INDEX", 0) +REDIS_ACTIVITY_URL = env( + "REDIS_ACTIVITY_URL", + f"redis://:{REDIS_ACTIVITY_PASSWORD}@{REDIS_ACTIVITY_HOST}:{REDIS_ACTIVITY_PORT}/{REDIS_ACTIVITY_DB_INDEX}", +) +MAX_STREAM_LENGTH = env.int("MAX_STREAM_LENGTH", 200) STREAMS = [ {"key": "home", "name": _("Home Timeline"), "shortname": _("Home")}, @@ -212,12 +226,12 @@ STREAMS = [ # Search configuration # total time in seconds that the instance will spend searching connectors -SEARCH_TIMEOUT = int(env("SEARCH_TIMEOUT", 8)) +SEARCH_TIMEOUT = env.int("SEARCH_TIMEOUT", 8) # timeout for a query to an individual connector -QUERY_TIMEOUT = int(env("QUERY_TIMEOUT", 5)) +QUERY_TIMEOUT = env.int("INTERACTIVE_QUERY_TIMEOUT", env.int("QUERY_TIMEOUT", 5)) # Redis cache backend -if env("USE_DUMMY_CACHE", False): +if env.bool("USE_DUMMY_CACHE", False): CACHES = { "default": { "BACKEND": "django.core.cache.backends.dummy.DummyCache", @@ -227,7 +241,7 @@ else: CACHES = { "default": { "BACKEND": "django_redis.cache.RedisCache", - "LOCATION": f"redis://:{REDIS_ACTIVITY_PASSWORD}@{REDIS_ACTIVITY_HOST}:{REDIS_ACTIVITY_PORT}/{REDIS_ACTIVITY_DB_INDEX}", + "LOCATION": REDIS_ACTIVITY_URL, "OPTIONS": { "CLIENT_CLASS": "django_redis.client.DefaultClient", }, @@ -247,7 +261,7 @@ DATABASES = { "USER": env("POSTGRES_USER", "bookwyrm"), "PASSWORD": env("POSTGRES_PASSWORD", "bookwyrm"), "HOST": env("POSTGRES_HOST", ""), - "PORT": env("PGPORT", 5432), + "PORT": env.int("PGPORT", 5432), }, } @@ -282,12 +296,15 @@ LANGUAGES = [ ("en-us", _("English")), ("ca-es", _("Català (Catalan)")), ("de-de", _("Deutsch (German)")), + ("eo-uy", _("Esperanto (Esperanto)")), ("es-es", _("Español (Spanish)")), + ("eu-es", _("Euskara (Basque)")), ("gl-es", _("Galego (Galician)")), ("it-it", _("Italiano (Italian)")), ("fi-fi", _("Suomi (Finnish)")), ("fr-fr", _("Français (French)")), ("lt-lt", _("Lietuvių (Lithuanian)")), + ("nl-nl", _("Nederlands (Dutch)")), ("no-no", _("Norsk (Norwegian)")), ("pl-pl", _("Polski (Polish)")), ("pt-br", _("Português do Brasil (Brazilian Portuguese)")), @@ -298,6 +315,10 @@ LANGUAGES = [ ("zh-hant", _("繁體中文 (Traditional Chinese)")), ] +LANGUAGE_ARTICLES = { + "English": {"the", "a", "an"}, + "Español (Spanish)": {"un", "una", "unos", "unas", "el", "la", "los", "las"}, +} TIME_ZONE = "UTC" @@ -320,14 +341,18 @@ IMAGEKIT_DEFAULT_CACHEFILE_STRATEGY = "bookwyrm.thumbnail_generation.Strategy" # https://docs.djangoproject.com/en/3.2/howto/static-files/ PROJECT_DIR = os.path.dirname(os.path.abspath(__file__)) +CSP_ADDITIONAL_HOSTS = env.list("CSP_ADDITIONAL_HOSTS", []) # Storage PROTOCOL = "http" if USE_HTTPS: PROTOCOL = "https" + SESSION_COOKIE_SECURE = True + CSRF_COOKIE_SECURE = True USE_S3 = env.bool("USE_S3", False) +USE_AZURE = env.bool("USE_AZURE", False) if USE_S3: # AWS settings @@ -349,14 +374,53 @@ if USE_S3: MEDIA_FULL_URL = MEDIA_URL STATIC_FULL_URL = STATIC_URL DEFAULT_FILE_STORAGE = "bookwyrm.storage_backends.ImagesStorage" + CSP_DEFAULT_SRC = ["'self'", AWS_S3_CUSTOM_DOMAIN] + CSP_ADDITIONAL_HOSTS + CSP_SCRIPT_SRC = ["'self'", AWS_S3_CUSTOM_DOMAIN] + CSP_ADDITIONAL_HOSTS +elif USE_AZURE: + AZURE_ACCOUNT_NAME = env("AZURE_ACCOUNT_NAME") + AZURE_ACCOUNT_KEY = env("AZURE_ACCOUNT_KEY") + AZURE_CONTAINER = env("AZURE_CONTAINER") + AZURE_CUSTOM_DOMAIN = env("AZURE_CUSTOM_DOMAIN") + # Azure Static settings + STATIC_LOCATION = "static" + STATIC_URL = ( + f"{PROTOCOL}://{AZURE_CUSTOM_DOMAIN}/{AZURE_CONTAINER}/{STATIC_LOCATION}/" + ) + STATICFILES_STORAGE = "bookwyrm.storage_backends.AzureStaticStorage" + # Azure Media settings + MEDIA_LOCATION = "images" + MEDIA_URL = ( + f"{PROTOCOL}://{AZURE_CUSTOM_DOMAIN}/{AZURE_CONTAINER}/{MEDIA_LOCATION}/" + ) + MEDIA_FULL_URL = MEDIA_URL + STATIC_FULL_URL = STATIC_URL + DEFAULT_FILE_STORAGE = "bookwyrm.storage_backends.AzureImagesStorage" + CSP_DEFAULT_SRC = ["'self'", AZURE_CUSTOM_DOMAIN] + CSP_ADDITIONAL_HOSTS + CSP_SCRIPT_SRC = ["'self'", AZURE_CUSTOM_DOMAIN] + CSP_ADDITIONAL_HOSTS else: STATIC_URL = "/static/" MEDIA_URL = "/images/" MEDIA_FULL_URL = f"{PROTOCOL}://{DOMAIN}{MEDIA_URL}" STATIC_FULL_URL = f"{PROTOCOL}://{DOMAIN}{STATIC_URL}" + CSP_DEFAULT_SRC = ["'self'"] + CSP_ADDITIONAL_HOSTS + CSP_SCRIPT_SRC = ["'self'"] + CSP_ADDITIONAL_HOSTS + +CSP_INCLUDE_NONCE_IN = ["script-src"] OTEL_EXPORTER_OTLP_ENDPOINT = env("OTEL_EXPORTER_OTLP_ENDPOINT", None) OTEL_EXPORTER_OTLP_HEADERS = env("OTEL_EXPORTER_OTLP_HEADERS", None) OTEL_SERVICE_NAME = env("OTEL_SERVICE_NAME", None) +OTEL_EXPORTER_CONSOLE = env.bool("OTEL_EXPORTER_CONSOLE", False) -TWO_FACTOR_LOGIN_MAX_SECONDS = 60 +TWO_FACTOR_LOGIN_MAX_SECONDS = env.int("TWO_FACTOR_LOGIN_MAX_SECONDS", 60) +TWO_FACTOR_LOGIN_VALIDITY_WINDOW = env.int("TWO_FACTOR_LOGIN_VALIDITY_WINDOW", 2) + +HTTP_X_FORWARDED_PROTO = env.bool("SECURE_PROXY_SSL_HEADER", False) +if HTTP_X_FORWARDED_PROTO: + SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https") + +# Instance Actor for signing GET requests to "secure mode" +# Mastodon servers. +# Do not change this setting unless you already have an existing +# user with the same username - in which case you should change it! +INSTANCE_ACTOR_USERNAME = "bookwyrm.instance.actor" diff --git a/bookwyrm/signatures.py b/bookwyrm/signatures.py index 61cafe71f..08780b731 100644 --- a/bookwyrm/signatures.py +++ b/bookwyrm/signatures.py @@ -15,29 +15,40 @@ MAX_SIGNATURE_AGE = 300 def create_key_pair(): """a new public/private key pair, used for creating new users""" random_generator = Random.new().read - key = RSA.generate(1024, random_generator) + key = RSA.generate(2048, random_generator) private_key = key.export_key().decode("utf8") - public_key = key.publickey().export_key().decode("utf8") + public_key = key.public_key().export_key().decode("utf8") return private_key, public_key -def make_signature(sender, destination, date, digest): +def make_signature(method, sender, destination, date, **kwargs): """uses a private key to sign an outgoing message""" inbox_parts = urlparse(destination) signature_headers = [ - f"(request-target): post {inbox_parts.path}", + f"(request-target): {method} {inbox_parts.path}", f"host: {inbox_parts.netloc}", f"date: {date}", - f"digest: {digest}", ] + headers = "(request-target) host date" + digest = kwargs.get("digest") + if digest is not None: + signature_headers.append(f"digest: {digest}") + headers = "(request-target) host date digest" + message_to_sign = "\n".join(signature_headers) signer = pkcs1_15.new(RSA.import_key(sender.key_pair.private_key)) signed_message = signer.sign(SHA256.new(message_to_sign.encode("utf8"))) + # For legacy reasons we need to use an incorrect keyId for older Bookwyrm versions + key_id = ( + f"{sender.remote_id}#main-key" + if kwargs.get("use_legacy_key") + else f"{sender.remote_id}/#main-key" + ) signature = { - "keyId": f"{sender.remote_id}#main-key", + "keyId": key_id, "algorithm": "rsa-sha256", - "headers": "(request-target) host date digest", + "headers": headers, "signature": b64encode(signed_message).decode("utf8"), } return ",".join(f'{k}="{v}"' for (k, v) in signature.items()) diff --git a/bookwyrm/static/css/bookwyrm/_all.scss b/bookwyrm/static/css/bookwyrm/_all.scss index f1b23db9a..5bb08b931 100644 --- a/bookwyrm/static/css/bookwyrm/_all.scss +++ b/bookwyrm/static/css/bookwyrm/_all.scss @@ -137,6 +137,10 @@ button:focus-visible .button-invisible-overlay { opacity: 1; } +button.button-paragraph { + vertical-align: middle; +} + /** States ******************************************************************************/ diff --git a/bookwyrm/static/css/bookwyrm/components/_book_cover.scss b/bookwyrm/static/css/bookwyrm/components/_book_cover.scss index d1125197e..48b564a0b 100644 --- a/bookwyrm/static/css/bookwyrm/components/_book_cover.scss +++ b/bookwyrm/static/css/bookwyrm/components/_book_cover.scss @@ -5,7 +5,7 @@ * - .book-cover is positioned and sized based on its container. * * To have the cover within specific dimensions, specify a width or height for - * standard bulma’s named breapoints: + * standard bulma’s named breakpoints: * * `is-(w|h)-(auto|xs|s|m|l|xl|xxl)[-(mobile|tablet|desktop)]` * @@ -43,7 +43,7 @@ max-height: 100%; /* Useful when stretching under-sized images. */ - image-rendering: optimizequality; + image-rendering: optimizeQuality; image-rendering: smooth; } diff --git a/bookwyrm/static/css/bookwyrm/components/_copy.scss b/bookwyrm/static/css/bookwyrm/components/_copy.scss index e0c4246e6..7a47c1dba 100644 --- a/bookwyrm/static/css/bookwyrm/components/_copy.scss +++ b/bookwyrm/static/css/bookwyrm/components/_copy.scss @@ -28,3 +28,31 @@ .vertical-copy button { width: 100%; } + +.copy-tooltip { + overflow: visible; + visibility: hidden; + width: 140px; + background-color: #555; + color: #fff; + text-align: center; + border-radius: 6px; + padding: 5px; + position: absolute; + z-index: 1; + margin-left: -30px; + margin-top: -45px; + opacity: 0; + transition: opacity 0.3s; +} + +.copy-tooltip::after { + content: ""; + position: absolute; + top: 100%; + left: 50%; + margin-left: -60px; + border-width: 5px; + border-style: solid; + border-color: #555 transparent transparent transparent; + } diff --git a/bookwyrm/static/css/bookwyrm/components/_details.scss b/bookwyrm/static/css/bookwyrm/components/_details.scss index de29629c8..4145554eb 100644 --- a/bookwyrm/static/css/bookwyrm/components/_details.scss +++ b/bookwyrm/static/css/bookwyrm/components/_details.scss @@ -81,7 +81,19 @@ details.dropdown .dropdown-menu a:focus-visible { details.details-panel { box-shadow: 0 0 0 1px $border; transition: box-shadow 0.2s ease; - padding: 0.75rem; + padding: 0; + + > * { + padding: 0.75rem; + } + + summary { + position: relative; + + .details-close { + padding: 0.75rem; + } + } } details[open].details-panel, @@ -89,10 +101,6 @@ details.details-panel:hover { box-shadow: 0 0 0 1px $border; } -details.details-panel summary { - position: relative; -} - details summary .details-close { position: absolute; right: 0; diff --git a/bookwyrm/static/css/bookwyrm/components/_stars.scss b/bookwyrm/static/css/bookwyrm/components/_stars.scss index 1a8e3680f..db2772dc0 100644 --- a/bookwyrm/static/css/bookwyrm/components/_stars.scss +++ b/bookwyrm/static/css/bookwyrm/components/_stars.scss @@ -5,6 +5,10 @@ white-space: nowrap; } +.stars .no-rating { + font-style: italic; +} + /** Stars in a review form * * Specificity makes hovering taking over checked inputs. diff --git a/bookwyrm/static/css/bookwyrm/components/_tabs.scss b/bookwyrm/static/css/bookwyrm/components/_tabs.scss index 8e00f6a88..2d68a383b 100644 --- a/bookwyrm/static/css/bookwyrm/components/_tabs.scss +++ b/bookwyrm/static/css/bookwyrm/components/_tabs.scss @@ -44,12 +44,12 @@ .bw-tabs a:hover { border-bottom-color: transparent; - color: $text; + color: $text } .bw-tabs a.is-active { border-bottom-color: transparent; - color: $link; + color: $link } .bw-tabs.is-left { diff --git a/bookwyrm/static/css/bookwyrm/overrides/_bulma_overrides.scss b/bookwyrm/static/css/bookwyrm/overrides/_bulma_overrides.scss index f46e7b957..9ab44f89d 100644 --- a/bookwyrm/static/css/bookwyrm/overrides/_bulma_overrides.scss +++ b/bookwyrm/static/css/bookwyrm/overrides/_bulma_overrides.scss @@ -1,3 +1,53 @@ +.summary-on-open { + display: none; +} + +@media only screen and (max-width: 768px) { + .navbar-menu { + text-align: right; + padding-right: 1rem; + + .tags { + justify-content: flex-end; + } + + #navbar-dropdown { + &[open] { + .summary-on-open { + display: initial; + position: fixed; + top: 0; + left: 0; + right: 0; + height: 3rem; + z-index: 31; + background-color: $dropdown-content-background-color; + padding: 1rem 1.75rem; + line-height: 1; + } + } + + .dropdown-menu { + padding-top: 0; + top: 3rem; + } + + .dropdown-content { + padding-top: 0; + box-shadow: none; + border-top-left-radius: 0; + border-top-right-radius: 0; + } + + .navbar-item { + /* see ../components/_details.scss :: Navbar details */ + padding-right: 1.75rem; + font-size: 1rem; + } + } + } +} + .image { overflow: hidden; } @@ -59,3 +109,9 @@ max-height: 35em; overflow: hidden; } + +.dropdown-menu .button { + @include mobile { + font-size: $size-6; + } +} diff --git a/bookwyrm/static/css/bookwyrm/utilities/_size.scss b/bookwyrm/static/css/bookwyrm/utilities/_size.scss index cbc74d7ab..258aa9a73 100644 --- a/bookwyrm/static/css/bookwyrm/utilities/_size.scss +++ b/bookwyrm/static/css/bookwyrm/utilities/_size.scss @@ -40,6 +40,10 @@ width: 500px !important; } +.is-h-em { + height: 1em !important; +} + .is-h-xs { height: 80px !important; } diff --git a/bookwyrm/static/css/fonts/icomoon.eot b/bookwyrm/static/css/fonts/icomoon.eot index 69628662b..33dc07eec 100644 Binary files a/bookwyrm/static/css/fonts/icomoon.eot and b/bookwyrm/static/css/fonts/icomoon.eot differ diff --git a/bookwyrm/static/css/fonts/icomoon.svg b/bookwyrm/static/css/fonts/icomoon.svg index c67c8b225..058c19226 100644 --- a/bookwyrm/static/css/fonts/icomoon.svg +++ b/bookwyrm/static/css/fonts/icomoon.svg @@ -39,9 +39,12 @@{% trans "That’s great!" %}
-- {% blocktrans with pages=pages_average|intcomma %}That makes an average of {{ pages }} pages per book.{% endblocktrans %} -
+ {% if pages > 0 %} ++ {% blocktrans with pages=pages_average|intcomma %}That makes an average of {{ pages }} pages per book.{% endblocktrans %} +
+ {% endif %} {% if no_page_number %}{% blocktrans trimmed count counter=no_page_number %} - ({{ no_page_number }} book doesn’t have pages) + (No page data was available for {{ no_page_number }} book) {% plural %} - ({{ no_page_number }} books don’t have pages) + (No page data was available for {{ no_page_number }} books) {% endblocktrans %}
{% endif %} diff --git a/bookwyrm/templates/author/author.html b/bookwyrm/templates/author/author.html index f186c0f6e..e24a77dcd 100644 --- a/bookwyrm/templates/author/author.html +++ b/bookwyrm/templates/author/author.html @@ -28,7 +28,7 @@ {% firstof author.aliases author.born author.died as details %} - {% firstof author.wikipedia_link author.openlibrary_key author.inventaire_id author.isni as links %} + {% firstof author.wikipedia_link author.website author.openlibrary_key author.inventaire_id author.isni author.isfdb as links %} {% if details or links %}{{ form.website }}
+ + {% include 'snippets/form_errors.html' with errors_list=form.website.errors id="desc_website" %} +