Previously we were querying the full book objects just to get a list of
id's, which is much slower and also takes a lot more memory, which can
cause the process to be killed on memory-limited machines with a large
number of books.
Instead, since we're just dispatching jobs here, we can just ask for the
id's, which is faster and much more practical memory-wise.
The map is a little annoying, I didn't see a way to directly get just a
list of the value of one field, so we have to get a list of
dictionairies with one key and then pull that key out. Whatevs.
This adds the page number for quote and comment statuses where a page number is provided:
- all ActivityPub posts
- Explore cards for comments (quotes already have the page number)
This responds to #2136
When an activity needs to be broadcast to the whole wide fediverse, the
number of requests can get enormous and the broadcast task ends up
taking ages to run. This change sends these requests out in one aiohttp
session, to improve performance.
Loading every edition in one task takes ages, and produces a large task
that clogs up the queue. This will create more, smaller tasks that will
finish more quickly.
Set OpenLibrary search condifidence based on the provided result order,
just using 1/(list index), so the first has rank 1, the second 0.5, the
third 0.33, et cetera.
By default, OpenLibrary and Inventaire were prioritzed below other
BookWyrm nodes. In practice, people have gotten better search results
from these connectors, hence the change. With the search refactor, this
has much less impact, but it will show these search results higher in
the list.
If the results page shows all the connectors' results integrated, this
field should be removed entirely.
Since we get all the results quickly now, this aggregates all the
results that came back and sorts them by confidence, and returns the
highest confidence result. The confidences aren't great on free text
search, but conceptually that's how it should work at least.
It may make sense to aggregate the search results in all contexts, but
I'll propose that in a separate PR.
The parser was extracting the list of search results from the json
object returned by the search endpoint, and the formatter was converting
an individual json entry into a SearchResult object. This just merged
them into one function, because they are never used separately.
Adds logging and error handling for some of the numerous ways a request
could fail (the remote site is down, the url is blocked, etc).
I also have the results boxes open by default, which makes it more
legible imo.
The database lookup doesn't work during the asyn process, so this change
loops through the connectors and grabs the formatted urls before sending
it to the async handler.
Instead of having individual search functions that make individual
requests, the connectors will always be searched asynchronously
together. The process_seach_response combines the parse and format
functions, which could probably be merged into one over-rideable
function.
The current to-do on this is to remove Inventaire search results that
are below the confidence threshhold after search, which used to happen
in the `search` function.
This is the untest first pass at re-arranging remote search to work in
parallel rather than sequence. It moves a couple functions around
(raise_not_valid_url, for example, needs to be in connector_manager.py
now to avoid circular imports). It adds a function to Connector objects
that generates a search result (either to the isbn endpoint or the free
text endpoint) based on the query, which was previously done as part of
the search.
I also lowered the timeout to 8 seconds by default.
The original system customized how a rating is displayed to every user
based on the privacy settings of the reviews and, relatedly, who the
user follows. This is cool, but the query is too complicated to load in
sessions, and the initial load, which isn't mitigated by caching, is too
much and causes timeouts for many users. Also the cache clearing wasn't
working correctly because I put in a wildcard, which does not work.
