mirror of https://github.com/searxng/searxng.git synced 2025-02-28 17:56:48 +00:00

Markus Heiser edfbf1e118 [refactor] typification of SearXNG (initial) / result items (part 1)

Typification of SearXNG
=======================

This patch introduces the typing of the results.  The why and how is described
in the documentation, please generate the documentation ..

    $ make docs.clean docs.live

and read the following articles in the "Developer documentation":

- result types --> http://0.0.0.0:8000/dev/result_types/index.html

The result types are available from the `searx.result_types` module.  The
following have been implemented so far:

- base result type: `searx.result_type.Result`
  --> http://0.0.0.0:8000/dev/result_types/base_result.html

- answer results
  --> http://0.0.0.0:8000/dev/result_types/answer.html

including the type for translations (inspired by #3925).  For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.

Doc of the fields used in Templates
===================================

The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development).  It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.

- https://github.com/searxng/searxng/issues/357

Answer Templates
================

With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).

    !en-de dog

Plugins & Answerer
==================

The implementation for `Plugin` and `Answer` has been revised, see
documentation:

- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html

With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)

Autocomplete
============

The autocompletion had a bug where the results from `Answer` had not been shown
in the past.  To test activate autocompletion and try search terms for which we
have answerers

- statistics: type `min 1 2 3` .. in the completion list you should find an
  entry like `[de] min(1, 2, 3) = 1`

- random: type `random uuid` .. in the completion list, the first item is a
  random UUID

Extended Types
==============

SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:

- Extended Types
  --> http://0.0.0.0:8000/dev/extended_types.html

Unit-Tests
==========

The unit tests have been completely revised.  In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it.  This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:

- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
  issues, with non-deterministic behavior ..

- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005

Why msgspec.Struct
==================

We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:

- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356

In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...

The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.

Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!

Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.

HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec.  They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.

Why just one commit?
====================

I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

2025-01-28 07:07:08 +01:00

14 KiB

Raw Blame History

Simple Theme Templates

The simple template is complex, it consists of many different elements and also uses macros and include statements. The following is a rough overview that we would like to give the developerat hand, details must still be taken from the sources <searx/templates/simple/>.

A result item <result types> can be of different media types. The media type of a result is defined by the :pyresult_type.Result.template. To set another media-type as template default, the field template in the result item must be set to the desired type.

Contents

Result template macros

`result_header`

Execpt image.html and some others this macro is used in nearly all result types in the main result list.

Fields used in the template macro result_header <searx/templates/simple/macros.html>:

url : :pystr: Link URL of the result item.
title : :pystr: Link title of the result item.
img_src, thumbnail : :pystr: URL of a image or thumbnail that is displayed in the result item.

`result_sub_header`

Execpt image.html and some others this macro is used in nearly all result types in the main result list.

Fields used in the template macro result_sub_header <searx/templates/simple/macros.html>:

publishedDate : :pydatetime.datetime: The date on which the object was published.
length: :pytime.struct_time: Playing duration in seconds.
views: :pystr: View count in humanized number format.
author : :pystr: Author of the title.
metadata : :pystr: Miscellaneous metadata.

`engine_data_form`

The engine_data_form macro is used in results,html <searx/templates/simple/results.html> in a HTML <form/> element. The intention of this macro is to pass data of a engine from one :pyresponse <searx.engines.demo_online.response> to the :pysearx.search.SearchQuery of the next :pyrequest <searx.engines.demo_online.request>.

To pass data, engine's response handler can append result items of typ engine_data. This is by example used to pass a token from the response to the next request:

def response(resp):
    ...
    results.append({
       'engine_data': token,
       'key': 'next_page_token',
    })
    ...
    return results

def request(query, params):
    page_token = params['engine_data'].get('next_page_token')

Main Result List

The media types of the main result type are the template files in the result_templates <searx/templates/simple/result_templates>.

`default.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the default.html <searx/templates/simple/result_templates/default.html>:

content : :pystr: General text of the result item.
iframe_src : :pystr: URL of an embedded <iframe> / the frame is collapsible.
audio_src : uri,: URL of an embedded <audio controls>.

`images.html`

The images are displayed as small thumbnails in the main results list.

title : :pystr: Title of the image.
thumbnail_src : :pystr: URL of a preview of the image.
resolution :pystr: The resolution of the image (e.g. 1920 x 1080 pixel)

Image labels

Clicking on the preview opens a gallery view in which all further metadata for the image is displayed. Addition fields used in the images.html <searx/templates/simple/result_templates/images.html>:

img_src : :pystr: URL of the full size image.
content: :pystr: Description of the image.
author: :pystr: Name of the author of the image.
img_format : :pystr: The format of the image (e.g. png).
source : :pystr: Source of the image.
filesize: :pystr: Size of bytes in :pyhuman readable <searx.humanize_bytes> notation (e.g. MB for 1024 * 1024 Bytes filesize).
url : :pystr: URL of the page from where the images comes from (source).

`videos.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the videos.html <searx/templates/simple/result_templates/videos.html>:

iframe_src : :pystr

URL of an embedded <iframe> / the frame is collapsible.

The videos are displayed as small thumbnails in the main results list, there is an additional button to collaps/open the embeded video.

content : :pystr

Description of the code fragment.

`torrent.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the torrent.html <searx/templates/simple/result_templates/torrent.html>:

magnetlink:: URL of the magnet link.
torrentfile: URL of the torrent file.
seed : int: Number of seeders.
leech : int: Number of leecher
filesize : int: Size in Bytes (rendered to human readable unit of measurement).
files : int: Number of files.

`map.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the map.html <searx/templates/simple/result_templates/map.html>:

content : :pystr

Description of the item.

address_label : :pystr

Label of the address / default _('address').

geojson : GeoJSON

Geometries mapped to HTMLElement.dataset (data-map-geojson) and used by Leaflet.

boundingbox : [ min-lon, min-lat, max-lon, max-lat]

A bbox area defined by min longitude , min latitude , max longitude and max latitude. The bounding box is mapped to HTMLElement.dataset (data-map-boundingbox) and is used by Leaflet.

longitude, latitude : :pystr

Geographical coordinates, mapped to HTMLElement.dataset (data-map-lon, data-map-lat) and is used by Leaflet.

address : {...}

A dicticonary with the address data:

address = {
    'name'          : str,  # name of object
    'road'          : str,  # street name of object
    'house_number'  : str,  # house number of object
    'postcode'      : str,  # postcode of object
    'country'       : str,  # country of object
    'country_code'  : str,
    'locality'      : str,
}

country_code : :pystr: Country code of the object.
locality : :pystr: The name of the city, town, township, village, borough, etc. in which this object is located.

links : [link1, link2, ...]

A list of links with labels:

links.append({
    'label'       : str,
    'url'         : str,
    'url_label'   : str,  # set by some engines but unused (oscar)
})

data : [data1, data2, ...]

A list of additional data, shown in two columns and containing a label and value.

data.append({
   'label'   : str,
   'value'   : str,
   'key'     : str,  # set by some engines but unused
})

type : :pystr # set by some engines but unused (oscar)

Tag label from OSM_KEYS_TAGS['tags'] <update_osm_keys_tags.py>.

type_icon : :pystr # set by some engines but unused (oscar)

Type's icon.

osm : {...}

OSM-type and OSM-ID, can be used to Lookup OSM data (Nominatim). There is also a discussion about "place_id is not a persistent id" and the perma_id.

osm = {
    'type': str,
    'id':   str,
}

type : :pystr: Type of osm-object (if OSM-Result).
id :: ID of osm-object (if OSM-Result).

Hint

The osm property is set by engine openstreetmap.py, but it is not used in the map.html template yet.

`paper.html`

Displays result fields from:

macro result_header

Additional fields used in the paper.html <searx/templates/simple/result_templates/paper.html>:

content : :pystr: An abstract or excerpt from the document.
comments : :pystr: Free text display in italic below the content.
tags : :pyList <list>[:pystr]: Free tag list.
type : :pystr: Short description of medium type, e.g. book, pdf or html ...
authors : :pyList <list>[:pystr]: List of authors of the work (authors with a "s" suffix, the "author" is in the macro result_sub_header).
editor : :pystr: Editor of the book/paper.
publisher : :pystr: Name of the publisher.
journal : :pystr: Name of the journal or magazine the article was published in.
volume : :pystr: Volume number.
pages : :pystr: Page range where the article is.
number : :pystr: Number of the report or the issue number for a journal article.
doi : :pystr: DOI number (like 10.1038/d41586-018-07848-2).
issn : :pyList <list>[:pystr]: ISSN number like 1476-4687
isbn : :pyList <list>[:pystr]: ISBN number like 9780201896831
pdf_url : :pystr: URL to the full article, the PDF version
html_url : :pystr: URL to full article, HTML version

`packages`

Displays result fields from:

macro result_header

Additional fields used in the packages.html <searx/templates/simple/result_templates/packages.html>:

package_name : :pystr: The name of the package.
version : :pystr: The current version of the package.
maintainer : :pystr: The maintainer or author of the project.
publishedDate : :pydatetime <datetime.datetime>: Date of latest update or release.
tags : :pyList <list>[:pystr]: Free tag list.
popularity : :pystr: The popularity of the package, e.g. rating or download count.
license_name : :pystr: The name of the license.
license_url : :pystr: The web location of a license copy.
homepage : :pystr: The url of the project's homepage.
source_code_url: :pystr: The location of the project's source code.
links : :pydict: Additional links in the form of {'link_name': 'http://example.com'}

`code.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the code.html <searx/templates/simple/result_templates/code.html>:

content : :pystr: Description of the code fragment.
codelines : [line1, line2, ...]: Lines of the code fragment.
code_language : :pystr: Name of the code language, the value is passed to :pypygments.lexers.get_lexer_by_name.
repository : :pystr: URL of the repository of the code fragment.

`files.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the code.html <searx/templates/simple/result_templates/files.html>:

filename, size, time: :pystr: Filename, Filesize and Date of the file.
mtype : audio | video | :pystr: Mimetype type of the file.
subtype : :pystr: Mimetype / subtype of the file.
abstract : :pystr: Abstract of the file.
author : :pystr: Name of the author of the file
embedded : :pystr: URL of an embedded media type (audio or video) / is collapsible.

`products.html`

Displays result fields from:

macro result_header and
macro result_sub_header

Additional fields used in the products.html <searx/templates/simple/result_templates/products.html>:

content : :pystr: Description of the product.
price : :pystr: The price must include the currency.
shipping : :pystr: Shipping details.
source_country : :pystr: Place from which the shipment is made.

Answer results

See result_types.answer

Suggestion results

See result_types.suggestion

Correction results

See result_types.corrections

Infobox results

See result_types.infobox

14 KiB Raw Blame History

Simple Theme Templates

Result template macros

result_header

result_sub_header

engine_data_form

Main Result List

default.html

images.html

Image labels

videos.html

torrent.html

map.html

paper.html

packages

code.html

files.html

products.html

Answer results

Suggestion results

Correction results

Infobox results

14 KiB

Raw Blame History

`result_header`

`result_sub_header`

`engine_data_form`

`default.html`

`images.html`

`videos.html`

`torrent.html`

`map.html`

`paper.html`

`packages`

`code.html`

`files.html`

`products.html`