[doc] rearranges Settings & Engines docs for better readability

We have built up detailed documentation of the *settings* and the *engines* over
the past few years.  However, this documentation was still spread over various
chapters and was difficult to navigate in its entirety.

This patch rearranges the Settings & Engines documentation for better
readability.

To review new ordered docs::

   make docs.clean docs.live

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2023-06-30 18:07:02 +02:00 committed by Markus Heiser
parent 8e8d8dabe9
commit 5720844fcd
84 changed files with 1715 additions and 1414 deletions

View file

@ -9,7 +9,7 @@ Buildhosts
If you have any contribution send us your :pull:`PR <../pulls>`, see If you have any contribution send us your :pull:`PR <../pulls>`, see
:ref:`how to contribute`. :ref:`how to contribute`.
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -1,79 +0,0 @@
.. _engine command:
====================
Command Line Engines
====================
.. sidebar:: info
- :origin:`command.py <searx/engines/command.py>`
- :ref:`offline engines`
With *command engines* administrators can run engines to integrate arbitrary
shell commands.
When creating and enabling a ``command`` engine on a public instance, you must
be careful to avoid leaking private data. The easiest solution is to limit the
access by setting ``tokens`` as described in section :ref:`private engines`.
The engine base is flexible. Only your imagination can limit the power of this
engine (and maybe security concerns). The following options are available:
``command``:
A comma separated list of the elements of the command. A special token
``{{QUERY}}`` tells where to put the search terms of the user. Example:
.. code:: yaml
['ls', '-l', '-h', '{{QUERY}}']
``delimiter``:
A mapping containing a delimiter ``char`` and the *titles* of each element in
``keys``.
``parse_regex``:
A dict containing the regular expressions for each result key.
``query_type``:
The expected type of user search terms. Possible values: ``path`` and
``enum``.
``path``:
Checks if the user provided path is inside the working directory. If not,
the query is not executed.
``enum``:
Is a list of allowed search terms. If the user submits something which is
not included in the list, the query returns an error.
``query_enum``:
A list containing allowed search terms if ``query_type`` is set to ``enum``.
``working_dir``:
The directory where the command has to be executed. Default: ``./``
``result_separator``:
The character that separates results. Default: ``\n``
The example engine below can be used to find files with a specific name in the
configured working directory:
.. code:: yaml
- name: find
engine: command
command: ['find', '.', '-name', '{{QUERY}}']
query_type: path
shortcut: fnd
delimiter:
chars: ' '
keys: ['line']
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View file

@ -1,26 +0,0 @@
.. _engines and settings:
==================
Engines & Settings
==================
.. sidebar:: Further reading ..
- :ref:`settings engine`
- :ref:`engine settings` & :ref:`engine file`
.. toctree::
:maxdepth: 3
settings
.. toctree::
:maxdepth: 1
private-engines
recoll
sql-engines
nosql-engines
search-indexer-engines
command-line-engines
searx.engines.xpath

View file

@ -1,49 +0,0 @@
.. _private engines:
============================
Private Engines (``tokens``)
============================
Administrators might find themselves wanting to limit access to some of the
enabled engines on their instances. It might be because they do not want to
expose some private information through :ref:`offline engines`. Or they would
rather share engines only with their trusted friends or colleagues.
To solve this issue the concept of *private engines* exists.
A new option was added to engines named `tokens`. It expects a list of
strings. If the user making a request presents one of the tokens of an engine,
they can access information about the engine and make search requests.
Example configuration to restrict access to the Arch Linux Wiki engine:
.. code:: yaml
- name: arch linux wiki
engine: archlinux
shortcut: al
tokens: [ 'my-secret-token' ]
Unless a user has configured the right token, the engine is going
to be hidden from him/her. It is not going to be included in the
list of engines on the Preferences page and in the output of
`/config` REST API call.
Tokens can be added to one's configuration on the Preferences page
under "Engine tokens". The input expects a comma separated list of
strings.
The distribution of the tokens from the administrator to the users
is not carved in stone. As providing access to such engines
implies that the admin knows and trusts the user, we do not see
necessary to come up with a strict process. Instead,
we would like to add guidelines to the documentation of the feature.
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View file

@ -1,50 +0,0 @@
.. _engine recoll:
=============
Recoll Engine
=============
.. sidebar:: info
- `Recoll <https://www.lesbonscomptes.com/recoll/>`_
- `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
- :origin:`searx/engines/recoll.py`
Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_
does not offer WEB or API access, this can be achieved using recoll-webui_
Configuration
=============
You must configure the following settings:
``base_url``:
Location where recoll-webui can be reached.
``mount_prefix``:
Location where the file hierarchy is mounted on your *local* filesystem.
``dl_prefix``:
Location where the file hierarchy as indexed by recoll can be reached.
``search_dir``:
Part of the indexed file hierarchy to be search, if empty the full domain is
searched.
Example
=======
Scenario:
#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
#. the Recoll search interface can be reached at https://recoll.example.org/ and
#. the contents of this filesystem can be reached though https://download.example.org/reference
.. code:: yaml
base_url: https://recoll.example.org/
mount_prefix: /export/documents
dl_prefix: https://download.example.org
search_dir: ''

View file

@ -1,136 +0,0 @@
====================
Local Search Engines
====================
.. sidebar:: further read
- `Comparison to alternatives
<https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
Administrators might find themselves wanting to integrate locally running search
engines. The following ones are supported for now:
* `Elasticsearch`_
* `Meilisearch`_
* `Solr`_
Each search engine is powerful, capable of full-text search. All of the engines
above are added to ``settings.yml`` just commented out, as you have to
``base_url`` for all them.
Please note that if you are not using HTTPS to access these engines, you have to enable
HTTP requests by setting ``enable_http`` to ``True``.
Furthermore, if you do not want to expose these engines on a public instance, you
can still add them and limit the access by setting ``tokens`` as described in
section :ref:`private engines`.
.. _engine meilisearch:
MeiliSearch
===========
.. sidebar:: info
- :origin:`meilisearch.py <searx/engines/meilisearch.py>`
- `MeiliSearch <https://www.meilisearch.com>`_
- `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
- `Install MeiliSearch
<https://docs.meilisearch.com/learn/getting_started/installation.html>`_
MeiliSearch_ is aimed at individuals and small companies. It is designed for
small-scale (less than 10 million documents) data collections. E.g. it is great
for storing web pages you have visited and searching in the contents later.
The engine supports faceted search, so you can search in a subset of documents
of the collection. Furthermore, you can search in MeiliSearch_ instances that
require authentication by setting ``auth_token``.
Here is a simple example to query a Meilisearch instance:
.. code:: yaml
- name: meilisearch
engine: meilisearch
shortcut: mes
base_url: http://localhost:7700
index: my-index
enable_http: true
.. _engine elasticsearch:
Elasticsearch
=============
.. sidebar:: info
- :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
- `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
- `Elasticsearch Guide
<https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
- `Install Elasticsearch
<https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
Elasticsearch_ supports numerous ways to query the data it is storing. At the
moment the engine supports the most popular search methods (``query_type``):
- ``match``,
- ``simple_query_string``,
- ``term`` and
- ``terms``.
If none of the methods fit your use case, you can select ``custom`` query type
and provide the JSON payload to submit to Elasticsearch in
``custom_query_json``.
The following is an example configuration for an Elasticsearch_ instance with
authentication configured to read from ``my-index`` index.
.. code:: yaml
- name: elasticsearch
shortcut: es
engine: elasticsearch
base_url: http://localhost:9200
username: elastic
password: changeme
index: my-index
query_type: match
# custom_query_json: '{ ... }'
enable_http: true
.. _engine solr:
Solr
====
.. sidebar:: info
- :origin:`solr.py <searx/engines/solr.py>`
- `Solr <https://solr.apache.org>`_
- `Solr Resources <https://solr.apache.org/resources.html>`_
- `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But
instead of searching in indices, you can search in collections.
This is an example configuration for searching in the collection
``my-collection`` and get the results in ascending order.
.. code:: yaml
- name: solr
engine: solr
shortcut: slr
base_url: http://localhost:8983
collection: my-collection
sort: asc
enable_http: true
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View file

@ -1,778 +0,0 @@
.. _settings.yml:
================
``settings.yml``
================
This page describe the options possibilities of the :origin:`searx/settings.yml`
file.
.. sidebar:: Further reading ..
- :ref:`use_default_settings.yml`
- :ref:`search API`
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _settings location:
settings.yml location
=====================
The initial ``settings.yml`` we be load from these locations:
1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
2. ``/etc/searxng/settings.yml``
If these files don't exist (or are empty or can't be read), SearXNG uses the
:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to
see how you can simplify your *user defined* ``settings.yml``.
.. _settings global:
Global Settings
===============
.. _settings brand:
``brand:``
----------
.. code:: yaml
brand:
issue_url: https://github.com/searxng/searxng/issues
docs_url: https://docs.searxng.org
public_instances: https://searx.space
wiki_url: https://github.com/searxng/searxng/wiki
``issue_url`` :
If you host your own issue tracker change this URL.
``docs_url`` :
If you host your own documentation change this URL.
``public_instances`` :
If you host your own https://searx.space change this URL.
``wiki_url`` :
Link to your wiki (or ``false``)
.. _settings general:
``general:``
------------
.. code:: yaml
general:
debug: false
instance_name: "SearXNG"
privacypolicy_url: false
donation_url: false
contact_url: false
enable_metrics: true
``debug`` : ``$SEARXNG_DEBUG``
Allow a more detailed log if you run SearXNG directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
``donation_url`` :
Set value to ``true`` to use your own donation page written in the
:ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
the donation link altogether.
``privacypolicy_url``:
Link to privacy policy.
``contact_url``:
Contact ``mailto:`` address or WEB form.
``enable_metrics``:
Enabled by default. Record various anonymous metrics availabled at ``/stats``,
``/stats/errors`` and ``/preferences``.
.. _settings search:
``search:``
-----------
.. code:: yaml
search:
safe_search: 0
autocomplete: ""
default_lang: ""
ban_time_on_fail: 5
max_ban_time_on_fail: 120
suspended_times:
SearxEngineAccessDenied: 86400
SearxEngineCaptcha: 86400
SearxEngineTooManyRequests: 3600
cf_SearxEngineCaptcha: 1296000
cf_SearxEngineAccessDenied: 86400
recaptcha_SearxEngineCaptcha: 604800
formats:
- html
``safe_search``:
Filter results.
- ``0``: None
- ``1``: Moderate
- ``2``: Strict
``autocomplete``:
Existing autocomplete backends, leave blank to turn it off.
- ``dbpedia``
- ``duckduckgo``
- ``google``
- ``startpage``
- ``swisscows``
- ``qwant``
- ``wikipedia``
``default_lang``:
Default search language - leave blank to detect from browser information or
use codes from :origin:`searx/languages.py`.
``languages``:
List of available languages - leave unset to use all codes from
:origin:`searx/languages.py`. Otherwise list codes of available languages.
The ``all`` value is shown as the ``Default language`` in the user interface
(in most cases, it is meant to send the query without a language parameter ;
in some cases, it means the English language) Example:
.. code:: yaml
languages:
- all
- en
- en-US
- de
- it-IT
- fr
- fr-BE
``ban_time_on_fail``:
Ban time in seconds after engine errors.
``max_ban_time_on_fail``:
Max ban time in seconds after engine errors.
``suspended_times``:
Engine suspension time after error (in seconds; set to 0 to disable)
``SearxEngineAccessDenied``: 86400
For error "Access denied" and "HTTP error [402, 403]"
``SearxEngineCaptcha``: 86400
For error "CAPTCHA"
``SearxEngineTooManyRequests``: 3600
For error "Too many request" and "HTTP error 429"
Cloudflare CAPTCHA:
- ``cf_SearxEngineCaptcha``: 1296000
- ``cf_SearxEngineAccessDenied``: 86400
Google CAPTCHA:
- ``recaptcha_SearxEngineCaptcha``: 604800
``formats``:
Result formats available from web, remove format to deny access (use lower
case).
- ``html``
- ``csv``
- ``json``
- ``rss``
.. _settings server:
``server:``
-----------
.. code:: yaml
server:
base_url: http://example.org/location # change this!
port: 8888
bind_address: "127.0.0.1"
secret_key: "ultrasecretkey" # change this!
limiter: false
image_proxy: false
default_http_headers:
X-Content-Type-Options : nosniff
X-XSS-Protection : 1; mode=block
X-Download-Options : noopen
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
The base URL where SearXNG is deployed. Used to create correct inbound links.
If you change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
Port number and *bind address* of the SearXNG web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG
services running behind a proxy and using socket communications. If you
change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``secret_key`` : ``$SEARXNG_SECRET``
Used for cryptography purpose.
.. _limiter:
``limiter`` :
Rate limit the number of request on the instance, block some bots. The
:ref:`limiter src` requires a :ref:`settings redis` database.
.. _image_proxy:
``image_proxy`` :
Allow your instance of SearXNG of being able to proxy images. Uses memory space.
.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
``default_http_headers`` :
Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
.. _settings ui:
``ui:``
-------
.. _cache busting:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
.. code:: yaml
ui:
static_use_hash: false
default_locale: ""
query_in_title: false
infinite_scroll: false
center_alignment: false
cache_url: https://web.archive.org/web/
default_theme: simple
theme_args:
simple_style: auto
.. _static_use_hash:
``static_use_hash`` :
Enables `cache busting`_ of static files.
``default_locale`` :
SearXNG interface language. If blank, the locale is detected by using the
browser language. If it doesn't work, or you are deploying a language
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
``query_in_title`` :
When true, the result page's titles contains the query it decreases the
privacy, since the browser can records the page titles.
``infinite_scroll``:
When true, automatically loads the next page when scrolling to bottom of the current page.
``center_alignment`` : default ``false``
When enabled, the results are centered instead of being in the left (or RTL)
side of the screen. This setting only affects the *desktop layout*
(:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
.. cache_url:
``cache_url`` : ``https://web.archive.org/web/``
URL prefix of the internet archive or cache, don't forgett trailing slash (if
needed). The default is https://web.archive.org/web/ alternatives are:
- https://webcache.googleusercontent.com/search?q=cache:
- https://archive.today/
``default_theme`` :
Name of the theme you want to use by default on your SearXNG instance.
``theme_args.simple_style``:
Style of simple theme: ``auto``, ``light``, ``dark``
``results_on_new_tab``:
Open result links in a new tab by default.
.. _settings redis:
``redis:``
----------
.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
will find a description to test your redis connection in SerXNG. When using
sockets, don't forget to check the access rights on the socket::
ls -la /usr/local/searxng-redis/run/redis.sock
srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
In this example read/write access is given to the *searxng-redis* group. To get
access rights to redis instance (the socket), your SearXNG (or even your
developer) account needs to be added to the *searxng-redis* group.
``url`` : ``$SEARXNG_REDIS_URL``
URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
redis://[[username]:[password]]@localhost:6379/0
rediss://[[username]:[password]]@localhost:6379/0
unix://[[username]:[password]]@/path/to/socket.sock?db=0
.. admonition:: Tip for developers
To set up a local redis instance, first set the socket path of the Redis DB
in your YAML setting:
.. code:: yaml
redis:
url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
Then use the following commands to install the redis instance ::
$ ./manage redis.build
$ sudo -H ./manage redis.install
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group
.. _settings outgoing:
``outgoing:``
-------------
Communication with search engines.
.. code:: yaml
outgoing:
request_timeout: 2.0 # default timeout in seconds, can be override by engine
max_request_timeout: 10.0 # the maximum timeout in seconds
useragent_suffix: "" # information like an email address to the administrator
pool_connections: 100 # Maximum number of allowable connections, or null
# for no limits. The default is 100.
pool_maxsize: 10 # Number of allowable keep-alive connections, or null
# to always allow. The default is 10.
enable_http2: true # See https://www.python-httpx.org/http2/
# uncomment below section if you want to use a custom server certificate
# see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
# and https://www.python-httpx.org/compatibility/#ssl-configuration
# verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
#
# uncomment below section if you want to use a proxyq see: SOCKS proxies
# https://2.python-requests.org/en/latest/user/advanced/#proxies
# are also supported: see
# https://2.python-requests.org/en/latest/user/advanced/#socks
#
# proxies:
# all://:
# - http://proxy1:8080
# - http://proxy2:8080
#
# using_tor_proxy: true
#
# Extra seconds to add in order to account for the time taken by the proxy
#
# extra_proxy_timeout: 10.0
#
``request_timeout`` :
Global timeout of the requests made to others engines in seconds. A bigger
timeout will allow to wait for answers from slow engines, but in consequence
will slow SearXNG reactivity (the result page may take the time specified in the
timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`.
``useragent_suffix`` :
Suffix to the user-agent SearXNG uses to send requests to others engines. If an
engine wish to block you, a contact info here may be useful to avoid that.
.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
``pool_maxsize``:
Number of allowable keep-alive connections, or ``null`` to always allow. The
default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_.
``pool_connections`` :
Maximum number of allowable connections, or ``null`` # for no limits. The
default is 100. See ``max_connections`` `Pool limit configuration`_.
``keepalive_expiry`` :
Number of seconds to keep a connection in the pool. By default 5.0 seconds.
See ``keepalive_expiry`` `Pool limit configuration`_.
.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
``proxies`` :
Define one or more proxies you wish to use, see `httpx proxies`_.
If there are more than one proxy for one protocol (http, https),
requests to the engines are distributed in a round-robin fashion.
``source_ips`` :
If you use multiple network interfaces, define from which IP the requests must
be made. Example:
* ``0.0.0.0`` any local IPv4 address.
* ``::`` any local IPv6 address.
* ``192.168.0.1``
* ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
* ``fe80::60a2:1691:e5a2:ee1f``
* ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
* ``[ 192.168.0.1, fe80::/126 ]``
``retries`` :
Number of retry in case of an HTTP error. On each retry, SearXNG uses an
different proxy and source ip.
``enable_http2`` :
Enable by default. Set to ``false`` to disable HTTP/2.
.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
Allow to specify a path to certificate.
see `httpx verification defaults`_.
In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
see `httpx ssl configuration`_.
``max_redirects`` :
30 by default. Maximum redirect before it is an error.
``using_tor_proxy`` :
Using tor proxy (``true``) or not (``false``) for all engines. The default is
``false`` and can be overwritten in the :ref:`settings engine`
.. _settings categories_as_tabs:
``categories_as_tabs:``
-----------------------
A list of the categories that are displayed as tabs in the user interface.
Categories not listed here can still be searched with the :ref:`search-syntax`.
.. code-block:: yaml
categories_as_tabs:
general:
images:
videos:
news:
map:
music:
it:
science:
files:
social media:
Engines are added to ``categories:`` (compare :ref:`engine categories`), the
categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If
there are no active engines in a category, the tab is not displayed (e.g. if a
user disables all engines in a category).
On the preferences page (``/preferences``) -- under *engines* -- there is an
additional tab, called *other*. In this tab are all engines listed that are not
in one of the UI tabs (not included in ``categories_as_tabs``).
.. _settings engine:
Engine settings
===============
.. sidebar:: Further reading ..
- :ref:`configured engines`
- :ref:`engines-dev`
In the code example below a *full fledged* example of a YAML setup from a dummy
engine is shown. Most of the options have a default value or even are optional.
.. code:: yaml
- name: example engine
engine: example
shortcut: demo
base_url: 'https://{language}.example.com/'
send_accept_language_header: false
categories: general
timeout: 3.0
api_key: 'apikey'
disabled: false
language: en_US
tokens: [ 'my-secret-token' ]
weight: 1
display_error_messages: true
about:
website: https://example.com
wikidata_id: Q306656
official_api_documentation: https://example.com/api-doc
use_official_api: true
require_api_key: true
results: HTML
# overwrite values from section 'outgoing:'
enable_http2: false
retries: 1
max_connections: 100
max_keepalive_connections: 10
keepalive_expiry: 5.0
using_tor_proxy: false
proxies:
http:
- http://proxy1:8080
- http://proxy2:8080
https:
- http://proxy1:8080
- http://proxy2:8080
- socks5://user:password@proxy3:1080
- socks5h://user:password@proxy4:1080
# other network settings
enable_http: false
retry_on_http_error: true # or 403 or [404, 429]
``name`` :
Name that will be used across SearXNG to define this engine. In settings, on
the result page...
``engine`` :
Name of the python file used to handle requests and responses to and from this
search engine.
``shortcut`` :
Code used to execute bang requests (in this case using ``!bi``)
``base_url`` : optional
Part of the URL that should be stable across every request. Can be useful to
use multiple sites using only one engine, or updating the site URL without
touching at the code.
``send_accept_language_header`` :
Several engines that support languages (or regions) deal with the HTTP header
``Accept-Language`` to build a response that fits to the locale. When this
option is activated, the language (locale) that is selected by the user is used
to build and send a ``Accept-Language`` header in the request to the origin
search engine.
.. _engine categories:
``categories`` : optional
Specifies to which categories the engine should be added. Engines can be
assigned to multiple categories.
Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
UI. A search in a tab (in the UI) will query all engines that are active in
this tab. In the preferences page (``/preferences``) -- under *engines* --
users can select what engine should be active when querying in this tab.
Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
in a category, regardless of whether they are active or not, or whether they
are in a tab of the UI or not. For example, ``!dictionaries`` can be used to
query all search engines in that category (group).
``timeout`` : optional
Timeout of the search with the current search engine. Overwrites
``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will
modify the global timeout of SearXNG.**
``api_key`` : optional
In a few cases, using an API needs the use of a secret key. How to obtain them
is described in the file.
``disabled`` : optional
To disable by default the engine, but not deleting it. It will allow the user
to manually activate it in the settings.
``inactive``: optional
Remove the engine from the settings (*disabled & removed*).
``language`` : optional
If you want to use another language for a specific engine, you can define it
by using the ISO code of language (and region), like ``fr``, ``en-US``,
``de-DE``.
``tokens`` : optional
A list of secret tokens to make this engine *private*, more details see
:ref:`private engines`.
``weight`` : default ``1``
Weighting of the results of this engine.
``display_error_messages`` : default ``true``
When an engine returns an error, the message is displayed on the user interface.
``network`` : optional
Use the network configuration from another engine.
In addition, there are two default networks:
- ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
- ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
``enable_http`` : optional
Enable HTTP for this engine (by default only HTTPS is enabled).
``retry_on_http_error`` : optional
Retry request on some HTTP status code.
Example:
* ``true`` : on HTTP status code between 400 and 599.
* ``403`` : on HTTP status code 403.
* ``[403, 429]``: on HTTP status code 403 and 429.
``proxies`` :
Overwrites proxy settings from :ref:`settings outgoing`.
``using_tor_proxy`` :
Using tor proxy (``true``) or not (``false``) for this engine. The default is
taken from ``using_tor_proxy`` of the :ref:`settings outgoing`.
``max_keepalive_connection#s`` :
`Pool limit configuration`_, overwrites value ``pool_maxsize`` from
:ref:`settings outgoing` for this engine.
``max_connections`` :
`Pool limit configuration`_, overwrites value ``pool_connections`` from
:ref:`settings outgoing` for this engine.
``keepalive_expiry`` :
`Pool limit configuration`_, overwrites value ``keepalive_expiry`` from
:ref:`settings outgoing` for this engine.
.. note::
A few more options are possible, but they are pretty specific to some
engines, and so won't be described here.
Example: Multilingual Search
----------------------------
SearXNG does not support true multilingual search. You have to use the language
prefix in your search query when searching in a different language.
But there is a workaround: By adding a new search engine with a different
language, SearXNG will search in your default and other language.
Example configuration in settings.yml for a German and English speaker:
.. code-block:: yaml
search:
default_lang : "de"
...
engines:
- name : google english
engine : google
language : en
...
When searching, the default google engine will return German results and
"google english" will return English results.
.. _settings use_default_settings:
use_default_settings
====================
.. sidebar:: ``use_default_settings: true``
- :ref:`settings location`
- :ref:`use_default_settings.yml`
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
The user defined ``settings.yml`` is loaded from the :ref:`settings location`
and can relied on the default configuration :origin:`searx/settings.yml` using:
``use_default_settings: true``
``server:``
In the following example, the actual settings are the default settings defined
in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
the ``bind_address``:
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
bind_address: "0.0.0.0"
``engines:``
With ``use_default_settings: true``, each settings can be override in a
similar way, the ``engines`` section is merged according to the engine
``name``. In this example, SearXNG will load all the default engines, will
enable the ``bing`` engine and define a :ref:`token <private engines>` for
the arch linux engine:
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
- name: bing
disabled: false
``engines:`` / ``remove:``
It is possible to remove some engines from the default settings. The following
example is similar to the above one, but SearXNG doesn't load the the google
engine:
.. code-block:: yaml
use_default_settings:
engines:
remove:
- google
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
``engines:`` / ``keep_only:``
As an alternative, it is possible to specify the engines to keep. In the
following example, SearXNG has only two engines:
.. code-block:: yaml
use_default_settings:
engines:
keep_only:
- google
- duckduckgo
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: google
tokens: ['$ecretValue']
- name: duckduckgo
tokens: ['$ecretValue']

View file

@ -4,8 +4,8 @@ Administrator documentation
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
settings/index
installation installation
installation-docker installation-docker
installation-scripts installation-scripts
@ -15,7 +15,6 @@ Administrator documentation
installation-apache installation-apache
update-searxng update-searxng
answer-captcha answer-captcha
engines/index
api api
architecture architecture
plugins plugins

View file

@ -61,7 +61,7 @@ section might give you some guidance.
- `Apache Fedora`_ - `Apache Fedora`_
- `Apache directives`_ - `Apache directives`_
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -41,7 +41,7 @@ section might give you some guidance.
- `uWSGI support from nginx`_ - `uWSGI support from nginx`_
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,7 +4,7 @@
Step by step installation Step by step installation
========================= =========================
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -73,7 +73,7 @@ Configuration
.. sidebar:: ``use_default_settings: True`` .. sidebar:: ``use_default_settings: True``
- :ref:`settings global` - :ref:`settings.yml`
- :ref:`settings location` - :ref:`settings location`
- :ref:`settings use_default_settings` - :ref:`settings use_default_settings`
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`

View file

@ -9,7 +9,7 @@ uWSGI
- `systemd.unit`_ - `systemd.unit`_
- `uWSGI Emperor`_ - `uWSGI Emperor`_
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -0,0 +1,25 @@
========
Settings
========
.. sidebar:: Further reading ..
- :ref:`engine settings`
- :ref:`engine file`
.. toctree::
:maxdepth: 2
settings
settings_engine
settings_brand
settings_general
settings_search
settings_server
settings_ui
settings_redis
settings_outgoing
settings_categories_as_tabs

View file

@ -0,0 +1,117 @@
.. _settings.yml:
================
``settings.yml``
================
This page describe the options possibilities of the :origin:`searx/settings.yml`
file.
.. sidebar:: Further reading ..
- :ref:`use_default_settings.yml`
- :ref:`search API`
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _settings location:
settings.yml location
=====================
The initial ``settings.yml`` we be load from these locations:
1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
2. ``/etc/searxng/settings.yml``
If these files don't exist (or are empty or can't be read), SearXNG uses the
:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to
see how you can simplify your *user defined* ``settings.yml``.
.. _settings use_default_settings:
use_default_settings
====================
.. sidebar:: ``use_default_settings: true``
- :ref:`settings location`
- :ref:`use_default_settings.yml`
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
The user defined ``settings.yml`` is loaded from the :ref:`settings location`
and can relied on the default configuration :origin:`searx/settings.yml` using:
``use_default_settings: true``
``server:``
In the following example, the actual settings are the default settings defined
in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
the ``bind_address``:
.. code:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
bind_address: "0.0.0.0"
``engines:``
With ``use_default_settings: true``, each settings can be override in a
similar way, the ``engines`` section is merged according to the engine
``name``. In this example, SearXNG will load all the default engines, will
enable the ``bing`` engine and define a :ref:`token <private engines>` for
the arch linux engine:
.. code:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
- name: bing
disabled: false
``engines:`` / ``remove:``
It is possible to remove some engines from the default settings. The following
example is similar to the above one, but SearXNG doesn't load the the google
engine:
.. code:: yaml
use_default_settings:
engines:
remove:
- google
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
``engines:`` / ``keep_only:``
As an alternative, it is possible to specify the engines to keep. In the
following example, SearXNG has only two engines:
.. code:: yaml
use_default_settings:
engines:
keep_only:
- google
- duckduckgo
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: google
tokens: ['$ecretValue']
- name: duckduckgo
tokens: ['$ecretValue']

View file

@ -0,0 +1,25 @@
.. _settings brand:
==========
``brand:``
==========
.. code:: yaml
brand:
issue_url: https://github.com/searxng/searxng/issues
docs_url: https://docs.searxng.org
public_instances: https://searx.space
wiki_url: https://github.com/searxng/searxng/wiki
``issue_url`` :
If you host your own issue tracker change this URL.
``docs_url`` :
If you host your own documentation change this URL.
``public_instances`` :
If you host your own https://searx.space change this URL.
``wiki_url`` :
Link to your wiki (or ``false``)

View file

@ -0,0 +1,31 @@
.. _settings categories_as_tabs:
=======================
``categories_as_tabs:``
=======================
A list of the categories that are displayed as tabs in the user interface.
Categories not listed here can still be searched with the :ref:`search-syntax`.
.. code:: yaml
categories_as_tabs:
general:
images:
videos:
news:
map:
music:
it:
science:
files:
social media:
Engines are added to ``categories:`` (compare :ref:`engine categories`), the
categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If
there are no active engines in a category, the tab is not displayed (e.g. if a
user disables all engines in a category).
On the preferences page (``/preferences``) -- under *engines* -- there is an
additional tab, called *other*. In this tab are all engines listed that are not
in one of the UI tabs (not included in ``categories_as_tabs``).

View file

@ -0,0 +1,244 @@
.. _settings engine:
===========
``engine:``
===========
.. sidebar:: Further reading ..
- :ref:`configured engines`
- :ref:`engines-dev`
In the code example below a *full fledged* example of a YAML setup from a dummy
engine is shown. Most of the options have a default value or even are optional.
.. hint::
A few more options are possible, but they are pretty specific to some
engines (:ref:`engine implementations`).
.. code:: yaml
- name: example engine
engine: example
shortcut: demo
base_url: 'https://{language}.example.com/'
send_accept_language_header: false
categories: general
timeout: 3.0
api_key: 'apikey'
disabled: false
language: en_US
tokens: [ 'my-secret-token' ]
weight: 1
display_error_messages: true
about:
website: https://example.com
wikidata_id: Q306656
official_api_documentation: https://example.com/api-doc
use_official_api: true
require_api_key: true
results: HTML
# overwrite values from section 'outgoing:'
enable_http2: false
retries: 1
max_connections: 100
max_keepalive_connections: 10
keepalive_expiry: 5.0
using_tor_proxy: false
proxies:
http:
- http://proxy1:8080
- http://proxy2:8080
https:
- http://proxy1:8080
- http://proxy2:8080
- socks5://user:password@proxy3:1080
- socks5h://user:password@proxy4:1080
# other network settings
enable_http: false
retry_on_http_error: true # or 403 or [404, 429]
``name`` :
Name that will be used across SearXNG to define this engine. In settings, on
the result page...
``engine`` :
Name of the python file used to handle requests and responses to and from this
search engine.
``shortcut`` :
Code used to execute bang requests (in this case using ``!bi``)
``base_url`` : optional
Part of the URL that should be stable across every request. Can be useful to
use multiple sites using only one engine, or updating the site URL without
touching at the code.
``send_accept_language_header`` :
Several engines that support languages (or regions) deal with the HTTP header
``Accept-Language`` to build a response that fits to the locale. When this
option is activated, the language (locale) that is selected by the user is used
to build and send a ``Accept-Language`` header in the request to the origin
search engine.
.. _engine categories:
``categories`` : optional
Specifies to which categories the engine should be added. Engines can be
assigned to multiple categories.
Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
UI. A search in a tab (in the UI) will query all engines that are active in
this tab. In the preferences page (``/preferences``) -- under *engines* --
users can select what engine should be active when querying in this tab.
Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
in a category, regardless of whether they are active or not, or whether they
are in a tab of the UI or not. For example, ``!dictionaries`` can be used to
query all search engines in that category (group).
``timeout`` : optional
Timeout of the search with the current search engine. Overwrites
``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will
modify the global timeout of SearXNG.**
``api_key`` : optional
In a few cases, using an API needs the use of a secret key. How to obtain them
is described in the file.
``disabled`` : optional
To disable by default the engine, but not deleting it. It will allow the user
to manually activate it in the settings.
``inactive``: optional
Remove the engine from the settings (*disabled & removed*).
``language`` : optional
If you want to use another language for a specific engine, you can define it
by using the ISO code of language (and region), like ``fr``, ``en-US``,
``de-DE``.
``tokens`` : optional
A list of secret tokens to make this engine *private*, more details see
:ref:`private engines`.
``weight`` : default ``1``
Weighting of the results of this engine.
``display_error_messages`` : default ``true``
When an engine returns an error, the message is displayed on the user interface.
``network`` : optional
Use the network configuration from another engine.
In addition, there are two default networks:
- ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
- ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
``enable_http`` : optional
Enable HTTP for this engine (by default only HTTPS is enabled).
``retry_on_http_error`` : optional
Retry request on some HTTP status code.
Example:
* ``true`` : on HTTP status code between 400 and 599.
* ``403`` : on HTTP status code 403.
* ``[403, 429]``: on HTTP status code 403 and 429.
``proxies`` :
Overwrites proxy settings from :ref:`settings outgoing`.
``using_tor_proxy`` :
Using tor proxy (``true``) or not (``false``) for this engine. The default is
taken from ``using_tor_proxy`` of the :ref:`settings outgoing`.
.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
``max_keepalive_connection#s`` :
`Pool limit configuration`_, overwrites value ``pool_maxsize`` from
:ref:`settings outgoing` for this engine.
``max_connections`` :
`Pool limit configuration`_, overwrites value ``pool_connections`` from
:ref:`settings outgoing` for this engine.
``keepalive_expiry`` :
`Pool limit configuration`_, overwrites value ``keepalive_expiry`` from
:ref:`settings outgoing` for this engine.
.. _private engines:
Private Engines (``tokens``)
============================
Administrators might find themselves wanting to limit access to some of the
enabled engines on their instances. It might be because they do not want to
expose some private information through :ref:`offline engines`. Or they would
rather share engines only with their trusted friends or colleagues.
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
To solve this issue the concept of *private engines* exists.
A new option was added to engines named `tokens`. It expects a list of strings.
If the user making a request presents one of the tokens of an engine, they can
access information about the engine and make search requests.
Example configuration to restrict access to the Arch Linux Wiki engine:
.. code:: yaml
- name: arch linux wiki
engine: archlinux
shortcut: al
tokens: [ 'my-secret-token' ]
Unless a user has configured the right token, the engine is going to be hidden
from him/her. It is not going to be included in the list of engines on the
Preferences page and in the output of `/config` REST API call.
Tokens can be added to one's configuration on the Preferences page under "Engine
tokens". The input expects a comma separated list of strings.
The distribution of the tokens from the administrator to the users is not carved
in stone. As providing access to such engines implies that the admin knows and
trusts the user, we do not see necessary to come up with a strict process.
Instead, we would like to add guidelines to the documentation of the feature.
Example: Multilingual Search
============================
SearXNG does not support true multilingual search. You have to use the language
prefix in your search query when searching in a different language.
But there is a workaround: By adding a new search engine with a different
language, SearXNG will search in your default and other language.
Example configuration in settings.yml for a German and English speaker:
.. code-block:: yaml
search:
default_lang : "de"
...
engines:
- name : google english
engine : google
language : en
...
When searching, the default google engine will return German results and
"google english" will return English results.

View file

@ -0,0 +1,34 @@
.. _settings general:
============
``general:``
============
.. code:: yaml
general:
debug: false
instance_name: "SearXNG"
privacypolicy_url: false
donation_url: false
contact_url: false
enable_metrics: true
``debug`` : ``$SEARXNG_DEBUG``
Allow a more detailed log if you run SearXNG directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
``donation_url`` :
Set value to ``true`` to use your own donation page written in the
:ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
the donation link altogether.
``privacypolicy_url``:
Link to privacy policy.
``contact_url``:
Contact ``mailto:`` address or WEB form.
``enable_metrics``:
Enabled by default. Record various anonymous metrics availabled at ``/stats``,
``/stats/errors`` and ``/preferences``.

View file

@ -0,0 +1,110 @@
.. _settings outgoing:
=============
``outgoing:``
=============
Communication with search engines.
.. code:: yaml
outgoing:
request_timeout: 2.0 # default timeout in seconds, can be override by engine
max_request_timeout: 10.0 # the maximum timeout in seconds
useragent_suffix: "" # information like an email address to the administrator
pool_connections: 100 # Maximum number of allowable connections, or null
# for no limits. The default is 100.
pool_maxsize: 10 # Number of allowable keep-alive connections, or null
# to always allow. The default is 10.
enable_http2: true # See https://www.python-httpx.org/http2/
# uncomment below section if you want to use a custom server certificate
# see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
# and https://www.python-httpx.org/compatibility/#ssl-configuration
# verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
#
# uncomment below section if you want to use a proxyq see: SOCKS proxies
# https://2.python-requests.org/en/latest/user/advanced/#proxies
# are also supported: see
# https://2.python-requests.org/en/latest/user/advanced/#socks
#
# proxies:
# all://:
# - http://proxy1:8080
# - http://proxy2:8080
#
# using_tor_proxy: true
#
# Extra seconds to add in order to account for the time taken by the proxy
#
# extra_proxy_timeout: 10.0
#
``request_timeout`` :
Global timeout of the requests made to others engines in seconds. A bigger
timeout will allow to wait for answers from slow engines, but in consequence
will slow SearXNG reactivity (the result page may take the time specified in the
timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`.
``useragent_suffix`` :
Suffix to the user-agent SearXNG uses to send requests to others engines. If an
engine wish to block you, a contact info here may be useful to avoid that.
.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
``pool_maxsize``:
Number of allowable keep-alive connections, or ``null`` to always allow. The
default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_.
``pool_connections`` :
Maximum number of allowable connections, or ``null`` # for no limits. The
default is 100. See ``max_connections`` `Pool limit configuration`_.
``keepalive_expiry`` :
Number of seconds to keep a connection in the pool. By default 5.0 seconds.
See ``keepalive_expiry`` `Pool limit configuration`_.
.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
``proxies`` :
Define one or more proxies you wish to use, see `httpx proxies`_.
If there are more than one proxy for one protocol (http, https),
requests to the engines are distributed in a round-robin fashion.
``source_ips`` :
If you use multiple network interfaces, define from which IP the requests must
be made. Example:
* ``0.0.0.0`` any local IPv4 address.
* ``::`` any local IPv6 address.
* ``192.168.0.1``
* ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
* ``fe80::60a2:1691:e5a2:ee1f``
* ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
* ``[ 192.168.0.1, fe80::/126 ]``
``retries`` :
Number of retry in case of an HTTP error. On each retry, SearXNG uses an
different proxy and source ip.
``enable_http2`` :
Enable by default. Set to ``false`` to disable HTTP/2.
.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
Allow to specify a path to certificate.
see `httpx verification defaults`_.
In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
see `httpx ssl configuration`_.
``max_redirects`` :
30 by default. Maximum redirect before it is an error.
``using_tor_proxy`` :
Using tor proxy (``true``) or not (``false``) for all engines. The default is
``false`` and can be overwritten in the :ref:`settings engine`

View file

@ -0,0 +1,43 @@
.. _settings redis:
==========
``redis:``
==========
.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
will find a description to test your redis connection in SerXNG. When using
sockets, don't forget to check the access rights on the socket::
ls -la /usr/local/searxng-redis/run/redis.sock
srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
In this example read/write access is given to the *searxng-redis* group. To get
access rights to redis instance (the socket), your SearXNG (or even your
developer) account needs to be added to the *searxng-redis* group.
``url`` : ``$SEARXNG_REDIS_URL``
URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
redis://[[username]:[password]]@localhost:6379/0
rediss://[[username]:[password]]@localhost:6379/0
unix://[[username]:[password]]@/path/to/socket.sock?db=0
.. admonition:: Tip for developers
To set up a local redis instance, first set the socket path of the Redis DB
in your YAML setting:
.. code:: yaml
redis:
url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
Then use the following commands to install the redis instance ::
$ ./manage redis.build
$ sudo -H ./manage redis.install
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group

View file

@ -0,0 +1,97 @@
.. _settings search:
===========
``search:``
===========
.. code:: yaml
search:
safe_search: 0
autocomplete: ""
default_lang: ""
ban_time_on_fail: 5
max_ban_time_on_fail: 120
suspended_times:
SearxEngineAccessDenied: 86400
SearxEngineCaptcha: 86400
SearxEngineTooManyRequests: 3600
cf_SearxEngineCaptcha: 1296000
cf_SearxEngineAccessDenied: 86400
recaptcha_SearxEngineCaptcha: 604800
formats:
- html
``safe_search``:
Filter results.
- ``0``: None
- ``1``: Moderate
- ``2``: Strict
``autocomplete``:
Existing autocomplete backends, leave blank to turn it off.
- ``dbpedia``
- ``duckduckgo``
- ``google``
- ``startpage``
- ``swisscows``
- ``qwant``
- ``wikipedia``
``default_lang``:
Default search language - leave blank to detect from browser information or
use codes from :origin:`searx/languages.py`.
``languages``:
List of available languages - leave unset to use all codes from
:origin:`searx/languages.py`. Otherwise list codes of available languages.
The ``all`` value is shown as the ``Default language`` in the user interface
(in most cases, it is meant to send the query without a language parameter ;
in some cases, it means the English language) Example:
.. code:: yaml
languages:
- all
- en
- en-US
- de
- it-IT
- fr
- fr-BE
``ban_time_on_fail``:
Ban time in seconds after engine errors.
``max_ban_time_on_fail``:
Max ban time in seconds after engine errors.
``suspended_times``:
Engine suspension time after error (in seconds; set to 0 to disable)
``SearxEngineAccessDenied``: 86400
For error "Access denied" and "HTTP error [402, 403]"
``SearxEngineCaptcha``: 86400
For error "CAPTCHA"
``SearxEngineTooManyRequests``: 3600
For error "Too many request" and "HTTP error 429"
Cloudflare CAPTCHA:
- ``cf_SearxEngineCaptcha``: 1296000
- ``cf_SearxEngineAccessDenied``: 86400
Google CAPTCHA:
- ``recaptcha_SearxEngineCaptcha``: 604800
``formats``:
Result formats available from web, remove format to deny access (use lower
case).
- ``html``
- ``csv``
- ``json``
- ``rss``

View file

@ -0,0 +1,54 @@
.. _settings server:
===========
``server:``
===========
.. code:: yaml
server:
base_url: http://example.org/location # change this!
port: 8888
bind_address: "127.0.0.1"
secret_key: "ultrasecretkey" # change this!
limiter: false
image_proxy: false
default_http_headers:
X-Content-Type-Options : nosniff
X-XSS-Protection : 1; mode=block
X-Download-Options : noopen
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
The base URL where SearXNG is deployed. Used to create correct inbound links.
If you change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
Port number and *bind address* of the SearXNG web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG
services running behind a proxy and using socket communications. If you
change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``secret_key`` : ``$SEARXNG_SECRET``
Used for cryptography purpose.
.. _limiter:
``limiter`` :
Rate limit the number of request on the instance, block some bots. The
:ref:`limiter src` requires a :ref:`settings redis` database.
.. _image_proxy:
``image_proxy`` :
Allow your instance of SearXNG of being able to proxy images. Uses memory space.
.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
``default_http_headers`` :
Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__

View file

@ -0,0 +1,62 @@
.. _settings ui:
=======
``ui:``
=======
.. _cache busting:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
.. code:: yaml
ui:
static_use_hash: false
default_locale: ""
query_in_title: false
infinite_scroll: false
center_alignment: false
cache_url: https://web.archive.org/web/
default_theme: simple
theme_args:
simple_style: auto
.. _static_use_hash:
``static_use_hash`` :
Enables `cache busting`_ of static files.
``default_locale`` :
SearXNG interface language. If blank, the locale is detected by using the
browser language. If it doesn't work, or you are deploying a language
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
``query_in_title`` :
When true, the result page's titles contains the query it decreases the
privacy, since the browser can records the page titles.
``infinite_scroll``:
When true, automatically loads the next page when scrolling to bottom of the current page.
``center_alignment`` : default ``false``
When enabled, the results are centered instead of being in the left (or RTL)
side of the screen. This setting only affects the *desktop layout*
(:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
.. cache_url:
``cache_url`` : ``https://web.archive.org/web/``
URL prefix of the internet archive or cache, don't forgett trailing slash (if
needed). The default is https://web.archive.org/web/ alternatives are:
- https://webcache.googleusercontent.com/search?q=cache:
- https://archive.today/
``default_theme`` :
Name of the theme you want to use by default on your SearXNG instance.
``theme_args.simple_style``:
Style of simple theme: ``auto``, ``light``, ``dark``
``results_on_new_tab``:
Open result links in a new tab by default.

View file

@ -9,7 +9,7 @@ SearXNG maintenance
- :ref:`toolboxing` - :ref:`toolboxing`
- :ref:`uWSGI maintenance` - :ref:`uWSGI maintenance`
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,7 +4,7 @@
How to contribute How to contribute
================= =================
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,6 +4,11 @@
Demo Offline Engine Demo Offline Engine
=================== ===================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.demo_offline .. automodule:: searx.engines.demo_offline
:members: :members:

View file

@ -4,6 +4,11 @@
Demo Online Engine Demo Online Engine
================== ==================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.demo_online .. automodule:: searx.engines.demo_online
:members: :members:

View file

@ -4,6 +4,11 @@
Engine Overview Engine Overview
=============== ===============
.. contents::
:depth: 3
:local:
:backlinks: entry
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine .. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
.. sidebar:: Further reading .. .. sidebar:: Further reading ..
@ -11,10 +16,6 @@ Engine Overview
- :ref:`configured engines` - :ref:`configured engines`
- :ref:`settings engine` - :ref:`settings engine`
.. contents::
:depth: 3
:backlinks: entry
SearXNG is a metasearch-engine_, so it uses different search engines to provide SearXNG is a metasearch-engine_, so it uses different search engines to provide
better results. better results.
@ -49,12 +50,12 @@ Engine File
categories list categories, in which the engine is working categories list categories, in which the engine is working
paging boolean support multiple pages paging boolean support multiple pages
time_range_support boolean support search time range time_range_support boolean support search time range
engine_type str - ``online`` :ref:`[ref] <demo online engine>` by engine_type str - ``online`` :ref:`[ref] <online engines>` by
default, other possibles values are: default, other possibles values are:
- ``offline`` :ref:`[ref] <offline engines>` - ``offline`` :ref:`[ref] <offline engines>`
- ``online_dictionary`` - ``online_dictionary`` :ref:`[ref] <online dictionary>`
- ``online_currency`` - ``online_currency`` :ref:`[ref] <online currency>`
- ``online_url_search`` - ``online_url_search`` :ref:`[ref] <online url search>`
======================= =========== ======================================================== ======================= =========== ========================================================
.. _engine settings: .. _engine settings:
@ -239,12 +240,18 @@ following parameters can be used to specify a search request:
.. _engine results: .. _engine results:
.. _engine media types: .. _engine media types:
Media Types Result Types (``template``)
=========== ===========================
Each result item of an engine can be of different media-types. Currently the Each result item of an engine can be of different media-types. Currently the
following media-types are supported. To set another media-type as ``default``, following media-types are supported. To set another media-type as
the parameter ``template`` must be set to the desired type. :ref:`template default`, the parameter ``template`` must be set to the desired
type.
.. _template default:
``default``
-----------
.. table:: Parameter of the **default** media type: .. table:: Parameter of the **default** media type:
:width: 100% :width: 100%
@ -259,6 +266,11 @@ the parameter ``template`` must be set to the desired type.
========================= ===================================================== ========================= =====================================================
.. _template images:
``images``
----------
.. table:: Parameter of the **images** media type: .. table:: Parameter of the **images** media type:
:width: 100% :width: 100%
@ -277,6 +289,11 @@ the parameter ``template`` must be set to the desired type.
========================= ===================================================== ========================= =====================================================
.. _template videos:
``videos``
----------
.. table:: Parameter of the **videos** media type: .. table:: Parameter of the **videos** media type:
:width: 100% :width: 100%
@ -292,6 +309,12 @@ the parameter ``template`` must be set to the desired type.
thumbnail string, url to a small-preview image thumbnail string, url to a small-preview image
========================= ===================================================== ========================= =====================================================
.. _template torrent:
``torrent``
-----------
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme .. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
.. table:: Parameter of the **torrent** media type: .. table:: Parameter of the **torrent** media type:
@ -315,6 +338,12 @@ the parameter ``template`` must be set to the desired type.
torrentfile string, torrentfile of the result torrentfile string, torrentfile of the result
========================= ===================================================== ========================= =====================================================
.. _template map:
``map``
-------
.. table:: Parameter of the **map** media type: .. table:: Parameter of the **map** media type:
:width: 100% :width: 100%
@ -342,6 +371,12 @@ the parameter ``template`` must be set to the desired type.
address.country country of object address.country country of object
========================= ===================================================== ========================= =====================================================
.. _template paper:
``paper``
---------
.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ .. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types .. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
@ -430,3 +465,4 @@ the parameter ``template`` must be set to the desired type.
* - html_url * - html_url
- :py:class:`str` - :py:class:`str`
- URL to full article, HTML version - URL to full article, HTML version

View file

@ -1,15 +1,20 @@
.. _searx.enginelib: .. _searx.enginelib:
============ ==============
Engine model Engine Library
============ ==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.enginelib .. automodule:: searx.enginelib
:members: :members:
.. _searx.enginelib.traits: .. _searx.enginelib.traits:
=============
Engine traits Engine traits
============= =============

View file

@ -0,0 +1,9 @@
.. _searx.engines loader:
========================
SearXNG's engines loader
========================
.. automodule:: searx.engines
:members:

View file

@ -0,0 +1,97 @@
.. _engine implementations:
======================
Engine Implementations
======================
Framework Components
====================
.. toctree::
:maxdepth: 2
enginelib
engines
engine_overview
Engine Types
============
The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine
determines which :ref:`search processor <searx.search.processors>` is used by
the engine.
In this section a list of the enignes that are documented is given, a complete
list of the engines can be found in the source under: :origin:`searx/engines`.
.. _online engines:
Online Engines
--------------
.. sidebar:: info
- :py:obj:`processors.online <searx.search.processors.online>`
.. toctree::
:maxdepth: 1
:glob:
demo/demo_online
xpath
online/*
.. _offline engines:
Offline Engines
---------------
.. sidebar:: info
- :py:obj:`processors.offline <searx.search.processors.offline>`
.. toctree::
:maxdepth: 1
:glob:
offline_concept
demo/demo_offline
offline/*
.. _online url search:
Online URL Search
-----------------
.. sidebar:: info
- :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>`
.. toctree::
:maxdepth: 1
:glob:
online_url_search/*
.. _online currency:
Online Currency
---------------
.. sidebar:: info
- :py:obj:`processors.online_currency <searx.search.processors.online_currency>`
*no engine of this type is documented yet / comming soon*
.. _online dictionary:
Online Dictionary
-----------------
.. sidebar:: info
- :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>`
*no engine of this type is documented yet / comming soon*

View file

@ -0,0 +1,23 @@
.. _engine command:
====================
Command Line Engines
====================
.. sidebar:: info
- :origin:`command.py <searx/engines/command.py>`
- :ref:`offline engines`
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
.. automodule:: searx.engines.command
:members:

View file

@ -1,3 +1,5 @@
.. _nosql engines:
=============== ===============
NoSQL databases NoSQL databases
=============== ===============
@ -8,6 +10,16 @@ NoSQL databases
- `redis.io <https://redis.io/>`_ - `redis.io <https://redis.io/>`_
- `MongoDB <https://www.mongodb.com>`_ - `MongoDB <https://www.mongodb.com>`_
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
The following `NoSQL databases`_ are supported: The following `NoSQL databases`_ are supported:
- :ref:`engine redis_server` - :ref:`engine redis_server`
@ -30,15 +42,8 @@ can still add them and limit the access by setting ``tokens`` as described in
section :ref:`private engines`. section :ref:`private engines`.
Configure the engines
=====================
`NoSQL databases`_ are used for storing arbitrary data without first defining
their structure.
Extra Dependencies Extra Dependencies
------------------ ==================
For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to
install additional packages in Python's Virtual Environment of your SearXNG install additional packages in Python's Virtual Environment of your SearXNG
@ -49,6 +54,13 @@ instance. To switch into the environment (:ref:`searxng-src`) you can use
(searxng-pyenv)$ pip install ... (searxng-pyenv)$ pip install ...
Configure the engines
=====================
`NoSQL databases`_ are used for storing arbitrary data without first defining
their structure.
.. _engine redis_server: .. _engine redis_server:
Redis Server Redis Server
@ -62,29 +74,9 @@ Redis Server
- redis.io_ - redis.io_
- :origin:`redis_server.py <searx/engines/redis_server.py>` - :origin:`redis_server.py <searx/engines/redis_server.py>`
.. automodule:: searx.engines.redis_server
:members:
Redis is an open source (BSD licensed), in-memory data structure (key value
based) store. Before configuring the ``redis_server`` engine, you must install
the dependency redis_.
Select a database to search in and set its index in the option ``db``. You can
either look for exact matches or use partial keywords to find what you are
looking for by configuring ``exact_match_only``. You find an example
configuration below:
.. code:: yaml
# Required dependency: redis
- name: myredis
shortcut : rds
engine: redis_server
exact_match_only: false
host: '127.0.0.1'
port: 6379
enable_http: true
password: ''
db: 0
.. _engine mongodb: .. _engine mongodb:
@ -99,37 +91,7 @@ MongoDB
- MongoDB_ - MongoDB_
- :origin:`mongodb.py <searx/engines/mongodb.py>` - :origin:`mongodb.py <searx/engines/mongodb.py>`
MongoDB_ is a document based database program that handles JSON like data.
Before configuring the ``mongodb`` engine, you must install the dependency
redis_.
In order to query MongoDB_, you have to select a ``database`` and a .. automodule:: searx.engines.mongodb
``collection``. Furthermore, you have to select a ``key`` that is going to be :members:
searched. MongoDB_ also supports the option ``exact_match_only``, so configure
it as you wish. Below is an example configuration for using a MongoDB
collection:
.. code:: yaml
# MongoDB engine
# Required dependency: pymongo
- name: mymongo
engine: mongodb
shortcut: md
exact_match_only: false
host: '127.0.0.1'
port: 27017
enable_http: true
results_per_page: 20
database: 'business'
collection: 'reviews' # name of the db collection
key: 'name' # key in the collection to search for
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View file

@ -0,0 +1,62 @@
=================
Local Search APIs
=================
.. sidebar:: further read
- `Comparison to alternatives
<https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
.. contents::
:depth: 1
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
Administrators might find themselves wanting to integrate locally running search
engines. The following ones are supported for now:
* `Elasticsearch`_
* `Meilisearch`_
* `Solr`_
Each search engine is powerful, capable of full-text search. All of the engines
above are added to ``settings.yml`` just commented out, as you have to
``base_url`` for all them.
Please note that if you are not using HTTPS to access these engines, you have to
enable HTTP requests by setting ``enable_http`` to ``True``.
Furthermore, if you do not want to expose these engines on a public instance,
you can still add them and limit the access by setting ``tokens`` as described
in section :ref:`private engines`.
.. _engine meilisearch:
MeiliSearch
===========
.. automodule:: searx.engines.meilisearch
:members:
.. _engine elasticsearch:
Elasticsearch
=============
.. automodule:: searx.engines.elasticsearch
:members:
.. _engine solr:
Solr
====
.. automodule:: searx.engines.solr
:members:

View file

@ -10,6 +10,16 @@ SQL Engines
- `PostgreSQL <https://www.postgresql.org>`_ - `PostgreSQL <https://www.postgresql.org>`_
- `MySQL <https://www.mysql.com>`_ - `MySQL <https://www.mysql.com>`_
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
With the *SQL engines* you can bind SQL databases into SearXNG. The following With the *SQL engines* you can bind SQL databases into SearXNG. The following
Relational Database Management System (RDBMS) are supported: Relational Database Management System (RDBMS) are supported:
@ -42,6 +52,18 @@ add them and limit the access by setting ``tokens`` as described in section
:ref:`private engines`. :ref:`private engines`.
Extra Dependencies
==================
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
install additional packages in Python's Virtual Environment of your SearXNG
instance. To switch into the environment (:ref:`searxng-src`) you can use
:ref:`searxng.sh`::
$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
Configure the engines Configure the engines
===================== =====================
@ -64,45 +86,8 @@ SQLite
- :origin:`sqlite.py <searx/engines/sqlite.py>` - :origin:`sqlite.py <searx/engines/sqlite.py>`
.. _MediathekView: https://mediathekview.de/ .. automodule:: searx.engines.sqlite
:members:
SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency. To demonstrate the power of database engines, here is a
more complex example which reads from a MediathekView_ (DE) movie database. For
this example of the SQlite engine download the database:
- https://liste.mediathekview.de/filmliste-v2.db.bz2
and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g
Query to test: ``!mediathekview concert``
.. code:: yaml
- name: mediathekview
engine: sqlite
disabled: False
categories: general
result_template: default.html
database: searx/data/filmliste-v2.db
query_str: >-
SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
description AS content
FROM film
WHERE title LIKE :wildcard OR description LIKE :wildcard
ORDER BY duration DESC
Extra Dependencies
------------------
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
install additional packages in Python's Virtual Environment of your SearXNG
instance. To switch into the environment (:ref:`searxng-src`) you can use
:ref:`searxng.sh`::
$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
.. _engine postgresql: .. _engine postgresql:
@ -115,20 +100,10 @@ PostgreSQL
.. sidebar:: info .. sidebar:: info
- :origin:`postgresql.py <searx/engines/postgresql.py>` - :origin:`postgresql.py <searx/engines/postgresql.py>`
- ``pip install`` psycopg2_ - ``pip install`` `psycopg2-binary <psycopg2>`_
PostgreSQL is a powerful and robust open source database. Before configuring .. automodule:: searx.engines.postgresql
the PostgreSQL engine, you must install the dependency ``psychopg2``. You can :members:
find an example configuration below:
.. code:: yaml
- name: my_database
engine: postgresql
database: my_database
username: searxng
password: password
query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
.. _engine mysql_server: .. _engine mysql_server:
@ -140,27 +115,7 @@ MySQL
- :origin:`mysql_server.py <searx/engines/mysql_server.py>` - :origin:`mysql_server.py <searx/engines/mysql_server.py>`
- ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>` - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>`
MySQL is said to be the most popular open source database. Before enabling MySQL
engine, you must install the package ``mysql-connector-python``.
The authentication plugin is configurable by setting ``auth_plugin`` in the .. automodule:: searx.engines.mysql_server
attributes. By default it is set to ``caching_sha2_password``. This is an :members:
example configuration for querying a MySQL server:
.. code:: yaml
- name: my_database
engine: mysql_server
database: my_database
username: searxng
password: password
limit: 5
query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View file

@ -1,15 +1,14 @@
.. _offline engines:
=============== ===============
Offline Engines Offline Concept
=============== ===============
.. sidebar:: offline engines .. sidebar:: offline engines
- :ref:`demo offline engine` - :ref:`demo offline engine`
- :ref:`sql engines`
- :ref:`engine command` - :ref:`engine command`
- :origin:`Redis <searx/engines/redis_server.py>` - :ref:`sql engines`
- :ref:`nosql engines`
- :py:obj:`searx.search.processors.offline`
To extend the functionality of SearXNG, offline engines are going to be To extend the functionality of SearXNG, offline engines are going to be
introduced. An offline engine is an engine which does not need Internet introduced. An offline engine is an engine which does not need Internet
@ -31,7 +30,6 @@ Programming Interface
in advance. in advance.
:py:func:`search(query, params) <searx.engines.demo_offline.searc>` :py:func:`search(query, params) <searx.engines.demo_offline.searc>`
Each offline engine has a function named ``search``. This function is Each offline engine has a function named ``search``. This function is
responsible to perform a search and return the results in a presentable responsible to perform a search and return the results in a presentable
format. (Where *presentable* means presentable by the selected result format. (Where *presentable* means presentable by the selected result
@ -69,10 +67,3 @@ administrators can set token(s) for each of the :ref:`private engines`. If a
query contains a valid token, then SearXNG performs the requested private query contains a valid token, then SearXNG performs the requested private
search. If not, requests from an offline engines return errors. search. If not, requests from an offline engines return errors.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .

View file

@ -0,0 +1,13 @@
.. _annas_archive engine:
==============
Anna's Archive
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.annas_archive
:members:

View file

@ -4,6 +4,11 @@
Arch Linux Arch Linux
========== ==========
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.archlinux .. automodule:: searx.engines.archlinux
:members: :members:

View file

@ -4,7 +4,7 @@
Bing Engines Bing Engines
============ ============
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,5 +4,10 @@
Dailymotion Dailymotion
=========== ===========
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.dailymotion .. automodule:: searx.engines.dailymotion
:members: :members:

View file

@ -1,10 +1,10 @@
.. _duckduckgo engines: .. _duckduckgo engines:
================= =================
DukcDukGo engines DukcDukGo Engines
================= =================
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,7 +4,7 @@
Google Engines Google Engines
============== ==============
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,7 +4,7 @@
Peertube Engines Peertube Engines
================ ================
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -0,0 +1,13 @@
.. _engine recoll:
=============
Recoll Engine
=============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.recoll
:members:

View file

@ -1,10 +1,10 @@
.. _startpage engines: .. _startpage engines:
================= =================
Startpage engines Startpage Engines
================= =================
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -0,0 +1,13 @@
.. _torznab engine:
==============
Torznab WebAPI
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.torznab
:members:

View file

@ -4,7 +4,7 @@
Wikimedia Wikimedia
========= =========
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,5 +4,10 @@
Yahoo Engine Yahoo Engine
============ ============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.yahoo .. automodule:: searx.engines.yahoo
:members: :members:

View file

@ -4,6 +4,11 @@
Tineye Tineye
====== ======
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.tineye .. automodule:: searx.engines.tineye
:members: :members:

View file

@ -4,6 +4,11 @@
XPath Engine XPath Engine
============ ============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.xpath .. automodule:: searx.engines.xpath
:members: :members:

View file

@ -4,12 +4,10 @@ Developer documentation
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
quickstart quickstart
contribution_guide contribution_guide
engine_overview engines/index
offline_engines
search_api search_api
plugins plugins
translation translation

View file

@ -22,7 +22,7 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in
section :ref:`internet connectivity docker`. section :ref:`internet connectivity docker`.
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -22,7 +22,7 @@ Calling the ``help`` target gives a first overview (``make help``):
.. program-output:: bash -c "cd ..; make --no-print-directory help" .. program-output:: bash -c "cd ..; make --no-print-directory help"
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -243,14 +243,14 @@ calling ``make clean`` stop all processes using the :ref:`make install` or
We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
Documentation <contrib docs>` section. If you want to edit the documentation Documentation <contrib docs>` section. If you want to edit the documentation
read our :ref:`make docs.live` section. If you are working in your own brand, read our :ref:`make docs.live` section. If you are working in your own brand,
adjust your :ref:`settings global`. adjust your :ref:`settings brand`.
.. _make docs.gh-pages: .. _make docs.gh-pages:
``make docs.gh-pages`` ``make docs.gh-pages``
====================== ======================
To deploy on github.io first adjust your :ref:`settings global`. For any To deploy on github.io first adjust your :ref:`settings brand`. For any
further read :ref:`deploy on github.io`. further read :ref:`deploy on github.io`.
.. _make test: .. _make test:

View file

@ -37,7 +37,7 @@ docs.live <make docs.live>` to build HTML while editing.
- DOT_, `Graphviz's dot`_, Graphviz_ - DOT_, `Graphviz's dot`_, Graphviz_
.. contents:: Contents .. contents::
:depth: 3 :depth: 3
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -9,7 +9,6 @@ developers.
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
update update
standalone_searx.py standalone_searx.py

View file

@ -43,7 +43,6 @@ If you don't trust anyone, you can set up your own, see :ref:`installation`.
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
user/index user/index
own-instance own-instance

View file

@ -7,7 +7,7 @@ Why use a private instance?
\.\. is a common question among SearXNG users. Before answering this \.\. is a common question among SearXNG users. Before answering this
question, see what options a SearXNG user has. question, see what options a SearXNG user has.
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -8,7 +8,6 @@ every item from the source code, but we will add documentation when requested.
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
:glob: :glob:
searx.* searx.*

View file

@ -4,7 +4,7 @@
Bot Detection Bot Detection
============= =============
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -1,2 +0,0 @@
.. automodule:: searx.engines.annas_archive
:members:

View file

@ -1,8 +0,0 @@
.. _searx.engines:
=================
SearXNG's engines
=================
.. automodule:: searx.engines
:members:

View file

@ -1,2 +0,0 @@
.. automodule:: searx.engines.torznab
:members:

View file

@ -4,7 +4,7 @@
Locales Locales
======= =======
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -4,7 +4,7 @@
Search processors Search processors
================= =================
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -34,7 +34,7 @@ Online currency processor
.. automodule:: searx.search.processors.online_currency .. automodule:: searx.search.processors.online_currency
:members: :members:
Online Dictionary processor Online dictionary processor
=========================== ===========================
.. automodule:: searx.search.processors.online_dictionary .. automodule:: searx.search.processors.online_dictionary

View file

@ -22,7 +22,7 @@ Configured Engines
called *tabs*), engines can be queried by their name or the categories they called *tabs*), engines can be queried by their name or the categories they
belong to, by using a :ref:`\!bing syntax <search-syntax>`. belong to, by using a :ref:`\!bing syntax <search-syntax>`.
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -2,7 +2,7 @@
User information User information
================ ================
.. contents:: Contents .. contents::
:depth: 3 :depth: 3
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -10,7 +10,6 @@ and developers.
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
searxng.sh searxng.sh
lxc.sh lxc.sh

View file

@ -26,7 +26,7 @@ to care about*).
- `LXC/LXD Image Server`_ - `LXC/LXD Image Server`_
- `LXD@github`_ - `LXD@github`_
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -15,7 +15,7 @@ script :origin:`utils/searxng.sh`.
- :ref:`installation nginx` - :ref:`installation nginx`
- :ref:`installation apache` - :ref:`installation apache`
.. contents:: Contents .. contents::
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View file

@ -1,18 +1,15 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""Engine related implementations """Implementations of the framework for the SearXNG engines.
.. note:: .. hint::
The long term goal is to modularize all relevant implementations to the The long term goal is to modularize all implementations of the engine
engines here in this Python package. In addition to improved modularization, framework here in this Python package. ToDo:
this will also be necessary in part because the probability of circular
imports will increase due to the increased typification of implementations in
the future.
ToDo: - move implementations of the :ref:`searx.engines loader` to a new module in
the :py:obj:`searx.enginelib` namespace.
- move :py:obj:`searx.engines.load_engine` to a new module `searx.enginelib`.
""" """
@ -36,7 +33,7 @@ class Engine: # pylint: disable=too-few-public-methods
# Common options in the engine module # Common options in the engine module
engine_type: str engine_type: str
"""Type of the engine (:origin:`searx/search/processors`)""" """Type of the engine (:ref:`searx.search.processors`)"""
paging: bool paging: bool
"""Engine supports multiple pages.""" """Engine supports multiple pages."""

View file

@ -1,8 +1,6 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""This module implements the engine loader. """Load and initialize the ``engines``, see :py:func:`load_engines` and register
Load and initialize the ``engines``, see :py:func:`load_engines` and register
:py:obj:`engine_shortcuts`. :py:obj:`engine_shortcuts`.
usage:: usage::

View file

@ -1,24 +1,12 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
""".. _annas_archive engine: """`Anna's Archive`_ is a free non-profit online shadow library metasearch
engine providing access to a variety of book resources (also via IPFS), created
============== by a team of anonymous archivists (AnnaArchivist_).
Anna's Archive
==============
.. _Anna's Archive: https://annas-archive.org/ .. _Anna's Archive: https://annas-archive.org/
.. _AnnaArchivist: https://annas-software.org/AnnaArchivist/annas-archive .. _AnnaArchivist: https://annas-software.org/AnnaArchivist/annas-archive
`Anna's Archive`_ is a free non-profit online shadow library metasearch engine
providing access to a variety of book resources (also via IPFS), created by a
team of anonymous archivists (AnnaArchivist_).
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
Configuration Configuration
============= =============
@ -41,7 +29,6 @@ for *newest* articles and journals (PDF) / by shortcut ``!aaa <search-term>``.
aa_ext: 'pdf' aa_ext: 'pdf'
aa_sort: 'newest' aa_sort: 'newest'
Implementations Implementations
=============== ===============

View file

@ -1,6 +1,77 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
""" """With *command engines* administrators can run engines to integrate arbitrary
Command (offline) shell commands.
.. attention::
When creating and enabling a ``command`` engine on a public instance, you
must be careful to avoid leaking private data.
The easiest solution is to limit the access by setting ``tokens`` as described
in section :ref:`private engines`. The engine base is flexible. Only your
imagination can limit the power of this engine (and maybe security concerns).
Configuration
=============
The following options are available:
``command``:
A comma separated list of the elements of the command. A special token
``{{QUERY}}`` tells where to put the search terms of the user. Example:
.. code:: yaml
['ls', '-l', '-h', '{{QUERY}}']
``delimiter``:
A mapping containing a delimiter ``char`` and the *titles* of each element in
``keys``.
``parse_regex``:
A dict containing the regular expressions for each result key.
``query_type``:
The expected type of user search terms. Possible values: ``path`` and
``enum``.
``path``:
Checks if the user provided path is inside the working directory. If not,
the query is not executed.
``enum``:
Is a list of allowed search terms. If the user submits something which is
not included in the list, the query returns an error.
``query_enum``:
A list containing allowed search terms if ``query_type`` is set to ``enum``.
``working_dir``:
The directory where the command has to be executed. Default: ``./``.
``result_separator``:
The character that separates results. Default: ``\\n``.
Example
=======
The example engine below can be used to find files with a specific name in the
configured working directory:
.. code:: yaml
- name: find
engine: command
command: ['find', '.', '-name', '{{QUERY}}']
query_type: path
shortcut: fnd
delimiter:
chars: ' '
keys: ['line']
Implementations
===============
""" """
import re import re

View file

@ -1,6 +1,44 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
""" """.. sidebar:: info
Elasticsearch
- :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
- `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
- `Elasticsearch Guide
<https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
- `Install Elasticsearch
<https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
Elasticsearch_ supports numerous ways to query the data it is storing. At the
moment the engine supports the most popular search methods (``query_type``):
- ``match``,
- ``simple_query_string``,
- ``term`` and
- ``terms``.
If none of the methods fit your use case, you can select ``custom`` query type
and provide the JSON payload to submit to Elasticsearch in
``custom_query_json``.
Example
=======
The following is an example configuration for an Elasticsearch_ instance with
authentication configured to read from ``my-index`` index.
.. code:: yaml
- name: elasticsearch
shortcut: es
engine: elasticsearch
base_url: http://localhost:9200
username: elastic
password: changeme
index: my-index
query_type: match
# custom_query_json: '{ ... }'
enable_http: true
""" """
from json import loads, dumps from json import loads, dumps

View file

@ -1,7 +1,35 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
""" """.. sidebar:: info
Meilisearch
- :origin:`meilisearch.py <searx/engines/meilisearch.py>`
- `MeiliSearch <https://www.meilisearch.com>`_
- `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
- `Install MeiliSearch
<https://docs.meilisearch.com/learn/getting_started/installation.html>`_
MeiliSearch_ is aimed at individuals and small companies. It is designed for
small-scale (less than 10 million documents) data collections. E.g. it is great
for storing web pages you have visited and searching in the contents later.
The engine supports faceted search, so you can search in a subset of documents
of the collection. Furthermore, you can search in MeiliSearch_ instances that
require authentication by setting ``auth_token``.
Example
=======
Here is a simple example to query a Meilisearch instance:
.. code:: yaml
- name: meilisearch
engine: meilisearch
shortcut: mes
base_url: http://localhost:7700
index: my-index
enable_http: true
""" """
# pylint: disable=global-statement # pylint: disable=global-statement

View file

@ -1,11 +1,53 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""MongoDB engine (Offline) """MongoDB_ is a document based database program that handles JSON like data.
Before configuring the ``mongodb`` engine, you must install the dependency
pymongo_.
Configuration
=============
In order to query MongoDB_, you have to select a ``database`` and a
``collection``. Furthermore, you have to select a ``key`` that is going to be
searched. MongoDB_ also supports the option ``exact_match_only``, so configure
it as you wish.
Example
=======
Below is an example configuration for using a MongoDB collection:
.. code:: yaml
# MongoDB engine
# Required dependency: pymongo
- name: mymongo
engine: mongodb
shortcut: md
exact_match_only: false
host: '127.0.0.1'
port: 27017
enable_http: true
results_per_page: 20
database: 'business'
collection: 'reviews' # name of the db collection
key: 'name' # key in the collection to search for
Implementations
===============
""" """
import re import re
from pymongo import MongoClient # pyright: ignore # pylint: disable=import-error
try:
from pymongo import MongoClient # type: ignore
except ImportError:
# import error is ignored because the admin has to install pymongo manually
# to use the engine
pass
engine_type = 'offline' engine_type = 'offline'

View file

@ -1,12 +1,37 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""MySQL database (offline) """MySQL is said to be the most popular open source database. Before enabling
MySQL engine, you must install the package ``mysql-connector-python``.
The authentication plugin is configurable by setting ``auth_plugin`` in the
attributes. By default it is set to ``caching_sha2_password``.
Example
=======
This is an example configuration for querying a MySQL server:
.. code:: yaml
- name: my_database
engine: mysql_server
database: my_database
username: searxng
password: password
limit: 5
query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
Implementations
===============
""" """
# import error is ignored because the admin has to install mysql manually to use try:
# the engine import mysql.connector # type: ignore
import mysql.connector # pyright: ignore # pylint: disable=import-error except ImportError:
# import error is ignored because the admin has to install mysql manually to use
# the engine
pass
engine_type = 'offline' engine_type = 'offline'
auth_plugin = 'caching_sha2_password' auth_plugin = 'caching_sha2_password'

View file

@ -1,12 +1,33 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""PostgreSQL database (offline) """PostgreSQL is a powerful and robust open source database. Before configuring
the PostgreSQL engine, you must install the dependency ``psychopg2``.
Example
=======
Below is an example configuration:
.. code:: yaml
- name: my_database
engine: postgresql
database: my_database
username: searxng
password: password
query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
Implementations
===============
""" """
# import error is ignored because the admin has to install mysql manually to use try:
# the engine import psycopg2 # type: ignore
import psycopg2 # pyright: ignore # pylint: disable=import-error except ImportError:
# import error is ignored because the admin has to install postgresql
# manually to use the engine.
pass
engine_type = 'offline' engine_type = 'offline'
host = "127.0.0.1" host = "127.0.0.1"

View file

@ -1,6 +1,51 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
""" # lint: pylint
Recoll (local search engine) """.. sidebar:: info
- `Recoll <https://www.lesbonscomptes.com/recoll/>`_
- `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
- :origin:`searx/engines/recoll.py`
Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_
does not offer WEB or API access, this can be achieved using recoll-webui_
Configuration
=============
You must configure the following settings:
``base_url``:
Location where recoll-webui can be reached.
``mount_prefix``:
Location where the file hierarchy is mounted on your *local* filesystem.
``dl_prefix``:
Location where the file hierarchy as indexed by recoll can be reached.
``search_dir``:
Part of the indexed file hierarchy to be search, if empty the full domain is
searched.
Example
=======
Scenario:
#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
#. the Recoll search interface can be reached at https://recoll.example.org/ and
#. the contents of this filesystem can be reached though https://download.example.org/reference
.. code:: yaml
base_url: https://recoll.example.org/
mount_prefix: /export/documents
dl_prefix: https://download.example.org
search_dir: ''
Implementations
===============
""" """
from datetime import date, timedelta from datetime import date, timedelta
@ -33,7 +78,7 @@ embedded_url = '<{ttype} controls height="166px" ' + 'src="{url}" type="{mtype}"
# helper functions # helper functions
def get_time_range(time_range): def get_time_range(time_range):
sw = {'day': 1, 'week': 7, 'month': 30, 'year': 365} sw = {'day': 1, 'week': 7, 'month': 30, 'year': 365} # pylint: disable=invalid-name
offset = sw.get(time_range, 0) offset = sw.get(time_range, 0)
if not offset: if not offset:

View file

@ -1,6 +1,37 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""Redis engine (offline) """Redis is an open source (BSD licensed), in-memory data structure (key value
based) store. Before configuring the ``redis_server`` engine, you must install
the dependency redis_.
Configuration
=============
Select a database to search in and set its index in the option ``db``. You can
either look for exact matches or use partial keywords to find what you are
looking for by configuring ``exact_match_only``.
Example
=======
Below is an example configuration:
.. code:: yaml
# Required dependency: redis
- name: myredis
shortcut : rds
engine: redis_server
exact_match_only: false
host: '127.0.0.1'
port: 6379
enable_http: true
password: ''
db: 0
Implementations
===============
""" """

View file

@ -1,7 +1,31 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
""" """.. sidebar:: info
Solr
- :origin:`solr.py <searx/engines/solr.py>`
- `Solr <https://solr.apache.org>`_
- `Solr Resources <https://solr.apache.org/resources.html>`_
- `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But
instead of searching in indices, you can search in collections.
Example
=======
This is an example configuration for searching in the collection
``my-collection`` and get the results in ascending order.
.. code:: yaml
- name: solr
engine: solr
shortcut: slr
base_url: http://localhost:8983
collection: my-collection
sort: asc
enable_http: true
""" """
# pylint: disable=global-statement # pylint: disable=global-statement

View file

@ -1,7 +1,40 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
"""SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency.
"""SQLite database (Offline) Example
=======
.. _MediathekView: https://mediathekview.de/
To demonstrate the power of database engines, here is a more complex example
which reads from a MediathekView_ (DE) movie database. For this example of the
SQlite engine download the database:
- https://liste.mediathekview.de/filmliste-v2.db.bz2
and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g
Query to test: ``!mediathekview concert``
.. code:: yaml
- name: mediathekview
engine: sqlite
disabled: False
categories: general
result_template: default.html
database: searx/data/filmliste-v2.db
query_str: >-
SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
description AS content
FROM film
WHERE title LIKE :wildcard OR description LIKE :wildcard
ORDER BY duration DESC
Implementations
===============
""" """
@ -26,14 +59,15 @@ def init(engine_settings):
@contextlib.contextmanager @contextlib.contextmanager
def sqlite_cursor(): def sqlite_cursor():
"""Implements a `Context Manager`_ for a :py:obj:`sqlite3.Cursor`. """Implements a :py:obj:`Context Manager <contextlib.contextmanager>` for a
:py:obj:`sqlite3.Cursor`.
Open database in read only mode: if the database doesn't exist. Open database in read only mode: if the database doesn't exist. The default
The default mode creates an empty file on the file system. mode creates an empty file on the file system. See:
see:
* https://docs.python.org/3/library/sqlite3.html#sqlite3.connect * https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
* https://www.sqlite.org/uri.html * https://www.sqlite.org/uri.html
""" """
uri = 'file:' + database + '?mode=ro' uri = 'file:' + database + '?mode=ro'
with contextlib.closing(sqlite3.connect(uri, uri=True)) as connect: with contextlib.closing(sqlite3.connect(uri, uri=True)) as connect:

View file

@ -1,17 +1,6 @@
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
# lint: pylint # lint: pylint
""".. _torznab engine: """Torznab_ is an API specification that provides a standardized way to query
==============
Torznab WebAPI
==============
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
Torznab_ is an API specification that provides a standardized way to query
torrent site for content. It is used by a number of torrent applications, torrent site for content. It is used by a number of torrent applications,
including Prowlarr_ and Jackett_. including Prowlarr_ and Jackett_.
@ -55,7 +44,6 @@ The engine has the following settings:
.. _Jackett-categories: .. _Jackett-categories:
https://github.com/Jackett/Jackett/wiki/Jackett-Categories https://github.com/Jackett/Jackett/wiki/Jackett-Categories
Implementations Implementations
=============== ===============

View file

@ -3,8 +3,55 @@
"""The XPath engine is a *generic* engine with which it is possible to configure """The XPath engine is a *generic* engine with which it is possible to configure
engines in the settings. engines in the settings.
Here is a simple example of a XPath engine configured in the .. _XPath selector: https://quickref.me/xpath.html#xpath-selectors
:ref:`settings engine` section, further read :ref:`engines-dev`.
Configuration
=============
Request:
- :py:obj:`search_url`
- :py:obj:`lang_all`
- :py:obj:`soft_max_redirects`
- :py:obj:`cookies`
- :py:obj:`headers`
Paging:
- :py:obj:`paging`
- :py:obj:`page_size`
- :py:obj:`first_page_num`
Time Range:
- :py:obj:`time_range_support`
- :py:obj:`time_range_url`
- :py:obj:`time_range_map`
Safe-Search:
- :py:obj:`safe_search_support`
- :py:obj:`safe_search_map`
Response:
- :py:obj:`no_result_for_http_status`
`XPath selector`_:
- :py:obj:`results_xpath`
- :py:obj:`url_xpath`
- :py:obj:`title_xpath`
- :py:obj:`content_xpath`
- :py:obj:`thumbnail_xpath`
- :py:obj:`suggestion_xpath`
Example
=======
Here is a simple example of a XPath engine configured in the :ref:`settings
engine` section, further read :ref:`engines-dev`.
.. code:: yaml .. code:: yaml
@ -16,6 +63,9 @@ Here is a simple example of a XPath engine configured in the
title_xpath : //article[@class="repo-summary"]//a[@class="repo-link"] title_xpath : //article[@class="repo-summary"]//a[@class="repo-link"]
content_xpath : //article[@class="repo-summary"]/p content_xpath : //article[@class="repo-summary"]/p
Implementations
===============
""" """
from urllib.parse import urlencode from urllib.parse import urlencode
@ -26,7 +76,7 @@ from searx.network import raise_for_httperror
search_url = None search_url = None
""" """
Search URL of the engine. Example:: Search URL of the engine. Example::
https://example.org/?search={query}&page={pageno}{time_range}{safe_search} https://example.org/?search={query}&page={pageno}{time_range}{safe_search}
@ -74,30 +124,33 @@ soft_max_redirects = 0
'''Maximum redirects, soft limit. Record an error but don't stop the engine''' '''Maximum redirects, soft limit. Record an error but don't stop the engine'''
results_xpath = '' results_xpath = ''
'''XPath selector for the list of result items''' '''`XPath selector`_ for the list of result items'''
url_xpath = None url_xpath = None
'''XPath selector of result's ``url``.''' '''`XPath selector`_ of result's ``url``.'''
content_xpath = None content_xpath = None
'''XPath selector of result's ``content``.''' '''`XPath selector`_ of result's ``content``.'''
title_xpath = None title_xpath = None
'''XPath selector of result's ``title``.''' '''`XPath selector`_ of result's ``title``.'''
thumbnail_xpath = False thumbnail_xpath = False
'''XPath selector of result's ``img_src``.''' '''`XPath selector`_ of result's ``img_src``.'''
suggestion_xpath = '' suggestion_xpath = ''
'''XPath selector of result's ``suggestion``.''' '''`XPath selector`_ of result's ``suggestion``.'''
cached_xpath = '' cached_xpath = ''
cached_url = '' cached_url = ''
cookies = {} cookies = {}
'''Some engines might offer different result based on cookies.
Possible use-case: To set safesearch cookie.'''
headers = {} headers = {}
'''Some engines might offer different result based on cookies or headers. '''Some engines might offer different result based headers. Possible use-case:
Possible use-case: To set safesearch cookie or header to moderate.''' To set header to moderate.'''
paging = False paging = False
'''Engine supports paging [True or False].''' '''Engine supports paging [True or False].'''