[doc] add blog post about SQL servers

Modified merge of [22a79a4] from searx.

[22a79a4] 22a79a4896

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2021-05-28 19:40:29 +02:00
parent 87a01a1736
commit e7a4d0961b
2 changed files with 161 additions and 0 deletions

View file

@ -13,3 +13,4 @@ Blog
private-engines private-engines
command-line-engines command-line-engines
search-indexer-engines search-indexer-engines
sql-engines

160
docs/blog/sql-engines.rst Normal file
View file

@ -0,0 +1,160 @@
===========
SQL engines
===========
.. sidebar:: further read
- `SQLite <https://www.sqlite.org/index.html>`_
- `PostgreSQL <https://www.postgresql.org>`_
- `MySQL <https://www.mysql.com>`_
With the *SQL engines* you can bind SQL databases into SearxNG. The following
Relational Database Management System (RDBMS) are supported:
- :ref:`engine sqlite`
- :ref:`engine postgresql`
- :ref:`engine mysql_server`
All of the engines above are just commented out in the :origin:`settings.yml
<searx/settings.yml>`, as you have to set the required attributes for the
engines, e.g. ``database:`` ...
.. code:: yaml
- name: ...
engine: {sqlite|postgresql|mysql_server}
database: ...
result_template: {template_name}
query_str: ...
By default, the engines use the ``key-value`` template for displaying results /
see :origin:`oscar <searx/templates/oscar/result_templates/key-value.html>` &
:origin:`simple <searx/templates/simple/result_templates/key-value.html>`
themes. If you are not satisfied with the original result layout, you can use
your own template, set ``result_template`` attribute to ``{template_name}`` and
place the templates at::
searx/templates/{theme_name}/result_templates/{template_name}
As mentioned in previous blog posts, if you do not wish to expose these engines
on a public instance, you can still add them and limit the access by setting
``tokens`` as described in section :ref:`private engines`.
Configure the engines
=====================
The configuration of the new database engines are similar. You must put a valid
SQL-SELECT query in ``query_str``. At the moment you can only bind at most one
parameter in your query. By setting the attribute ``limit`` you can define how
many results you want from the SQL server. Basically, it is the same as the
``LIMIT`` keyword in SQL.
Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines
rely on these keywords during paging. If you want to configure the number of
returned results use the option ``limit``.
.. _engine sqlite:
SQLite
------
.. _MediathekView: https://mediathekview.de/
SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency. To demonstrate the power of database engines, here is a
more complex example which reads from a MediathekView_ (DE) movie database. For
this example of the SQlite engine download the database:
- https://liste.mediathekview.de/filmliste-v2.db.bz2
and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g
Query to test: ``!mediathekview concert``
.. code:: yaml
- name: mediathekview
engine: sqlite
disabled: False
categories: general
result_template: default.html
database: searx/data/filmliste-v2.db
query_str: >-
SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
description AS content
FROM film
WHERE title LIKE :wildcard OR description LIKE :wildcard
ORDER BY duration DESC
Extra Dependencies
------------------
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
install additional packages in Python's Virtual Environment of your SearxNG
instance. To switch into the environment (:ref:`searx-src`) you can use
:ref:`searx.sh`::
$ sudo utils/searx.sh shell
(searx-pyenv)$ pip install ...
.. _engine postgresql:
PostgreSQL
----------
.. _psycopg2: https://www.psycopg.org/install
.. sidebar:: requirements
``pip install`` psycopg2_
PostgreSQL is a powerful and robust open source database. Before configuring
the PostgreSQL engine, you must install the dependency ``psychopg2``. You can
find an example configuration below:
.. code:: yaml
- name: my_database
engine: postgresql
database: my_database
username: searx
password: password
query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
.. _engine mysql_server:
MySQL
-----
.. _mysql-connector-python: https://pypi.org/project/mysql-connector-python
.. sidebar:: requirements
``pip install`` mysql-connector-python_
MySQL is said to be the most popular open source database. Before enabling MySQL
engine, you must install the package ``mysql-connector-python``.
The authentication plugin is configurable by setting ``auth_plugin`` in the
attributes. By default it is set to ``caching_sha2_password``. This is an
example configuration for quering a MySQL server:
.. code:: yaml
- name: my_database
engine: mysql_server
database: my_database
username: searx
password: password
limit: 5
query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .