Commit graph

499 commits

Author SHA1 Message Date
Ember
cb945276b6 Change 'his/her' to 'them' 2024-06-13 10:23:01 +02:00
Markus Heiser
845a0b678d [doc] add 'hostnames' plugin to the online documentation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-06-07 14:42:52 +02:00
Bnyro
3bec04079c [feat] hostname replace plugin: possibility to prioritize certain websites
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2024-06-07 14:42:52 +02:00
allendema_searxng_pi
ee146dbc07 [enh] Add engine for discourse forums 2024-06-07 10:16:09 +02:00
Daniel Kukula
cc8b537e34 [mod] package.html template: additional links (a python dict)
- Closes: https://github.com/searxng/searxng/issues/3456
2024-05-15 12:50:35 +02:00
Bnyro
82b6c0d05f [feat] engine: implementation of gitea 2024-05-15 07:23:57 +02:00
Markus Heiser
742303d030 [mod] improve unit converter plugin
- l10n support: parse and format decimal numbers by babel
- ability to add additional units
- improved unit detection (symbols are not unique)
- support for alias units (0,010C to F --> 32,018 °F)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-05-09 17:16:31 +02:00
Markus Heiser
542f7d0d7b [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.

- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two

  1. ./searx/engines -> lint engines with additional builtins
  2. ./searx ./searxng_extra ./tests -> lint all other python files

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11 14:55:38 +01:00
Markus Heiser
707d6270c8 [doc] engine: mullvad leta
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-10 18:20:07 +01:00
Bnyro
f3b4bf86a7 [feat] engine: implementation of void linux packages
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2024-02-29 13:12:40 +01:00
Bnyro
e76ab1a4b3 [refactor] images: add resolution, image format and filesize fields
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2024-02-25 16:22:37 +01:00
Bnyro
60d8414ef1 [docs] engine_overview: add packages result template 2024-02-25 14:56:57 +01:00
Alexandre Flament
ed66ed758d [mod] reduce memory footprint by not calling babel.Locale.parse at runtime
babel.Locale.parse loads more than 60MB in RAM.  The only purpose is to get:

    LOCALE_NAMES   - searx.data.LOCALES["LOCALE_NAMES"]
    RTL_LOCALES    - searx.data.LOCALES["RTL_LOCALES"]

This commit calls babel.Locale.parse when the translations are update from
weblate and stored in::

    searx/data/locales.json

This file can be build by::

    ./manage data.locales

By store these variables in searx.data when the translations are updated we save
round about 65MB (usually 4 worker = 260MB of RAM saved.

Suggested-by: https://github.com/searxng/searxng/discussions/2633#discussioncomment-8490494
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2024-02-20 10:43:20 +01:00
Markus Heiser
c197c0e35e [fix] remove twine from requirements-dev
SearXNG is a rolling release / we do not deploy packages on PyPi

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-02-16 18:29:40 +01:00
Markus Heiser
ab8e5383fb [mod] remove X-XSS-Protection headers
Deprecated header not used by browsers nowadays[1]:

"""In modern browsers, X-XSS-Protection has been deprecated in favor of the
Content-Security-Policy to disable the use of inline JavaScript. Its use can
introduce XSS vulnerabilities in otherwise safe websites. This should not be
used unless you need to support older web browsers that don’t yet support CSP.
It is thus recommended to set the header as X-XSS-Protection: 0."""[2]

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection
[2] https://infosec.mozilla.org/guidelines/web_security#x-xss-protection

Closes: https://github.com/searxng/searxng/issues/3171
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-31 17:23:41 +01:00
Markus Heiser
e560d7e373 [mod] presearch: add language & region support
In Presearch there are languages for the UI and regions for narrowing down the
search.  With this change the SearXNG engine supports a search by region.  The
details can be found in the documentation of the source code.

To test, you can search terms like::

   !presearch bmw :zh-TW
   !presearch bmw :en-CA

1. You should get results corresponding to the region (Taiwan, Canada)
2. and in the language (Chinese, Englisch).
3. The context in info box content is in the same language.

Exceptions:

1. Region or language is not supported by Presearch or
2. SearXNG user did not selected a region tag, example::

    !presearch bmw :en

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-15 19:23:26 +01:00
Markus Heiser
f9c5727ddc [mod] get rid of ./utils/brand.env and its workflow
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-09 16:31:19 +01:00
Bnyro
3dea7e609b [feat] autocompleter: implementation of stract (beta) 2024-01-07 11:18:16 +01:00
Bnyro
bf75a8c2a0 [feat] engine: implementation of bpb 2023-11-27 16:46:41 +01:00
Markus Heiser
76b91a3ef6 [dev] manage runtime versions with asdf
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-12 20:54:57 +01:00
Alexandre Flament
bd3f526859
Docker: add UWSGI_WORKERS and UWSGI_THREAD environment variables (#2992)
* Docker: add UWSGI_WORKERS and UWSGI_THREAD.

UWSGI_WORKERS specifies the number of process.
UWSGI_THREADS specifies the number of threads.

The Docker convention is to specify the whole configuration
through environment variables. While not done in SearXNG, these two
additional variables allows admins to skip uwsgi.ini

In additional, https://github.com/searxng/preview-environments starts Docker
without additional files through searxng-helm-chat.
Each instance consumes 1Go of RAM which is a lot especially when there are a
lot of instances / pull requests.

* [scripts] add environments UWSGI_WORKERS and UWSGI_THREADS

- UWSGI_WORKERS specifies the number of process.
- UWSGI_THREADS specifies the number of threads.

Templates for uwsgi scripts can be tested by::

    UWSGI_WORKERS=8 UWSGI_THREADS=9 \
      ./utils/searxng.sh --cmd\
      eval "echo \"$(cat utils/templates/etc/uwsgi/*/searxng.ini*)\""\
      | grep "workers\|threads"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

---------

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-12 16:46:34 +00:00
Markus Heiser
d13a8f6453 [mod] document server:public_instance & remove it out of the botdetection
- the option server:public_instance lacks some documentation
- the processing of this option belongs in the limiter and not
  in botdetection module

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
Markus Heiser
fd814aac86 [mod] isolation of botdetection from the limiter
This patch was inspired by the discussion around PR-2882 [2].  The goals of this
patch are:

1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
   enabled (Redis is available).

This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.

Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin.  The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.

[1] https://github.com/searxng/searxng/pull/2882
[2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891

To test:

- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
  - http://0.0.0.0:8000/admin/searx.limiter.html
  - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
Alex Balgavy
5d53aac20f [mod] add hotkeys option to settings.yml
The change in the hotkey mechanism introduced in 317db5b04 does not allow
configuration via `settings.yml`.  This commit adds that functionality.

Closes: #2898
2023-10-09 18:13:00 +02:00
Aine
213cb74378 [fix] matrixrooms add proper MRS integration
Related:

- https://github.com/searxng/searxng/issues/2918
2023-10-09 13:25:13 +02:00
Bnyro
48cb58bd2e [feat] duckduckgo: support for videos and news 2023-10-09 06:53:43 +02:00
Bnyro
ce270961e8 [feat] engine: implementation of mastodon 2023-10-06 10:58:23 +02:00
Markus Heiser
fd1422a670 [mod] engine - simplify region & lang handling, make filters configurable
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-05 10:55:08 +02:00
Bnyro
26fed56d51 [mod] settings.yml: remove plugin settings for plugins that don't exist anymore 2023-09-29 11:26:49 +02:00
Markus Heiser
597c68b4aa [doc] move dosc of botdetection from developer to admin section
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-23 14:00:03 +02:00
Bnyro
c999cfb422 [feat] engine: implementation of wallhaven 2023-09-21 14:25:43 +02:00
Bnyro
a55e0ac553 [feat] search on category select without JS
Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-09-18 21:29:11 +02:00
jazzzooo
223b3487c3 [fix] spelling 2023-09-18 16:20:27 +02:00
Markus Heiser
d9dbcedeb6 [feat] implementation of qwant lite for web search
Related: https://github.com/searxng/searxng/issues/2719
Replace: https://github.com/searxng/searxng/pull/2748
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 16:53:25 +02:00
Bnyro
b4e0d2eedc [feat] engine: implemenation of moviepilot (de) 2023-09-17 14:30:56 +02:00
Bnyro
f182abd6f8 [mod] library of congress: fix engine 2023-09-11 19:42:31 +02:00
Hackurei
4da7003ae0 [feat] engine: implementation of odysee 2023-09-02 09:14:12 +02:00
Markus Heiser
b0d2cd5ca9 [doc] add documentation of Mwmbl engine & autocompleter
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-27 17:25:26 +02:00
Bnyro
df71c24b20 [mod] autocomplete.py: add support for mwmbl completions 2023-08-27 17:25:26 +02:00
Markus Heiser
1e2d11fb57 [dev-env] upgrade Sphinx-doc 7.2.3 and unpin at v7.1.2 on py3.8
- Sphinx-doc 7.2.0 drops py3.8 support [1][2]
- last version with py3.8 support is 7.1.2

Many LTS distributions still have py3.8 which EOL is in 2024-10 [3].

To continue to support a development environment on py3.8 the rigid dependency
in the development environment is unpinned in py3.8 / environment markers [4].

To get 7.2.3. work, a fix in sphinx-notfound-page is needed [5][6].

[1] https://github.com/searxng/searxng/pull/2658#issuecomment-1684867270
[2] https://github.com/sphinx-doc/sphinx/issues/11621
[3] https://devguide.python.org/versions/#supported-versions
[4] https://peps.python.org/pep-0508/#environment-markers
[5] https://github.com/readthedocs/sphinx-notfound-page/issues/219
[6] https://github.com/readthedocs/sphinx-notfound-page/issues/219#issuecomment-1694691135

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-27 17:22:45 +02:00
Markus Heiser
9100a48541 [mod] improve seekr engines and add documentation
Tis patch adds some more fields to the result items and changed paging to the
``nextResultSet`` given in seekr's JSON response.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-15 16:17:42 +02:00
ToxyFlog1627
f175574f37 [fix] typos in documentation & messages 2023-08-13 08:50:29 +02:00
Markus Heiser
905ce2a6f6 [doc] add tagesschau API to the debveloper documentation
supplement to the commit e25d1c728

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-11 12:31:03 +02:00
Markus Heiser
01c0ec5d19 [fix] typo in docs/admin/settings/settings_general.rst (doc)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-11 11:07:38 +02:00
Bnyro
224f2250ae [feat] engine: support for lemmy communities, posts, comments and users 2023-08-10 12:58:40 +02:00
Markus Heiser
460bbe5b81 [mod] implement brave (WEB) engine to replace XPath configuration
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-08 16:21:45 +02:00
Markus Heiser
64100db904 [doc] improve documentation of make targets and ./manage script
BTW force modularization of the ./mange script into sub modules:

- utils/lib_sxng_data.sh
- utils/lib_sxng_node.sh
- utils/lib_sxng_static.sh
- utils/lib_sxng_test.sh
- utils/lib_sxng_themes.sh

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-06 19:37:12 +02:00
Markus Heiser
1d0abb7157 [doc] engine bt4g: add documentation to docs/dev/engines/online/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-06 09:30:48 +02:00
Markus Heiser
db522cf76d [mod] engine: wikimedia - improve results, add addition settings & doc
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-04 19:06:50 +02:00
Markus Heiser
1b030d4b41 [doc] engine: Yacy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 19:58:51 +02:00