Commit graph

386 commits

Author SHA1 Message Date
Markus Heiser
fe6bac5a08 [fix] pip install -e: legacy editable install (setup.py develop) is deprecated
From [1]: There is now a standardized mechanism [2] for an installer like pip to
request an editable install of a project.  pip is transitioning to using this
standard only instead of invoking the deprecated `setup.py develop` command.

For backward compatibility, we can use switches:

--use-pep517
  https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-use-pep517

--no-build-isolation
  https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-no-build-isolation

- [1] https://github.com/pypa/pip/issues/11457
- [2] https://peps.python.org/pep-0660/

Closes: https://github.com/searxng/searxng/issues/3701
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-08-21 12:09:14 +02:00
Markus Heiser
5c6b126d7f [fix] debian/ubuntu python-is-python3
Closes: https://github.com/searxng/searxng/issues/3235
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-08-20 20:20:50 +02:00
Bnyro
3bec04079c [feat] hostname replace plugin: possibility to prioritize certain websites
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2024-06-07 14:42:52 +02:00
Markus Heiser
63cf80aae5 [fix] docutils dependencies (docutils>=0.21.2)
Another trip into the hell of dependencies: docutils tends to put major changes
in minor patches: the executables have been renamed / e.g.

     rst2html.py --> rts2html

so we have to use docutils at least from version 0.21.2, but this version of
docutils is only supported by myst-parser from version 3.0.1 on.

Additionally, docutils decided to drop python 3.8 in version 0.21 [1]

Further, linuxdoc needed an update to cope with docutils 0.21 [2]

[1] https://docutils.sourceforge.io/RELEASE-NOTES.html#release-0-21-2024-04-09
[2] https://github.com/return42/linuxdoc/pull/36

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-05-09 17:15:21 +02:00
Markus Heiser
542f7d0d7b [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.

- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two

  1. ./searx/engines -> lint engines with additional builtins
  2. ./searx ./searxng_extra ./tests -> lint all other python files

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11 14:55:38 +01:00
Markus Heiser
8205f170ff [mod] pylint all engines without PYLINT_SEARXNG_DISABLE_OPTION
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11 14:55:38 +01:00
Markus Heiser
18028f3c13 [mod] CI: tag commits with leading scope/type of the modification
Add a leading tag (in square brackets) about the scope/type to commit messages
from automated tasks (commits from CI).

dependantbot::

    [upd] pypi: Bump .. from .. to ..
    [upd] npm: Bump .. from .. to .. in /searx/static/themes/simple

Weblate translation updates::

    [l10n] update translations from Weblate

updates of ./data::

    [data] update searx.data ...

build commit of gh-pages::

    [doc] build from commit ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-09 12:17:23 +01:00
KalokaK
fb36a06acd [fix] utils/searxng.sh uses searx.shared module which no longer exists
Issue exists since #1954 has been merged / the PR fixed all searx.shared except
this one fixed here in this commit.

Related: https://github.com/searxng/searxng/pull/1954
Closes: https://github.com/searxng/searxng/issues/3224
2024-02-20 15:18:40 +01:00
Alexandre Flament
ed66ed758d [mod] reduce memory footprint by not calling babel.Locale.parse at runtime
babel.Locale.parse loads more than 60MB in RAM.  The only purpose is to get:

    LOCALE_NAMES   - searx.data.LOCALES["LOCALE_NAMES"]
    RTL_LOCALES    - searx.data.LOCALES["RTL_LOCALES"]

This commit calls babel.Locale.parse when the translations are update from
weblate and stored in::

    searx/data/locales.json

This file can be build by::

    ./manage data.locales

By store these variables in searx.data when the translations are updated we save
round about 65MB (usually 4 worker = 260MB of RAM saved.

Suggested-by: https://github.com/searxng/searxng/discussions/2633#discussioncomment-8490494
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2024-02-20 10:43:20 +01:00
Markus Heiser
54a6626582 [fix] issues reported by shellcheck v0.9.0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-09 16:31:19 +01:00
Markus Heiser
f9c5727ddc [mod] get rid of ./utils/brand.env and its workflow
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-09 16:31:19 +01:00
Alexandre Flament
60bc5baea3 [mod] ./utils/get_setting.py tiny YAML parser for settings.yml
This allow to read settings on the fly even without virtualenv.  The ultimate
goal of the commit is to remove utils/brand.env from the git repository.

The code includes a tiny yaml parser that **should** be good enough.  The code
read searx/settings.yml directly (and ignore the environment variables).

yq [1] is a more reliable alternative but this require to download a binary from
github which is not great.

[1] https://github.com/mikefarah/yq/#install
2024-01-09 16:31:19 +01:00
Coccocoa's Helper
a85274eff5 Change the brand.env file to the original state 2023-11-28 09:31:54 +01:00
Bnyro
bf75a8c2a0 [feat] engine: implementation of bpb 2023-11-27 16:46:41 +01:00
Alexandre Flament
bd3f526859
Docker: add UWSGI_WORKERS and UWSGI_THREAD environment variables (#2992)
* Docker: add UWSGI_WORKERS and UWSGI_THREAD.

UWSGI_WORKERS specifies the number of process.
UWSGI_THREADS specifies the number of threads.

The Docker convention is to specify the whole configuration
through environment variables. While not done in SearXNG, these two
additional variables allows admins to skip uwsgi.ini

In additional, https://github.com/searxng/preview-environments starts Docker
without additional files through searxng-helm-chat.
Each instance consumes 1Go of RAM which is a lot especially when there are a
lot of instances / pull requests.

* [scripts] add environments UWSGI_WORKERS and UWSGI_THREADS

- UWSGI_WORKERS specifies the number of process.
- UWSGI_THREADS specifies the number of threads.

Templates for uwsgi scripts can be tested by::

    UWSGI_WORKERS=8 UWSGI_THREADS=9 \
      ./utils/searxng.sh --cmd\
      eval "echo \"$(cat utils/templates/etc/uwsgi/*/searxng.ini*)\""\
      | grep "workers\|threads"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

---------

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-12 16:46:34 +00:00
Bnyro
26fed56d51 [mod] settings.yml: remove plugin settings for plugins that don't exist anymore 2023-09-29 11:26:49 +02:00
Markus Heiser
51bcd95a75 [fix] harmonize default settings for caching of /static files
Caching files on the client side for more than a day can confuse the end user
when updating static files[1].

Depending on the way of providing a SearXNG instance via HTTP, there are several
ways to optimize the access to the /static files.  However, since we don't know
what optimization an admin has provided for his static files, we should have
moderate settings in the defaults that run robustly in a wide variety of
installations.

In this sense, all caches on the client side should be cleared after one day at
the latest.  So far the files were cached for one year on client side; as soon
as changes are made to the static files (with the option `static_use_hash:
true`) the old static files are kept for one year on the CLient side / which can
also be evaluated as unnecessary caching.

[1] https://github.com/searxng/searxng/discussions/2821

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-25 22:52:23 +02:00
Emilien Devos
827af00d9e Revert "[mod] activate limiter & link_token method (aka CSS ping) by default"
This reverts commit 3af629ec09.
2023-09-23 14:16:29 +02:00
Markus Heiser
3af629ec09 [mod] activate limiter & link_token method (aka CSS ping) by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-23 14:00:03 +02:00
Markus Heiser
90b0bfd1bf [fix] accidentally pushed utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18 18:18:37 +02:00
Bnyro
efd3a2d6d1 [feat] engine: implementation of internet archive scholar 2023-09-18 18:12:00 +02:00
jazzzooo
223b3487c3 [fix] spelling 2023-09-18 16:20:27 +02:00
ToxyFlog1627
f175574f37 [fix] typos in documentation & messages 2023-08-13 08:50:29 +02:00
Alexandre Flament
4573417b6c uwsgi.ini: remove unused cache2
cache2 was used before PR #1856
2023-08-13 07:55:05 +02:00
Markus Heiser
64100db904 [doc] improve documentation of make targets and ./manage script
BTW force modularization of the ./mange script into sub modules:

- utils/lib_sxng_data.sh
- utils/lib_sxng_node.sh
- utils/lib_sxng_static.sh
- utils/lib_sxng_test.sh
- utils/lib_sxng_themes.sh

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-06 19:37:12 +02:00
Markus Heiser
c9833ded9f [doc] update & fix documentation of the "SearXNG LXC suite"
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-21 18:12:39 +02:00
Markus Heiser
2499899554 [mod] Google: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.

When ever possible the implementations of the Google engines try to make use of
the async REST APIs.  The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - Google (WEB),
  - Google images,
  - Google news,
  - Google scholar and
  - Google videos

  and remove data from obsolete data type "supported_languages".

  A traits.custom type that maps region codes to *supported_domains* is fetched
  from https://www.google.com/supported_domains

searx/autocomplete.py:
  Reversed engineered autocomplete from Google WEB.  Supports Google's languages and
  subdomains.  The old API suggestqueries.google.com/complete has been replaced
  by the async REST API: https://{subdomain}/complete/search?{args}

searx/engines/google.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - always use the async REST API (formally known as 'use_mobile_ui')
  - use *supported_domains* from traits
  - improved the result list by fetching './/div[@data-content-feature]'
    and parsing the type of the various *content features* --> thumbnails are
    added

searx/engines/google_images.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - if exists, freshness_date is added to the result
  - issue 1864: result list has been improved a lot (due to the new cr parameter)

searx/engines/google_news.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
    *supported_domains* is not needed but a ceid list has been added.
  - different region handling compared to Google WEB
  - fixed for various languages & regions (due to the new ceid parameter) /
    avoid CONSENT page
  - Google News do no longer support time range
  - result list has been fixed: XPath of pub_date and pub_origin

searx/engines/google_videos.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - add paging support
  - implement a async request ('asearch': 'arc' & 'async':
    'use_ac:true,_fmt:html')
  - simplified code (thanks to '_fmt:html' request)
  - issue 1359: fixed xpath of video length data

searx/engines/google_scholar.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - request(): include patents & citations
  - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
  - hardening XPath to iterate over results
  - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
  - issue 1769 fixed: new request implementation is no longer incompatible

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
5820dc78ce [doc] slight improvements to the doc of the settings (base_url)
Closes: https://github.com/searxng/searxng/issues/2190

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17 12:08:58 +01:00
Markus Heiser
eebb489fb7 [fix] make nvm.env exit with error when VERBOSE is unset
$ make nvm.install
    INFO:  install (update) NVM at /800GBPCIex4/share/SearXNG/.nvm
    INFO:  already cloned at: /800GBPCIex4/share/SearXNG/.nvm
      || Fetching origin
    INFO:  checkout v0.39.1
      || HEAD is now at 9600617 v0.39.1
    make: *** [Makefile:96: nvm.install] Error 1

Without this fix we need to set VERBOSE environment to avoid the 'Error 1':

    $ VERBOSE=0 make nvm.install

BTW: fix an issue if there are any leftovers in ${NVM_DIR} from previos
installations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-11 13:18:07 +01:00
Markus Heiser
c90e157ddd [LXC] ubuntu 21.10 upgrade to ubuntu 22.04 LTS [1]
[1] https://wiki.ubuntu.com/Releases

Tested by::

    # build the container ..
    $ sudo -H ./utils/lxc.sh build searxng-ubu2204

    # install a complete SearXNG suite ..
    $ sudo -H ./utils/lxc.sh cmd searxng-ubu2204 FORCE_TIMEOUT=0 ./utils/searxng.sh install all

    # install nginx to export the SearXNG instance by HTTP
    $ sudo -H ./utils/lxc.sh cmd searxng-ubu2204 FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx

    # check instance
    $  sudo -H ./utils/lxc.sh cmd searxng-ubu2204 FORCE_TIMEOUT=0 ./utils/searxng.sh instance check
    INFO:  [searxng-ubu2204] FORCE_TIMEOUT=0 ./utils/searxng.sh instance check
    INFO:  wrapper:  utils/searxng.sh instance _call searxng.check

    SearXNG checks
    --------------
    ...
    INFO    searx                         : merge the default settings ( /usr/local/searxng/searxng-src/searx/settings.yml ) and the user settings ( /etc/searxng/settings.yml )
    INFO    searx                         : max_request_timeout=None
    INFO    searx.redisdb                 : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock'
    INFO    searx.redisdb                 : connected to Redis

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-01 09:30:13 +01:00
Milad-Laly
0af41ab15c [fix] typos in utils/searxng.sh 2023-01-08 10:08:53 +01:00
llmII
4e2eb9922e Fix checksum virtualenv checks.
There's already precedence for not using GNUism sha256sum longopts as
seen in searxng/utils/lib_go.sh so update lib.sh to not use them either.
A nice side effect is now the sha256sum usage doesn't care if you're
using BSD sha256sum or GNU sha256sum which makes this work under FreeBSD.
2023-01-03 18:44:55 -06:00
Alexandre Flament
b971167ced move searx.shared.redisdb to searx.redisdb 2022-12-10 09:26:38 +01:00
Alexandre FLAMENT
e92755d358 Initialize Redis in searx/webapp.py
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"

The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.

It worked before because SearXNG initialized the Redis connection when the limiter started.

In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
2022-11-05 17:45:52 +01:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
Léon Tiekötter
2b94fef7ec [fix] uWSGI: increase buffer-size
Increase max size of a request, by default it is 4k [1].  4096 as buffer-size is
too small and will result in the preference urls not working.

[1] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size
2022-07-31 12:40:06 +02:00
Markus Heiser
bded8750d5 [mod] fix minor leftovers from PR #1332
Related: https://github.com/searxng/searxng/pull/1332
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 16:13:47 +02:00
Markus Heiser
692708aa77 [clean up] drop obsolete searx, filtron and morty install scripts
Since ./utils/searxng.sh is implemented, the old installation procedures from
filtron, morty and searx can be removed.

For users who want to upgrade, the procedures for removing old installations
have still been retained.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
ed8a169029 [doc] update documentation of the installation procedures
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
782f73540e [utils/searxng.sh] implement new script to install SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
81bba44869 [install scripts] rename SEARX_<name> variables to SEARXNG_<name>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Alexandre Flament
9b3efa6d8a theme: remove __common__ 2022-05-07 19:40:48 +02:00
Markus Heiser
e8541b6006 [theme] peel out oscar from SearXNG development
This is the first step of removing oscar theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-30 13:20:27 +02:00
Markus Heiser
463254e3b4 [fix] since git v2.35.2 root can't read from unsafe repository
Git v2.35.2 closes an security issue, it is no longer possible that root uses a
git repo that is owned by someone else, the error message is::

    fatal: unsafe repository ('/share/darmarit.org/cache/searxng' is owned by someone else)

The fix is to run the `git diff --name-only` not as root in a sudo command.

[1] https://github.blog/2022-04-12-git-security-vulnerability-announced/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-22 17:37:04 +02:00
HZH
095beb8605
typo: wanr_msg -> warn_msg 2022-04-13 20:28:28 +08:00
Jabster28
89bcea56ec
don't reinstall existing packages on arch 2022-04-11 22:46:14 +01:00
Markus Heiser
7308a6e33c [LXC] ubuntu 20.10 EOL, upgrade to ubunut 21.10 [1]
[1] https://wiki.ubuntu.com/Releases

Tested by::

    # build the container ..
    $ sudo -H ./utils/lxc.sh build searx-ubu2110

    # open a shell in the container
    $ sudo -H ./utils/lxc.sh cmd searx-ubu2110 bash
    [root@searx-ubu2110 SearXNG]#

    # install a complete SearXNG suite ..
    [root@searx-ubu2110 SearXNG]# ./utils/searx.sh install all
    ...

    # install apache to export the SearXNG instance by HTTP
    [root@searx-ubu2110 SearXNG]# ./utils/searx.sh apache install
    ...
    INFO:  got 200 from http://10.174.184.94/searx

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12 18:00:45 +01:00
Markus Heiser
1c79c1a3d2 [LXC] fedora33 EOL [1], upgrade to fedora35 [2]
[1] https://docs.fedoraproject.org/en-US/releases/eol/
[2] https://docs.fedoraproject.org/en-US/releases/f35/

Tested by::

    # build the container ..
    $ sudo -H ./utils/lxc.sh build searx-fedora35

    # open a shell in the container
    $ sudo -H ./utils/lxc.sh cmd searx-fedora35 bash
    [root@searx-fedora35 SearXNG]#

    # install a complete SearXNG suite ..
    [root@searx-fedora35 SearXNG]# ./utils/searx.sh install all
    ...

    # install apache to export the SearXNG instance by HTTP
    [root@searx-fedora35 SearXNG]# ./utils/searx.sh apache install
    ...
    INFO:  got 200 from http://10.174.184.94/searx

To build wheel `python3-devel` needs to be added to SEARX_PACKAGES_fedora::

    |searx|   × Building wheel for setproctitle (pyproject.toml) did not run successfully.
    |searx|   │ exit code: 1
    ...
    |searx|       In file included from src/spt.h:15,
    |searx|                        from src/setproctitle.c:14:
    |searx|       src/spt_python.h:16:10: fatal error: Python.h: No such file or directory
    |searx|          16 | #include <Python.h>
    |searx|             |          ^~~~~~~~~~
    |searx|       compilation terminated.
    |searx|       error: command '/usr/bin/gcc' failed with exit code 1
    |searx|       [end of output]
    ...
    |searx|   ERROR: Failed building wheel for setproctitle

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12 17:44:00 +01:00
Markus Heiser
8d69ee5e7f [mod] Serving static files with uWSGI (searxng.ini)
1. Serving static files with uWSGI by using static file mount points [1].
2. Expires set to one year since there are hashes [2]

[1] https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html#mode-3-using-static-file-mount-points
[2] https://github.com/searxng/searxng/pull/932

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-08 18:13:13 +01:00
Markus Heiser
5eedd5b72a [fix] socket in SearXNG's uWSGI app (searxng.ini)
Use SEARX_UWSGI_SOCKET in uWSGI systemd service

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-08 17:44:27 +01:00