Commit graph

327 commits

Author SHA1 Message Date
Markus Heiser
57b8f340a6 [mod] utils/searx.sh - add command 'inspect settings'
Inspect YAML setting <key> from SearXNG instance (${SEARX_SRC})::

  utils/searx.sh inspect settings server.base_url

utils/lib_install.sh
  should not log on stdout which is used for the output of function
  prompt_installation_setting().  Turned build_msg (stdout) into
  info_msg (stderr).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:24:22 +02:00
Markus Heiser
28c874bf3f [brand] searx.sh & morty.sh - renamed 'searx' into 'SearXNG'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:39:02 +02:00
Markus Heiser
7167a7ee36 [mod] utils/.searx.sh: add commands: install [dot-config|init-src]
This patch implements two new commands:

install dot-config :
  - copy ./config.sh to ${SEARX_SRC}
  - implemented in new function install_DOT_CONFIG

install init-src:
  - sync files (SEARX_SRC_INIT_FILES) with ${SEARX_SRC}
  - implemented in new function init_SEARX_SRC()

new functions: verify_continue_install() and prompt_installation_status()
  These functions are used in installation procedures to get the status of the
  installation procedure.  The status is based on the existing function:

    usage: install_searx_get_state

        Prompts a string indicating the status of the installation procedure

    missing-searx-clone:
       There is no clone at ${SEARX_SRC}
    missing-searx-pyenv:
       There is no pyenv in ${SEARX_PYENV}
    installer-modified:
       There are files modified locally in the installer (clone),
       see ${SEARX_SRC_INIT_FILES} description.
    python-installed:
       Scripts can be executed in instance's environment
       - user:  ${SERVICE_USER}
       - pyenv: ${SEARX_PYENV}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:59 +02:00
Markus Heiser
f61c918dd4 [mod] normalize .config.sh with settings.yml
In commit 94851790 we have centralized all SearXNG setups in the settings.yml
file:

  94851790 [mod] move brand options from Makefile to settings.yml

This step has not yet been completed for the installation procedures!  Since all
SearXNG setups are done in the settings.yml these environment variables needs to
be removed from the ./conf.sh file.  Scripts and other tasks running outside of
an instance got the needed values from the ./utils/brand.env file.

By example: ATM the environment variables of the ./config.sh file are in
conflict with them from settings.yml:

  - PUBLIC_URL          --> {server:base_url}
  - SEARX_INTERNAL_HTTP --> {server:bind_address}.{server:port}
  - GIT_BRANCH          --> {brand:GIT_URL}

These environment variable of a SearXNG instance and additional

  - SEARX_SETTINGS_TEMPLATE

has been remove from the '.config.sh' file.  With this patch, the main focus of
./conf.sh resists on environment variables needed for the installation of morty,
filtron software.

modified  .config.sh:
  - removed no longer supported variables (see above)
  - add comment about: SearXNG setup in settings.yml

modified utils/searx.sh:
  - SEARX_INTERNAL_HTTP no longer take from .config.sh
  - SEARX_SETTINGS_PATH /etc/searx/settings.yml
  - SEARX_SETTINGS_TEMPLATE obsolete

modified utils/lib_install.sh:
  Initialize environment variables SEARX_PYENV, SEARX_SETTINGS_PATH and
  PUBLIC_URL.

modified:  utils/morty.sh
  Add missing hint about SEARX_SETTINGS_PATH and move PUBLIC_URL to
  utils/lib_install.sh

modified:  utils/morty.sh
  Move PUBLIC_URL to utils/lib_install.sh

Renamed utils/templates/etc/searx/use_default_settings.yml -> settings.yml
  - removed option which can't be modified after installation
  - add some comments with examples

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:57 +02:00
Markus Heiser
2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Markus Heiser
414a6105e7 [mod] load .config.sh from installation (utils/lib_install.sh)
**new** utils/lib_install.sh:
  Used to initialize installation procedures

  - Modified source_dot_config function that
    - loads .config.sh from an existing installation (at SEARX_SRC).
    - initialize **SEARX_SRC_INIT_FILES**
  - functions like:
    - install_log_searx_instance()
    - install_searx_get_state()

**modified** utils/searx.sh
    - obsolete environment SEARX_INSTANCE_NAME has been replaced
      by string 'SearXNG'.

**modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh
  - source utils/lib_install.sh
  - normalize logging of environment variables using new function
    install_log_searx_instance()

**modified** utils/lib.sh
  - fix marginal typos

**Installation scripts**
  The utils/lib_install.sh is sourced by the installations scripts:

  - utils/searx.sh
  - utils/morty.sh
  - utils/filtron.sh

  If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function
  loads this configuration (instead of './.config.sh').

**SEARX_SRC_INIT_FILES**
  Array of file names to sync into a installation at $SEARX_SRC.  The file names
  are relative to the $REPO_ROOT.  Set by function init_SEARX_SRC_INIT_FILES().
  Most often theses are files like:

  - .config.sh
  - searx/settings.yml
  - utils/brand.env
  - ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:33 +02:00
Markus Heiser
19abaf272d [mod] make pyenv.OK() less verbose when pyenv is OK
Prompt the verbose messages ...

  PYENV     [check] import yaml --> OK
  PYENV     OK

only when environment $VERBOSE is active.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser
c9220de690 [mod] drop unused setting option brand:twitter_url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser
3e50e8de3e [mod] drop usage of the searx.brand namespace (python procs)
Added function searx.get_setting(name, default=_unset):
  Returns the value to which ``name`` point.  If there is no such name in the
  settings and the ``default`` is unset, a KeyError exception is raised.

In all the python processes ..

- make docs
- make buildenv
- make install (setup.py)

the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.

- brand.SEARX_URL        --> get_setting('server.base_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.GIT_BRANCH'      --> get_setting('server.base_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')
- brand.WIKI_URL         --> get_setting('brand.wiki_url')
- brand.TWITTER_URL      --> get_setting('brand.twitter_url', '')

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-18 15:55:42 +02:00
Markus Heiser
b3ff510e10 [fix] pacman --noprogressbar in shell scripts
From pacman's man page:

  --noprogressbar
    Do not show a progress bar when downloading files. This can be useful for
    scripts that call pacman and capture the output.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:24:55 +02:00
Markus Heiser
73fc6499d8 [fix] utils/lib.sh: in a pipe, do not print ASCII escape code
In 2a12637 an ASCII escape code was added, such escape codes should not be
written to pipes (when the output is not a terminal).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:24:49 +02:00
Alexandre Flament
30e4a2a224
Merge pull request #174 from searxng/static_script
[mod] add make targets to manage the build files in the /searx/static directory
2021-06-28 10:53:32 +02:00
Markus Heiser
2a12637f59 [fix] utils/lib.sh: prefix_stdout show cursor (ANSI escape \e[?25h)
Some piped commands hide the cursor, show cursory when the stream ends.

Most often this is a bug of the command which piped.  The command should not
hide the cursor when it writes to a pipe.  I have seen this bug with the package
manager (pacman) from ArchLinux.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-27 11:48:57 +02:00
Markus Heiser
25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00
Markus Heiser
03d5d14d98 [mod] utils/manage_static.sh: remove git_log_current_branch
Marginal simplification of the procedure to get only the last local commit which
is not in remotes.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Markus Heiser
80710c666f [mod] utils/manage_static.sh: simplify git restore actions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Markus Heiser
2cb1f35086 [mod] utils/manage_static.sh: add script to shellcheck
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)

No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Alexandre Flament
0606cde105 [mod] add make targets to manage the build files in the /searx/static directory
The idea is to avoid conflict when there are differents branches with changes are made on the static files.
A solution is to ask the administrators to build the files from the sources, but it requires to install the npm dependencies.
So the solution in this commit keep the sources and the build files in the same git repository.

In one branch, the modification of the source (*.less, *.js) are commited without the built files.
The built files are commited in a uniq commit, with a commit message "Static build"
In case of merge or rebase, this commit can be dropped.

New make targets:
* static.build.commit.drop:
    drop the last "Static build".
    The command checks that there are only build files in the commit.
* static.build.commit :
    call "make static.build.commit.drop"
    call "make themes.all"
    commit the files
* static.git.restore.staged:
    git restore --staged <build files>
* static.git.restore:
    git restore <build files>

Related to https://github.com/searxng/searxng/issues/137
2021-06-26 13:13:31 +02:00
Markus Heiser
d19869b9f7
Merge pull request #173 from return42/fix-lxc-iptables
[mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
2021-06-25 08:23:40 +00:00
Markus Heiser
c8a6548592 [mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
Docker is blocking network of existing LXC containers / there is a conflict in
the iptables setup of Docker & LXC.  With this patch:

- utils/lxc.sh checks internet connectivity (instead of silently hang)
- Chapter "Internet Connectivity & Docker" describes the problem and made a
  suggestion for a solution a solution

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23 12:47:52 +02:00
Markus Heiser
b5f5ddd5a6 [fix] utils/searx.sh: signature of YAML settings has been changed
In 5a7b12ee we changed the signature of the YAML settings, this patch takes this
into account.

Related-to: 5a7b12ee [yamllint] searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23 09:56:59 +02:00
Alexandre Flament
dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Markus Heiser
ae677cb64b [enh] ./manage node.env - check build tools first
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS.  These tools can be installed by::

    sudo -H ./utils/searx.sh install buildhost

If one of the tools is not installed, the script node.env stops with a
appropriate message.

BTW: We ignore CentOS-7 as developer & build platform

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-17 09:23:02 +02:00
Alexandre Flament
7137d2893f [mod] simple theme: remove src/less/ion.less
This file is generated by webfont.
* It is now generated as searx/static/themes/simple/ion.less
* It is generated before the .less compilation.
* .gitignore includes this file

Add two new package depedencies: fontforge ttfautohint
See utils/searx.sh
2021-06-16 14:30:09 +02:00
Markus Heiser
1f91d85506 [fix] utils/morty.sh - PUBLIC_URL_MORTY is based on PUBLIC_URL
The public URL of the searx instance: PUBLIC_URL="https://mydomain.xy/searx"
The default is SEARX_URL which is taken from ./utils/brand.env.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-13 11:55:32 +02:00
Markus Heiser
4a814dabf3 [yamllint] ./utils/templates/etc/searx/*.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-12 20:46:07 +02:00
Markus Heiser
6ed4616da9 [enh] add settings option to enable/disable search formats
Access to formats can be denied by settings configuration::

    search:
        formats: [html, csv, json, rss]

Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Markus Heiser
666bf1d32b [fix] typo: 'interactiv' --> 'interactive'
Reported-by: https://github.com/searxng/searxng/pull/38#pullrequestreview-646940531
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 16:29:03 +02:00
Markus Heiser
d9ffb09011 [filtron.sh] make filtron rule file configurable
To select a different file with filtron rules, set environment

    FILTRON_RULES_TEMPLATE

the default is

    utils/templates/etc/filtron/rules.json

The installation is done by the new function install_rules() which offers a
multiple choice in case of collisions (known from searx.sh install setup).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 12:48:49 +02:00
Markus Heiser
28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 14:25:08 +02:00
Markus Heiser
d0e371f474 [mod] replace makefile boilerplate by 'manage' script
Replaces the make targets with the bash scripts

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
438d35c386 [fix] do not spit out error messages for utils/lxc.sh --help
Error::

  $ utils/lxc.sh --help
  ERROR: missing command lxc
  ...

This breaks also docs build::

  $ make docs
  ...
  WARNING: Unexpected return code 42 from command '../utils/lxc.sh --help'
  ...

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
d5eaa3fb31 [enh] utils/lib.sh - commands to build Sphinx-doc & deploy gh-pages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
036933599b [enh] utils/lib.sh - commands pyenv, pyenv.drop pyenv.(un)install
Implement a boilerplate to manage performance optimized virtualenv builds.
Shell scripts can use (e.g.) 'pyenv.cmd' to execute command in the virtualenv
without having to worry about whether and how the environment is provided. ::

  pyenv.cmd which python
  ..../local/py3/bin/python

  pyenv.cmd which pip
  ..../local/py3/bin/pip

If pyenv.cmd released multiple times the installation will only rebuild if the
function 'pyenv.OK' fails.  Function 'pyenv.OK' make some test to validate that
the virtualenv exists and works as expected.  The check also fails if
requirements listed requirements-dev.txt and requirements.txt has been edited.
Among these tests 'pyenv.OK' calls 'pyenv.check' which implements a python
script that validate the python installation.  Here is an example how a
'pyenv.check' implementation could look like::

    pyenv.check() {
       cat  <<EOF
    import yaml
    print('import yaml --> OK')
    EOF
    }

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
f9b05a6c44 [fix] utils/searx.sh - Double quote to prevent globbing
Fix issues reported by `make test.sh`

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:14:29 +02:00
Alexandre Flament
3755cf2eba More branding 2021-04-22 21:10:22 +02:00
Alexandre Flament
9d84025e4f searxng brand 2021-04-22 19:57:02 +02:00
3nprob
0fb423ea59 Allow overriding env vars SEARX_SETTINGS_PATH,UWSGI_SETTINGS_PATH 2021-04-08 12:56:15 +09:00
Markus Heiser
0305775e29 [mod] LXC switch to Fedora 33 / Fedora 31 reached its EOL
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05 13:17:47 +01:00
Alexandre Flament
aaae9a209e
Merge pull request #2600 from dalf/searx-extra
Add searx_extra package
2021-03-05 09:43:39 +01:00
Alexandre Flament
b8cd326464 Add searx_extra package
Split the utils directory into:
* searx_extra contains update scripts, standalone_searx.py
* utils contains the files to build and setup searx.
2021-03-04 11:59:14 +01:00
Alexandre Flament
1d10ae175c
Merge pull request #2618 from thezeroalpha/master
Fix security vulnerabilities in provided nginx configuration
2021-03-04 11:27:03 +01:00
Alexandre Flament
6ba37777f7
Merge pull request #2623 from return42/fix-centos7
[fix] add package `which` to CentOS-7 boilerplate
2021-03-04 11:19:05 +01:00
Alexandre Flament
d26261c5ab
Merge pull request #2619 from return42/drop-ubu1604
[mod] Drop Ubuntu 16.04 (Xenial Xerus) support
2021-03-04 11:18:21 +01:00
Alexandre Flament
4c2a8aea39
Merge pull request #2620 from return42/fix-git
[fix] support git versions <v2.22
2021-03-04 11:17:24 +01:00
Alex Balgavy
8736f5bd70 Use $host in nginx morty.conf template 2021-03-04 11:16:27 +01:00
Markus Heiser
c355bc3481 [fix] add package 'which' to CentOS-7 boilerplate
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer
include the which command.

Issue:

    $ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all
    INFO:  [searx-centos7] ./utils/filtron.sh install all
    ...
    Install Go in user's HOME
    -------------------------

    download and install go binary ..
    ...
    -bash: line 1: which: command not found
    -->|ERROR - Go Installation not found in PATH!?!
    -bash: line 2: which: command not found

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 19:37:03 +01:00
Markus Heiser
2e58988191 [fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:

    ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx

The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:57:20 +01:00
Markus Heiser
237b1cfdb3 [fix] support git versions <v2.22
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::

    $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
    ...
    git version 2.17.1

The option `--show-current` was added in git v2.22, the alternative to this
option is::

    git rev-parse --abbrev-ref HEAD

Issue when using option `--show-current`::

    [searx-ubu1804] Clone searx sources
    [searx-ubu1804] -------------------
    [searx-ubu1804]
    [searx-ubu1804] error: unknown option `show-current'
    [searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
    ....

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:22:41 +01:00
Markus Heiser
daa4b676df [mod] Drop Ubuntu 16.04 (Xenial Xerus) support
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time.  Issues are comming from dependencies (e.g. pip)::

    [searx-ubu1604]   |searx| SyntaxError: invalid syntax
    [searx-ubu1604]   |searx| Traceback (most recent call last):
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
    [searx-ubu1604]   |searx|     from pip._internal.cli.main import main
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
    [searx-ubu1604]   |searx|     sys.stderr.write(f"ERROR: {exc}")
    [searx-ubu1604]   |searx|                                    ^
    [searx-ubu1604]   |searx| SyntaxError: invalid syntax

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 17:52:50 +01:00
Alex Balgavy
6b59800dc6 Fix security vulnerabilities in suggested nginx configuration
The suggested configurations for nginx found in the documentation and
templates lead to vulnerabilities allowing host spoofing [1] and path
traversal [2], as reported by Gixy [3]. This commit fixes those issues.

[1] https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md
[2] https://github.com/yandex/gixy/blob/master/docs/en/plugins/aliastraversal.md
[3] https://github.com/yandex/gixy
2021-03-03 12:34:22 +01:00
Marc Abonce Seguin
d6681fd33b remove articles number from engines_languages.json 2021-02-25 23:54:21 -07:00
Alexandre Flament
7c1847d5f2 [mod] add utils/fetch_external_bangs.py
Based on duckduckgo bangs
Store bangs on a trie to allow autocomplete (not in this commit)
2021-02-24 18:48:36 +01:00
Alexandre Flament
5f4a085fc4
Merge pull request #2595 from dalf/update-wikidata-units
[mod] update wikidata_units.json and fetch_wikidata_units.py
2021-02-23 17:22:37 +01:00
Alexandre Flament
46ca32c3cc [mod] update currencies.json and fetch_currencies.py
use a sparql request on wikidata to get the list of currencies.

currencies.json contains the translation for all supported searx languages.

Supersede #993
2021-02-23 16:42:28 +01:00
Alexandre Flament
93d1da4906 [mod] update wikidata_units.json and fetch_wikidata_units.py
The fetch_wikidata_units.py result won't change randomly.
See comments in the script.
2021-02-23 13:10:38 +01:00
Alexandre Flament
90b9d0d6a8 [mod] CI: minor changes
* utils/makefile.python: travis-gh-pages renamed ci-gh-pages
2021-02-02 08:53:57 +01:00
Alexandre Flament
34de715e62
Merge pull request #2500 from dalf/github-action-data
[enh] every Sunday, call utils/fetch_*.py scripts and create a PR automatically
2021-02-01 17:16:58 +01:00
Alexandre Flament
99244440e4
Merge pull request #2514 from return42/fix-gh-pages
[fix] Makefile target gh-pages & flatten history of branch gh.pages
2021-02-01 17:07:08 +01:00
Markus Heiser
38b39ef0ae [fix] re-add 'pip-exe' target - partial revert 9b48ae47
Target pip-exe is a prerequisite of the targets:

  - pyinstall
  - pyuninstall

and was accidentally deleted in commit 9b48ae47.

HINT:
  do not confuse pyinstall with penvinstall

pyinstall & pyuninstall
    Installing into user's HOME using pip from OS,
    therefore the message is needed.

pyenvinstall & pyenvuninstall
    Installing into virtualenv (./local) using pip which is provided by
    prerequisite 'pyenv' in the virtualenv.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-01 16:58:04 +01:00
Alexandre Flament
d70c5a621a [mod] more robust make pyenv / make pyenvinstall
"make pyenv" ensures that ./local/py3/bin/python is an executable
2021-02-01 16:58:04 +01:00
Alexandre Flament
806af50738
Merge pull request #2494 from return42/rm-fabfile
[fix] remove Fabric file
2021-02-01 15:09:35 +01:00
Markus Heiser
40d2a116e1 [fix] Makefile target gh-pages & flatten history of branch gh.pages
1. This patch fixes error:

    rm -rf gh-pages/
    make V=1 gh-pages
    make[1]: Leaving directory '/800GBPCIex4/share/searx'
    [ -d "gh-pages/.git" ] || git clone  gh-pages
    fatal: repository 'gh-pages' does not exist

2. The gh-page build has been moved to ./build/gh-pages this also affects
   'travis-gh-pages'

3. The gh-pages commit messages now includes a ref to the repository and commit

4. Since a gh-pages history has only the drawback that the reposetory grows
   fast, this patch also flattens the history:

    cd build/gh-pages/; git log --oneline
    bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
    026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-29 11:41:48 +01:00
Alexandre Flament
6047087aac [mod] utils/fetch_languages.py: write files at the right location 2021-01-24 14:25:27 +01:00
Markus Heiser
ea5c992d4f [fix] lxc.sh - SC2034: ubu2010_boilerplate appears unused.
$ make test.sh
  In utils/lxc.sh line 42:
  ubu2010_boilerplate="$ubu1904_boilerplate"
  ^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-24 08:29:13 +01:00
Markus Heiser
a8544798ec [fix] remove Fabric file
The fabfile.py has not been updated since 5 years.  I also asked [1] if someone
still use Fabric wtihout any response.  Lets drop outdated Fabric file.

[1] https://github.com/searx/searx/discussions/2400

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-22 17:57:55 +01:00
Alexandre Flament
aa887eb375 [mod] checker : replace pycld3 by langdetect
pycld3 requires the native library cld3
langdetect is a pure python package
2021-01-19 21:26:04 +01:00
Alexandre Flament
709dd960f1
Merge pull request #2473 from return42/fix-setup.py
[fix] setup.py requires pyyaml installed
2021-01-16 19:05:36 +01:00
Markus Heiser
c4a98862bf [fix] setup.py requires pyyaml installed
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
    ERROR: Command errored out with exit status 1:
     command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
         cwd: /usr/local/searx/searx-src/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
        from searx.version import VERSION_STRING
      File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
        import searx.settings_loader
      File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
        import yaml
    ModuleNotFoundError: No module named 'yaml'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-16 08:58:13 +01:00
Alexandre Flament
a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
2021-01-14 20:57:17 +01:00
Alexandre Flament
6e2872f436 [enh] add searx.shared
shared dictionary between the workers (UWSGI or werkzeug)
scheduler: run a task once every x seconds (UWSGI or werkzeug)
2021-01-12 11:47:17 +01:00
Alexandre Flament
8cbc9f2d58 [enh] add checker 2021-01-12 11:47:17 +01:00
Alexandre Flament
d5882147e0 [fix] utils/build_env.py: make sure to always use searx/settings.yml
ignore SEARX_SETTINGS_PATH
2021-01-11 22:59:35 +01:00
Markus Heiser
a31cb68668 [fix] utils/build_env.py: make sure to always use searx/settings.yml
ignore SEARX_SETTINGS_PATH

[1] https://github.com/searx/searx/pull/2408#pullrequestreview-565247895

Suggested-by: @dalf [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:56:15 +01:00
Markus Heiser
9e53470b4c [mod] get rid of searx/brand.py
Removes module searx/brand.py and creates a namespace at searx.brand.

This patch is a first 'proof of concept'.  Later we can decide to remove the
brand namespace entirely or not.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Markus Heiser
9485179064 [mod] move brand options from Makefile to settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Markus Heiser
dc544498d0 [mod] add target upload-pypi-test https://test.pypi.org/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-10 17:05:37 +01:00
Markus Heiser
89e6ba5af7 [mod] add Ubuntu 20.10 image to the LXC suite
Ubuntu 20.10 image [1] is avialable from https://images.linuxcontainers.org

BTW: replace builhost package 'ttf-dejavu' by more common meta package
'fonts-dejavu' ('ttf-dejavu' is no longer available in 20.10).

[1] d161de601b

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-02 17:27:16 +01:00
Markus Heiser
c7d92d6475 [fix] remove Ubuntu 19.10 from the LXC suite (EOL)
Official support for Ubuntu 19.10 'Eoan Ermine' ended on July 17, 2020.

The image has been dropped [1] from https://images.linuxcontainers.org

[1] d161de601b

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-02 17:23:18 +01:00
Alexandre Flament
5fb9a5c7c1
Merge pull request #2411 from dalf/update-secret-key-check
Update secret key check
2020-12-27 18:04:53 +01:00
Markus Heiser
af584f2c40 [perf] pylint: Use multiple processes to speed up Pylint.
Specifying '-j 0' will auto-detect the number of processors available to use.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2421#issuecomment-751444257
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-27 12:04:39 +01:00
Alexandre Flament
568b9465e9 [mod] check secret_key when searx.webapp is imported
Without this commit the module searx checks the secret_key value.

With this commit, make docs, utils/standalone_searx.py,
utils/fetch_firefox_version.py works without SEARX_DEBUG=1

For reference see https://github.com/searx/searx/pull/2386
2020-12-27 10:30:20 +01:00
Alexandre Flament
1ba53e6129
Merge pull request #2420 from return42/fix-sphinx-book-targets
fix makefile `books/{name}.*` targets and add documentation about these targets
2020-12-27 09:47:49 +01:00
Alexandre Flament
1956ab4b50
Merge pull request #2412 from dalf/update-buildenv
[fix] update buildenv
2020-12-27 08:31:23 +01:00
Markus Heiser
d175a0fb7b [mod] LXC_ENV_FOLDER moved from ./lxc to ./lxc-env
BTW:

- add target $(LXC_ENV_FOLDER)
- fix indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-26 11:43:18 +01:00
Markus Heiser
bc7bd1d80a [fix] utils/lib.sh: various typos in messages and comments
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-26 11:38:47 +01:00
Markus Heiser
6316a8d49a [fix] message to install different python versions from apt PPA
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 23:17:21 +01:00
Markus Heiser
809bf85426 [perf] pyenvinstall: reinstall only when requirements are change
Create a (PY_ENV)/requirements.sha256 file with sha256 sums of the
./requirements*.txt files and reinstall only if the sha256 sums change.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2418#pullrequestreview-558426579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 16:25:09 +01:00
Markus Heiser
d85012b49b [perf] virtualenv: install boilerplate only once
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 15:59:57 +01:00
Markus Heiser
aace837665 [fix] makefile.sphinx: additions to commit c2a6f145
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.

This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 13:54:07 +01:00
Alexandre Flament
2fc2d17a27
Merge pull request #2418 from return42/fix-pyenv-install
[fix] makefile.python: remove duplicate pyenv-(un)install targets
2020-12-24 10:23:09 +01:00
Markus Heiser
af49bd2bd2 [fix] makefile.python: remove duplicate pyenv-(un)install targets
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 09:18:22 +01:00
Markus Heiser
5fc9273e23 [mod] utils/searx.sh install: create branch if not already exists
This patch keeps an eye on:

1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-23 16:06:06 +01:00
Markus Heiser
0fc3759d74 [fix] use SEARX_SETTINGS_TEMPLATE from .config environment
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-23 14:10:07 +01:00
Alexandre Flament
db5b060455 [fix] update buildenv
CONTACT_URL is unset in Makefile, but searx/brand.py and
utils/brand.env are not updated.

This commit fixes this issue.
2020-12-21 10:55:28 +01:00
Alexandre Flament
72d2bb2289
Merge pull request #2399 from return42/fix-lxc
[fix] determine path to makefile.lxc in a LXC
2020-12-20 09:32:00 +01:00
Markus Heiser
be08e5bfab [fix] yum installation of 'Development Tools'
'yum install' does not support '@development-tools' notation for group
installation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-18 22:32:05 +01:00
Markus Heiser
1a4524e03c [mod] remove obsolete virtualenv command
Installation and use of the command 'virtualenv' was only needed in py2 and py2
is no longer suported by searx.  In py3 the command is replaced by 'python -m
venv'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-18 22:31:13 +01:00
Markus Heiser
c89df6739f [fix] determine path to makefile.lxc in a LXC
The path to the makefile.lxc is not always static and
'/share/searx/utils/makefile.lxc' can be wrong.

When target 'lxc.activate' is called within the LXC container the path to
makefile.lxc depends on LXC_SHARE_FOLDER and the basename of the folder where
searx was cloned inital on the HOST system.

This patch determines the path of 'makefile.lxc' next to 'makefile.include'
itself.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-18 17:10:49 +01:00
BBaoVanC
19fce74443
Add link to contact instance maintainer to footer of each page (#2391) 2020-12-18 09:53:28 +01:00