Commit graph

33 commits

Author SHA1 Message Date
Markus Heiser
97355672cd [translations] ./manage rename shell functions to integrate weblate
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':

- babel.setup.translations.worktree --> weblate.translations.worktree

  Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
  'translations' from Weblate's counterpart (weblate) of the SearXNG
   (origin)::

       remote weblate https://weblate.bubu1.eu/git/searxng/searxng/

- babel.weblate.to.translations     --> weblate.to.translations

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.

- babel.translations.to.master      --> weblate.translations.commit

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.  Copy the changes to the master branch, compile translations and
  create a commit in the local branch (master)

- babel.master.to.translations      --> weblate.push.translations

  Push *translation changes* from SearXNG (origin) to Weblate's
  counterpart (weblate).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:06:12 +02:00
Markus Heiser
0dc8d244c5 [translation] ./manage: add comments to new functions, rename job
./mange

  Comment functions:
  - babel.setup.translations.worktree
  - babel.weblate.to.translations
  - babel.translations.to.master
  - babel.master.to.translations
  BTW:
  - changed some cd into pushd
  - removed obsolete `set -x`
  - remove not needed echo from `last_commit_message=$(echo ..`

.github/workflows/translations-update.yml
docs/dev/translation.rst

  Rename job to: "create PR for additons from weblate"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:00:16 +02:00
Alexandre Flament
901ab87717 [translations] web integration
* make babel.translations.to.master: pull weblate updates
* make babel.master.to.translations: push .pot and .po files to weblate
2021-08-07 15:06:06 +02:00
Markus Heiser
900baf2eae [mod] manage - implement babel commands, drop update_translations.sh
In ./manage implement babel.*:

-  extract   : extract messages from source files and generate POT file
-  update    : update existing message catalogs from POT file
-  compile   : compile translation catalogs into binary MO files

Replace searx_extra/update/update_translations.sh by command:

- ci.babel.update

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-02 10:06:27 +02:00
Alexandre Flament
a7a181064f manage script: remove unused reference to utils/brand.env and .config.sh 2021-08-01 07:05:00 +02:00
Alexandre Flament
4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
Markus Heiser
61d3914b63 [fix] test.robot - do not use environment from ./utils/brand.env
Some defaults in the settings.yml are taken from the environment.
By example;

The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment.  This enviroment *wins over* any settings in a YAML file.
Whe we run a::

    make test.robot

The searx/settings_robot.yml is used, in this file the server settings are::

    server:
      port: 11111
      bind_address: 127.0.0.1

To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.

The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted.  ATM it unset the enviroment
variables::

    unset GIT_URL
    unset GIT_BRANCH
    unset SEARX_URL
    unset SEARX_PORT
    unset SEARX_BIND_ADDRESS

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Markus Heiser
51c06722c3 [mod] ./manage docker.buildx make variables local
The environment variables ..

- SEARX_GIT_VERSION
- VERSION_GITCOMMIT
- SEARX_PYTHON_VERSION
- GITHUB_USER
- SEARX_IMAGE_NAME
- BUILD

are local to the docker.buildx() function, other variables like GIT_URL are
comming from the utils/brand.env file (rebuild by: make buildenv).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser
e179dba7e4 [fix] ./manage script, prompt the unknown command
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-17 17:15:28 +02:00
Markus Heiser
25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00
Alexandre Flament
2530576f24
Merge pull request #157 from searxng/mod_static_build
static build: move generated files to searx/static/themes/*/src/generated
2021-06-22 10:36:19 +02:00
Markus Heiser
1559ed386e [fix] pygments.less - remove pyenv_OK condition
BTW: add pygments.less to the help message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 10:22:33 +02:00
Alexandre Flament
dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Alexandre Flament
d20f6a1f19 [mod] make themes.all update pygments*.less
Add a searx/static/themes/*/src/generated empty folder
2021-06-22 08:15:11 +02:00
Markus Heiser
f41b7a7204 [fix] py.build - fix small typo in the build message
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 12:52:26 +02:00
Markus Heiser
a62e1123e2 [fix] node.clean - ignore npm dependencies when npm is not installed
error pattern::

    $ make clean
    CLEAN     pyenv
    PYENV     [virtualenv] drop local/py3
    CLEAN     docs -- build/docs dist/docs
    CLEAN     locally installed npm dependencies
    ./manage: line 318: npm: command not found
    ERROR: node.clean exit with error (127)
    make: *** [Makefile:90: node.clean] Error 127

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-19 16:58:18 +02:00
Markus Heiser
ae677cb64b [enh] ./manage node.env - check build tools first
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS.  These tools can be installed by::

    sudo -H ./utils/searx.sh install buildhost

If one of the tools is not installed, the script node.env stops with a
appropriate message.

BTW: We ignore CentOS-7 as developer & build platform

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-17 09:23:02 +02:00
Alexandre Flament
14ba56fd1a [mod] make node.clean: call the "clean" script from the packages.json
The dependencies and build process are described in the gruntfiles.js and packages.json files.
It makes sense to move the clean up to these files too.
2021-06-16 16:04:58 +02:00
Alexandre Flament
c013256881 [mod] remove common grunt install
Before this commit, there are 3 node_modules directory:
* one in .
* two others in ./searx/statics/themes/*

This is no desirable:
* it declares the npm depdenencies in the shell script.
* dependabot can't updates theses dependencies.
* this is a not standard way to build a package (two different locations for the dependencies).

With this commit and the PR  #150 there is one unique node_modules directory per theme.
2021-06-16 15:39:32 +02:00
Alexandre Flament
7137d2893f [mod] simple theme: remove src/less/ion.less
This file is generated by webfont.
* It is now generated as searx/static/themes/simple/ion.less
* It is generated before the .less compilation.
* .gitignore includes this file

Add two new package depedencies: fontforge ttfautohint
See utils/searx.sh
2021-06-16 14:30:09 +02:00
Markus Heiser
9b0151eaa8 [fix] test.yamllint - fixed typo
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-15 17:04:51 +02:00
Markus Heiser
4a814dabf3 [yamllint] ./utils/templates/etc/searx/*.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-12 20:46:07 +02:00
Markus Heiser
40956e8e6b [data.all] add searx_extra/update/update_osm_keys_tags.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-09 18:08:23 +02:00
Markus Heiser
393c06f537 [enh] add test.yamllint - lint yaml files
Usage::

    make test.yamllint

    ./manage test.yamllint

test.yamllint is also added to the `test` makefile target.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-05 17:41:24 +02:00
Alexandre Flament
20580bcbd4 [docker] multiarch support: linux/amd64,linux/arm64,linux/arm/v7
make docker.buildx : build and push multiarch build.
(it can't be only build)

use buildx with the --cache-from and --cache-to options to cache the layers
(only the last built is cached)
2021-04-28 14:40:00 +02:00
Markus Heiser
3bd2f3dc81 [fix] manage - fix miss usage of 'set -e'
The philosophy of set -e is typically that it only exits upon uncaught
errors. Here, the presence of || outside the subshell seems to tell the shell
that the error inside the subshell is 'caught' and therefore set -e does not
cause an exit after false [1].

The shell does not exit if the command that fails is ... part of any command
executed in a && or || list except the command following the final && or ||, any
command in a pipeline but the last, or if the command’s return status is being
inverted with ! [2]

[1] https://unix.stackexchange.com/questions/296526/set-e-in-a-subshell
[2] https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html#The-Set-Builtin

BTW: fix error reported by 'make test.shell'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 13:02:55 +02:00
Markus Heiser
a92a4cb443 [pylint] get PYLINT_FILES from tag '# lint: pylint'
These py files are linted by test.pylint(), all other files are linted by
test.pep8()

close: https://github.com/searxng/searxng/issues/21
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 20:52:52 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Alexandre Flament
4863c1933e Rerevert c54bf42 2021-04-24 07:59:16 +02:00
Markus Heiser
c69fc20be2 [fix] ./manage pyenv.install - die when pyenv build fails
ERROR: When pyenv build fails, the pyenv command runs into a recursion [1]:

    ....
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  ERROR: Cannot install idna==3.1 and requests[socks]==2.25.1 because these package versions have conflicting dependencies.
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  The conflict is caused by:

[1] https://github.com/return42/searx-next/pull/4#issuecomment-821999497

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
d0e371f474 [mod] replace makefile boilerplate by 'manage' script
Replaces the make targets with the bash scripts

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
720d0d48ad [enh] implement all build task in a bash script (manage)
note: in further patches script 'manage' will replace 'manage.sh'

pyenv.* :
  assert    : build virtualenv if not exists
  intsall   : developer install of searx into virtualenv
  unintsall : uninstall developer installation
  cmd ...   : run command ... in virtualenv
  OK        : test if virtualenv is OK
pypi.upload:
  Upload python packages to PyPi (to test use pypi.upload.test)
pybuild :
  Build python packages at ./${PYDIST}
pyclean :
  delete virtualenv and intermediate py files
test.* :
  pylint    : lint PYLINT_FILES, searx/engines, searx & tests
  pep8      : pycodestyle (pep8) for all files except PYLINT_FILES
  unit      : run unit tests
  coverage  : run unit tests with coverage
  robot     : run robot test
  clean     : clean intermediate test stuff
node.* :
  env       : download & install npm dependencies locally
  clean     : drop npm installations
buildenv :
  rebuild ./utils/brand.env
data.* :
  all       : update searx/languages.py and ./data/*
  languages : update searx/data/engines_languages.json & searx/languages.py
  useragents: update searx/data/useragents.json with the most recent versions of Firefox.
themes.* :
  all       : build all themes
  oscar     : build oscar theme
  simple    : build simple theme
  bootstrap : less compile bootstrap.min.css CSS
babel.compile :
  pybabel compile ./searx/translations
docs.* :
  html      : build HTML documentation
  gh-pages  : deploy on gh-pages branch
  autobuild : autobuild HTML documentation while editing
  prebuild  : build reST include files (./${DOCS_BUILD}/includes)
  clean     : clean documentation build
docker.build [push] :
  build (and push) docker image
gecko.driver :
  download & install geckodriver if not already installed (required for
  robot_tests)

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00