Commit graph

86 commits

Author SHA1 Message Date
Markus Heiser
9ed6261308 [fix] format.python: regexp argument '--include' in BLACK_TARGETS
--include TEXT

  A regular expression that matches files and directories that should be
  included on recursive searches. An empty value means all files are included
  regardless of the name.  ...  Exclusions are calculated first, inclusions
  later.  [default: \.pyi?$]

Closes: https://github.com/searxng/searxng/issues/1142
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-07 18:21:21 +02:00
Markus Heiser
e8541b6006 [theme] peel out oscar from SearXNG development
This is the first step of removing oscar theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-30 13:20:27 +02:00
Markus Heiser
50a4e30830 [mod] add test.pybabel to cover issues in translation tasks
We need to cover issue related to pybabel/translation tasks.  By example there
was an issue [1] (fixed [2]) when upgrading jinja2 from 3.0.3 to 3.1.0 [3] that
has not been covered by the CI tests.

[1] https://github.com/searxng/searxng/runs/5688624325?check_suite_focus=true#step:6:348
[2] https://github.com/searxng/searxng/pull/1011
[3] https://github.com/searxng/searxng/pull/1008

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-23 10:20:22 +02:00
Markus Heiser
cf644b413e [test.pyright] suppress unneeded error & warning messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-22 11:16:41 +02:00
Markus Heiser
b9cf3c82a1 [mod] add i18n infrastructure for SearXNG message files (searxng.msg)
With this patch ``searxng.msg`` files can be added to SearXNG.  In
``searxng.msg`` files messages can be defined which are not captured by babel's
gettext, like the generic names of the categories or messages that are stored in
constants.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-16 09:55:53 +01:00
Markus Heiser
b1912607ae [mod] replace /help by /info pages and include pages in project docs
This patch implements a bolierplate to share content from info-pages of the
SearXNG instance (URL /info) with the project documentation (path /docs/user).

The info pages are using Markdown (CommonMark), to include them in the project
documentation (reST) the myst-parser [1] is used in the Sphinx-doc build chain.

If base_url is known (defined in settings.yml) links to the instance are also
inserted into the project documentation::

    searxng_extra/docs_prebuild

[1] https://www.sphinx-doc.org/en/master/usage/markdown.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12 11:36:31 +01:00
Markus Heiser
73a6da4dd9 [test.pyright] suppress warnings about intentional monkey patching
The warnings:

- "logger" is not defined'
- "supported_languages" is not defined'
- "language_aliases" is not defined'

are very verbose and superfluous, since these messages are related to
intentional monkey patching.

[1] https://github.com/searxng/searxng/pull/783#issuecomment-1019818178

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-25 23:26:53 +01:00
Markus Heiser
309147d86f [mod] introduce node.env.devtools function 2022-01-24 07:51:31 +01:00
Martin Fischer
6d701d2fea [fix] ensure that test.pyright installs pyright
nodejs.ensure only sets up NVM if there isn't a system-wide
installation of Node that matches our NODE_MINIMUM_VERSION.

The ubuntu image in the CI comes with an up to date node version,
so pyright from .nvm_packages is never installed.

This commit fixes this by introducing a package.json file.
2022-01-23 08:00:39 +01:00
Martin Fischer
96450b17d4 [mod] add test.pyright to test & ci.test targets
Since we currently have many type checking errors,
we for now only test with typeCheckingMode: off
which makes pyright only check files that contain a comment:

    # pyright: basic

to enable basic type checking, or

    # pyright: strict

to enable strict type checking.
2022-01-23 08:00:39 +01:00
Martin Fischer
3cd5ce55e8 [enh] clarify why we run pyright in the pyenv 2022-01-23 08:00:39 +01:00
Markus Heiser
8342773216 [mod] manage test.pyright: a commandline to run pyright tests
This patch implements the command and Makefile target::

    ./manage test.pyright
    make test.pyright

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-23 08:00:39 +01:00
Markus Heiser
fea9ad7927 [fix] issue reported by: make test.shell
$ make test.shell
    ./manage line 716:
    build_msg TEST "[reST markup] ${RST_FILES[@]}""
                                  ^-------------^ SC2145: Argument mixes string and array. Use * or separate argument.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-18 12:27:42 +01:00
Markus Heiser
27e3e56800 [mod] CI target test.rst: test reST markup of README.rst
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-17 09:35:42 +01:00
Markus Heiser
977e9a4330
Merge pull request #686 from return42/lib_redis
Add redis DB and connector
2022-01-11 19:55:14 +01:00
Markus Heiser
c9cfa6181b [fix] debug console (pdb) in a session started by make run
Commit c7f27404 moves the `python -m searx.webapp` process to the background.  A
background job can't open a simple python-debugger (pdb) console.

This patch moves all processes to the background except the searx.webapp.

Insert a break point somewhere in the webapp application::

    import pdb
    pdb.set_trace()

And start a debug session by::

    make run

and test you break point.

To test that the entire 'make run' stops in the case of an error in the
themes.live [1] background process try:

    make LIVE_THEME=typo-theme-name run

[1] https://github.com/searxng/searxng/pull/664#discussion_r776419585

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-05 10:09:17 +01:00
Markus Heiser
00d3a7477c
Merge pull request #664 from return42/themes.live
[mod] manage: add themes.live command (rebuild on modification)
2022-01-04 19:44:51 +01:00
Markus Heiser
fe8b88c8a4 [mod] script to build & install a redis instance
A script to build & install a simple & isolated redis service, dedicated to
SearXNG and connected via Unix socket.

    $ ./manage redis.help
    redis.:
      devpkg    : install essential packages to compile redis
      build     : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64
      install   : create user (searxng-redis) and install systemd service (searxng-redis)
      remove    : delete user (searxng-redis) and remove service (searxng-redis)
      shell     : start bash interpreter from user searxng-redis
      src       : clone redis source code to <path> and checkput 6.2.6
      useradd   : create user (searxng-redis) at /usr/local/searxng-redis
      userdel   : delete user (searxng-redis)
      addgrp    : add <user> to group (searxng-redis)
      rmgrp     : remove <user> from group (searxng-redis)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 14:34:26 +01:00
Markus Heiser
c7f2740414 [mod] live build of a theme: LIVE_THEME=simple make run
Environment variable to get live builds while modifying CSS & JS of a theme::

    LIVE_THEME=simple make run

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
1706b130a4 [fix] remove leftover from 59f4c792
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
edfd069d16 [mod] manage: add themes.live command (rebuild on modification)
To have a live build while editing the HTML client of a theme.  The first
argument of the command is the theme name:

    ./manage themes.live simple

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
c849731eb1 [mod] test.black: show issues (diff) reported by python-black
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 08:59:47 +01:00
Alexandre Flament
3c77412d33 [mod] replace pycodestyle by black
"make test.black" checks for the code style
"make format.python" format the python code
2021-12-27 08:58:59 +01:00
Alexandre Flament
81abc5f4c2 [fix] ./manage: upgrade geckodriver when GECKODRIVER_VERSION change 2021-12-24 09:14:15 +01:00
Markus Heiser
69af219d39 [test.robot] update gecko driver / required by selenium 4.1.0
Update gecko driver to v0.30.0 [1]

[1] https://github.com/mozilla/geckodriver/releases/tag/v0.30.0

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-23 13:09:48 +01:00
Markus Heiser
8069bc83e6 [fix] weblate: separate commit description from commit body
It is a common convention to separate commit description from commit body by a
empty line [1].

[1] https://www.conventionalcommits.org/en/v1.0.0/#summary

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-17 10:48:12 +01:00
Markus Heiser
f287787c44 [simple theme] activate - SearXNG JavaScript Style Guide
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::

    make themes.simple
    make themes.simple.test

In the github CI the `themes.all` target enforce a `themes.simple`.

BTW: Remove 'jshint' left overs from 0ee316f3d

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-14 07:50:53 +01:00
Alexandre Flament
59f4c792b4 [mod] simple theme: use sharp instead of convert to create .png from .svg
define a custom grunt task, since grunt-sharp is too old (it can't be installed).
in gruntfile.js, the image tasks are moved at the end the build chain.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-28 20:05:37 +01:00
Markus Heiser
c68c53acd8 [brand] initial - folder src/brand for searxng-wordmark
Folder src/brand is intended to place logo, wordmark and more in.  This commit
moves the origin (source) of searxng-wordmark.svg from the simple theme into
this folder

About the new created src/ folder:

On the long term we can move all the files from static/themes/<name>/src/* into
a src/themes/<name>/* folder.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 10:14:55 +01:00
Markus Heiser
1e997386e9
Merge pull request #508 from return42/arm64
[mod] Tools to install and maintain golang binaries & packages.
2021-11-18 19:49:01 +00:00
Markus Heiser
dc1442a2d1 [mod] Tools to install and maintain NVM versions manager for Node.js
[1] https://github.com/nvm-sh/nvm

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 23:31:11 +01:00
Markus Heiser
abd5eee392 [mod] Tools to install and maintain golang [1] binaries & packages.
[1] https://golang.org/doc/devel/release#policy

A simple *helloworld* test with user 'my_user' :

  sudo -H adduser my_user
  ./manage go.golang go1.17.3 my_user
  ./manage go.install github.com/go-training/helloworld@latest my_user
  ./manage go.bash my_user
  $ helloword
  Hello World!!

Don't forget to remove 'my_user':  sudo -H deluser --remove-home my_user

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 18:00:47 +01:00
Markus Heiser
5db24d4818 [fix] make test.shell - ./manage line 80
make test.shell reports an issue that has been added in [PR-500]::

    In ./manage line 80:
      yamllint  : lint YAML files: $YAMLLINT_FILES
                                   ^-------------^
                                   SC2128: Expanding an array without an
                                   index only gives the first element.

[PR-500] https://github.com/searxng/searxng/pull/500

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-15 17:02:44 +01:00
Markus Heiser
e7a9cd7a34 [fix] manage help - add description of test.yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 15:35:08 +01:00
Markus Heiser
3a981257de [simple theme] create favicon.png from src/svg/searxng-wordmark.svg
new bash function convert_if_newer() / usage::

    convert_if_newer <origfile> <outfile> [<options>, ...]
    convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64

Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>.  The command line is to convert is::

    convert <origfile> [<options>, ...] <outfile>

PNG 'searx/static/themes/simple/img/favicon.png' has been created by::

  $ make themes.simple
  CONVERT   searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
  ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 09:05:20 +01:00
Markus Heiser
68effd5eda [brand] SearXNG - bash env PYLINT_SEARXNG_DISABLE_OPTION
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Alexandre Flament
9e03823b98 SearXNG: docker 2021-10-12 20:23:57 +02:00
Markus Heiser
60edf2623d [brand] SearXNG - reference /etc/searxng/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-11 12:28:55 +00:00
Alexandre Flament
3fad483b7a SearXNG: partial update: reference /etc/searxng/settings.yml 2021-10-11 12:28:55 +00:00
Markus Heiser
afc5c9870e [fix] commit message of 'make weblate.push.translations'
Fix `\n` issue in the commit message [1] by using multiple `-m` options [2]::

    7d9ffd680 translations     [translations] update messages.pot and messages.po files\nFrom cebc0e39 - 2021-10-04 - Markus Heiser <markus.heiser@darmarIT.de>

[1] https://github.com/searxng/searxng/pull/379#issuecomment-933242702
[2] https://git-scm.com/docs/git-commit#Documentation/git-commit.txt--mltmsggt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-04 11:09:30 +02:00
Alexandre Flament
f4dd5b226d
[fix] make weblate.push.translations
Even when there are changes, the function exit without pushing them.
2021-10-03 21:11:32 +02:00
Alexandre Flament
47eb836c65
Merge pull request #375 from dalf/searxng_extra
SearXNG: searx_extra
2021-10-03 19:09:07 +02:00
Alexandre Flament
1bb82a6b54 SearXNG: searxng_extra 2021-10-02 17:30:39 +02:00
Alexandre Flament
253b850376 SearXNG: SEARXNG_SETTINGS_PATH 2021-10-02 17:18:05 +02:00
Alexandre Flament
430babca25 SearXNG: environment variables 2021-10-02 16:54:11 +02:00
Markus Heiser
7a3a1cd115 [mod] manage - use pyenv.activate where it makes sense
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17 15:31:09 +02:00
Alexandre Flament
e5e322e8ed [mod] utils/lib.sh: add pyenv.activate
* initialize virtualenv (pyenv.install)
* activate virtualenv

close #310
2021-09-17 14:15:35 +02:00
Markus Heiser
3f2c10ef9c [fix] manage help(): commands babel.* has been renamed to weblate.*
In commit 97355672c the functions named babel.* has been renamed to weblate.*
but it was forgotten to change it also in the help().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-12 15:46:00 +02:00
Markus Heiser
82847df300 [fix] add 'categories' to PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES
androp no longer needed (see line 591 in 7b235a1)::

    # pylint: disable=undefined-variable

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 10:29:38 +02:00
Alexandre Flament
b57d776edb
Merge pull request #293 from dalf/fix-290
[fix] translation: lock weblate only when there is an actual change of messages.pot
2021-09-06 20:00:19 +02:00