Commit graph

4573 commits

Author SHA1 Message Date
Markus Heiser
25b5797a0c
Merge pull request #103 from searxng/add-sqlite-engine2
[enh] add offline engine for sqlite database
2021-05-27 14:06:42 +00:00
Alexandre Flament
b1da97d092
Merge pull request #96 from return42/unsplash
lint & fix unsplash engine
2021-05-27 14:58:59 +02:00
Alexandre Flament
2ea34a3c36 [enh] add offline engine for sqlite database
To test & demonstrate this implementation download:

  https://liste.mediathekview.de/filmliste-v2.db.bz2

and unpack into searx/data/filmliste-v2.db, in your settings.yml define a sqlite
engine named "demo"::

    - name : demo
      engine : sqlite
      shortcut: demo
      categories: general
      result_template: default.html
      database : searx/data/filmliste-v2.db
      query_str :  >-
        SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
               COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
               description AS content
          FROM film
         WHERE title LIKE :wildcard OR description LIKE :wildcard
         ORDER BY duration DESC
      disabled : False

Query to test: "!demo concert"

This is a rewrite of the implementation from commit [1]

[1] searx/searx@8e90a21

Suggested-by: @virtadpt searx/searx#2808
2021-05-27 14:27:11 +02:00
Markus Heiser
e1f244b2d5 [pylint] webapp.py - fix all messages from pylint
No functional change, just some linting.

- fix messages from pylint (see below)
- log where general Exceptions are catched (broad-except)
- normalized various indentation
- To avoid clashes with common names, add prefix 'route_' to all @app.route
  decorated functions.

Fixed messages::

    searx/webapp.py:744:0: C0301: Line too long (146/120) (line-too-long)
    searx/webapp.py:756:0: C0301: Line too long (132/120) (line-too-long)
    searx/webapp.py:730:9: W0511: TODO, check if timezone is calculated right (fixme)
    searx/webapp.py:1:0: C0114: Missing module docstring (missing-module-docstring)
    searx/webapp.py:126:8: I1101: Module 'setproctitle' has no 'setthreadtitle' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
    searx/webapp.py:126:36: W0212: Access to a protected member _name of a client class (protected-access)
    searx/webapp.py:131:4: R1722: Consider using sys.exit() (consider-using-sys-exit)
    searx/webapp.py:141:4: R1722: Consider using sys.exit() (consider-using-sys-exit)
    searx/webapp.py:255:38: W0621: Redefining name 'request' from outer scope (line 32) (redefined-outer-name)
    searx/webapp.py:307:4: W0702: No exception type(s) specified (bare-except)
    searx/webapp.py:374:24: W0621: Redefining name 'theme' from outer scope (line 155) (redefined-outer-name)
    searx/webapp.py:420:8: R1705: Unnecessary "else" after "return" (no-else-return)
    searx/webapp.py:544:4: W0621: Redefining name 'preferences' from outer scope (line 917) (redefined-outer-name)
    searx/webapp.py:551:4: W0702: No exception type(s) specified (bare-except)
    searx/webapp.py:566:15: W0703: Catching too general exception Exception (broad-except)
    searx/webapp.py:613:4: R1705: Unnecessary "elif" after "return" (no-else-return)
    searx/webapp.py:690:8: W0621: Redefining name 'search' from outer scope (line 661) (redefined-outer-name)
    searx/webapp.py:661:0: R0914: Too many local variables (22/20) (too-many-locals)
    searx/webapp.py:674:8: R1705: Unnecessary "else" after "return" (no-else-return)
    searx/webapp.py:697:11: W0703: Catching too general exception Exception (broad-except)
    searx/webapp.py:748:4: R1705: Unnecessary "elif" after "return" (no-else-return)
    searx/webapp.py:661:0: R0911: Too many return statements (9/6) (too-many-return-statements)
    searx/webapp.py:661:0: R0912: Too many branches (29/12) (too-many-branches)
    searx/webapp.py:661:0: R0915: Too many statements (74/50) (too-many-statements)
    searx/webapp.py:931:4: W0621: Redefining name 'image_proxy' from outer scope (line 1072) (redefined-outer-name)
    searx/webapp.py:946:4: W0621: Redefining name 'stats' from outer scope (line 1132) (redefined-outer-name)
    searx/webapp.py:917:0: R0914: Too many local variables (34/20) (too-many-locals)
    searx/webapp.py:917:0: R0912: Too many branches (19/12) (too-many-branches)
    searx/webapp.py:917:0: R0915: Too many statements (65/50) (too-many-statements)
    searx/webapp.py:1063:44: W0621: Redefining name 'preferences' from outer scope (line 917) (redefined-outer-name)
    searx/webapp.py:1072:0: R0911: Too many return statements (9/6) (too-many-return-statements)
    searx/webapp.py:1151:4: C0103: Variable name "SORT_PARAMETERS" doesn't conform to '(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*)|([a-z]))$' pattern (invalid-name)
    searx/webapp.py:1297:0: R1721: Unnecessary use of a comprehension (unnecessary-comprehension)
    searx/webapp.py:1303:0: C0103: Argument name "e" doesn't conform to '(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*))$' pattern (invalid-name)
    searx/webapp.py:1303:19: W0613: Unused argument 'e' (unused-argument)
    searx/webapp.py:1338:23: W0621: Redefining name 'app' from outer scope (line 162) (redefined-outer-name)
    searx/webapp.py:1318:0: R0903: Too few public methods (1/2) (too-few-public-methods)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-27 12:30:56 +02:00
Markus Heiser
ae25362205 [pylint] webapp.py - fix all wrong-import-order messages from pylint
pylint message: wrong-import-order
  Respect PEP8 import order (standard imports first, then third-party libraries,
  then local imports).

pylint message: wrong-import-position
  Do not mix code & imports

BTW:

- only one import per line
- replace licence text by SPDX tag

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-26 21:17:19 +02:00
Markus Heiser
85364edaf0 [fix] webapp.py - running without installation is not supported
Remove extension of the sys.path (aka PYTHONPATH).  Running instance directly
from repository's folder is a relict from the early beginning in
2014 (fd651083f) and is no longer supported.

Since commit dd46629 was merged the command line 'searx-run' exists and should
be used.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-26 21:17:19 +02:00
Alexandre Flament
7c76cef746
Merge pull request #94 from return42/pylint-network
Pylint searx.network
2021-05-26 18:35:29 +02:00
Markus Heiser
97d173eda0 [mod] unsplash engine - activated by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 17:41:10 +02:00
Markus Heiser
dc21cb5d4b [fix] unsplash engine - 'searx:result: invalid title:'
- Use result 'alt_description' as title, if not given use
  default title 'unknown'.
- Use result 'description' from unsplash as 'content'

Fix error::

    DEBUG:searx:result: invalid title: {..., 'title': None, 'content': '', 'engine': 'unsplash'}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 17:26:58 +02:00
Markus Heiser
a88e3e4fea [pylint] searx/engines/unsplash.py, add logger & norm indentation
- fix messages from pylint
- add logger and log request URL
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 16:45:32 +02:00
Alexandre Flament
8283ce3d79
Merge pull request #92 from return42/xpath-misc
Xpath enhencements
2021-05-25 09:06:06 +02:00
Markus Heiser
2128022f72 [coding-style] searx/network/network.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:48:47 +02:00
Markus Heiser
1499002ceb [coding-style] searx/network/client.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:44:43 +02:00
Markus Heiser
e4211da639 [pylint] searx/network/raise_for_httperror.py
No functional change!

- fix messages from pylint
- add ``global NETWORKS``
- normalized indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:40:10 +02:00
Markus Heiser
44efa911ba [pylint] searx/network/network.py & add global (NETWORKS)
No functional change!

- fix messages from pylint
- add ``global NETWORKS``

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:46 +02:00
Markus Heiser
b595c482d0 [pylint] searx/network/client.py & add global (TRANSPORT_KWARGS)
No functional change!

- fix messages from pylint
- add ``global TRANSPORT_KWARGS``
- normalized python_socks imports

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:37 +02:00
Markus Heiser
8033518899 [pylint] searx/network/__init__.py & add global (THREADLOCAL)
No functional change!

- fix messages from pylint
- add ``global THREADLOCAL``
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:14 +02:00
Markus Heiser
f963759ccc [fix] engine genius should not use the video template
Remove 'template' from result.  Engine genius should
not use the video template.  BTW: fix indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 16:31:14 +02:00
Markus Heiser
3a71d4b175 [pylint] searx/engines/genius.py, add logger & normalized indentation
- pylint searx/engines/genius.py
- add logger and log ignored exceptions
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 16:19:06 +02:00
Markus Heiser
84a943f867 [enh] XPath engine - add time safe-search support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 22:26:18 +02:00
Markus Heiser
6bfe3fd033 [enh] XPath engine - add time range support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 16:49:30 +02:00
Markus Heiser
1933577c8e [enh] XPath engine - add ISO 639-1 {lang} replacement to search-URL
BTW: remove obsolte params['query'] and not needed paging condition.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 15:05:36 +02:00
Markus Heiser
703f8c4a8b
Merge pull request #91 from return42/xpath-misc
[doc] add documentation about the XPath engine
2021-05-23 10:02:16 +00:00
Markus Heiser
2065a324d9 [doc] use a hint instead of a warning about searxng
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 11:54:21 +02:00
Markus Heiser
8cd544b2a6 [doc] add documentation about the XPath engine
- pylint searx/engines/xpath.py
- fix indentation of some long lines
- add logging
- add doc-strings

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 11:48:21 +02:00
Markus Heiser
2398e9a1fe
Merge pull request #85 from searxng/fix-metrics-offline-engine
Fix metrics offline engine
2021-05-22 13:34:44 +00:00
Markus Heiser
7728e25b11 [refactor] metrics.get_reliabilities() - make code more readable
- init stat values by None
- drop round_or_none
- don't try to get percentage if base is 'None'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-22 15:17:18 +02:00
Alexandre Flament
99aaf86b73 [fix] offline engines: fix templates /stats and /preferences 2021-05-22 15:17:18 +02:00
Alexandre Flament
3014463fed [fix] metrics: processing time = total time if there is no http time
It was previsouly None

Fix /stats
2021-05-22 15:17:18 +02:00
Alexandre Flament
ec83493538 [fix] offline engine: don't crash on time recording 2021-05-22 15:17:18 +02:00
Alexandre Flament
2f76b570ab
Merge pull request #86 from searxng/remove-call-to-gc
[mod] remove gc.collect() after each user request
2021-05-21 18:31:04 +02:00
Markus Heiser
6f1446d55f [pylint] searx/search/__init__.py & replace lic-text by SPDX tag
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-21 17:31:22 +02:00
Alexandre Flament
426fadccb3 [mod] remove gc.collect() after each user request 2021-05-21 17:23:18 +02:00
Markus Heiser
1ab4317f59
Merge pull request #88 from searxng/dependabot/pip/master/sphinx-4.0.2
Bump sphinx from 4.0.1 to 4.0.2
2021-05-21 06:33:34 +00:00
dependabot[bot]
1e9efd0822 Bump sphinx from 4.0.1 to 4.0.2
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.0.1 to 4.0.2.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.0.1...v4.0.2)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-21 08:31:50 +02:00
Markus Heiser
1cfb3c3221
Merge pull request #87 from searxng/dependabot/pip/master/pallets-sphinx-themes-2.0.1
Bump pallets-sphinx-themes from 2.0.0 to 2.0.1
2021-05-21 06:25:31 +00:00
dependabot[bot]
b565efb48f
Bump pallets-sphinx-themes from 2.0.0 to 2.0.1
Bumps [pallets-sphinx-themes](https://github.com/pallets/pallets-sphinx-themes) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/pallets/pallets-sphinx-themes/releases)
- [Changelog](https://github.com/pallets/pallets-sphinx-themes/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/pallets-sphinx-themes/compare/2.0.0...2.0.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-21 05:53:03 +00:00
Alexandre Flament
0c1ab0227d
Merge pull request #84 from searxng/fix-test-ignore-etc-settings
[fix] unit test: don't load /etc/searx/settings.yml
2021-05-20 13:31:53 +02:00
Alexandre Flament
94e6ed6110
Merge pull request #66 from return42/docker-buildx
review of docker entry point & docker documentation
2021-05-19 17:31:33 +02:00
Alexandre Flament
d1cffc55cb [fix] unit test: don't load /etc/searx/settings.yml
Add a new environment variable SEARX_DISABLE_ETC_SETTINGS
to disable loading of /etc/searx/settings.yml

unit tests:
* set SEARX_DISABLE_ETC_SETTINGS to 1
* remove SEARX_SETTINGS_PATH if it exists
2021-05-18 17:23:21 +02:00
Alexandre Flament
d255e5637b
Merge pull request #81 from return42/csp-bar-graph
[fix] make /stats more CSP compliant
2021-05-18 08:55:56 +02:00
Alexandre Flament
da2b0562d4
Merge pull request #83 from return42/fix-archive-is
[fix] engine "archive is" - search_url has been changed
2021-05-18 08:44:34 +02:00
Markus Heiser
66c8482725 [emacs] don't use tabs, except in makefile-gmake-mode
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:26:20 +02:00
Markus Heiser
73d73dcf9b [static] make /stats more CSP compliant- make themes.all
Based on commits

- 0507e185 [fix] bar graph and rename CSS class engine-scores -> engine-score
- 3e9ad7ae [fix] make /stats more CSP compliant - github issue form
- 34859d0e [fix] make /stats more CSP compliant - oscar theme
- 0a6c4884 [fix] make /stats more CSP compliant - simple theme
- cdfb4b7f [fix] make /stats more CSP compliant - bar graph
- 965817f2 [fix] simple theme - generate missing sourceMap file

this patch is generated by::

     make themes.all

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:25:32 +02:00
Markus Heiser
0507e185a5 [fix] bar graph and rename CSS class engine-scores -> engine-score
- drop #main_stats selector in stats.less
- 'engine-score' exists before this PR.
- untabify searx/static/themes/__common__/less/stats.less

for details see comment at: d93bec7638..1204e4f07e (r633571496)

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:24:19 +02:00
Markus Heiser
3e9ad7ae0c [fix] make /stats more CSP compliant - github issue form
Hide textarea from github issue form::

    ./__common__/new_issue.html:6:    <textarea name="body" style="display: none;">{{- '' -}}

BTW: fix indentation.

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:23:27 +02:00
Markus Heiser
34859d0e58 [fix] make /stats more CSP compliant - oscar theme
Replace oscar theme's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/__common__/less/stats.less`` ::

    ./oscar/stats.html:29:  <th scope="col" style="width:20rem;">{{ th_sort('name', _("Engine name")) }}</th>
    ./oscar/stats.html:30:  <th scope="col" style="width:7rem; text-align: right;">{{ th_sort('score', _('Scores')) }}</th>
    ./oscar/stats.html:33:  <th scope="col" style="text-align: right;">{{ th_sort('reliability', _('Reliability')) }}</th>
    ./oscar/stats.html:38:  <td style="text-align: right;">
    ./oscar/stats.html:91:  <td style="text-align: right;"> {{ engine_reliabilities.get(engine_stat.name, {}).get('reliablity') }}</td>
    ./oscar/stats.html:109: <tbody style="padding-top: 1rem;">
    ./oscar/stats.html:112: <th scope="row" style="width: 10rem">{{ _('Exception') }}</th><td>{{ error.exception_classname }}</td>
    ./oscar/stats.html:114: <th scope="row" style="width: 10rem">{{ _('Message') }}</th><td>{{ error.log_message }}</td>
    ./oscar/stats.html:116: <th scope="row" style="width: 10rem">{{ _('Percentage') }}</th><td style="width: 10rem">{{ error.percentage }}</td>
    ./oscar/stats.html:121: <span style="border-right: 1px solid gray; padding: 0 1rem 0 0; margin: 0 0 0 0.5rem;">{{ param }}</span>
    ./oscar/stats.html:138: <th scope="col" style="width: 10rem">{{ _('Failed test') }}</th>

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
0a6c488417 [fix] make /stats more CSP compliant - simple theme
Replace simple theme's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/simple/less/stats.less`` ::

    ./simple/stats.html:26:  <table style="max-width: 1280px; margin: 0 auto 0 0;">
    ./simple/stats.html:28:  <th scope="col" style="width:20rem;">{{ th_sort('name', _("Engine name")) }}</th>
    ./simple/stats.html:29:  <th scope="col" style="width:7rem; text-align: right;">{{ th_sort('score', _('Scores')) }}</th>
    ./simple/stats.html:32:  <th scope="col" style="text-align: right;">{{ th_sort('reliability', _('Reliability')) }}</th>
    ./simple/stats.html:37:  <td style="text-align: right;">
    ./simple/stats.html:90:  <td style="text-align: right;"> {{ engine_reliabilities.get(engine_stat.name, {}).get('reliablity') }}</td>
    ./simple/stats.html:106: <table style="max-width: 1280px; margin: 1rem; border: 1px solid gray;">
    ./simple/stats.html:107: <tbody style="padding-top: 1rem;">
    ./simple/stats.html:110: <th scope="row" style="width: 10rem">{{ _('Exception') }}</th><td>{{ error.exception_classname }}</td>
    ./simple/stats.html:112: <th scope="row" style="width: 10rem">{{ _('Message') }}</th><td>{{ error.log_message }}</td>
    ./simple/stats.html:114: <th scope="row" style="width: 10rem">{{ _('Percentage') }}</th><td style="width: 10rem">{{ error.percentage }}</td>
    ./simple/stats.html:119: <span style="border-right: 1px solid gray; padding: 0 1rem 0 0; margin: 0 0 0 0.5rem;">{{ param }}</span>
    ./simple/stats.html:136: <th scope="col" style="width: 10rem">{{ _('Failed test') }}</th>

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
cdfb4b7ff9 [fix] make /stats more CSP compliant - bar graph
Replace bar graph's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/__common__/less/stats.less`` ::

    ./simple/stats.html:49: <span style="width: calc(max(2px, 100%*{{ (engine_stat.result_count / engine_stats.max_result_count )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./simple/stats.html:57: <span style="width: calc(max(2px, 100%*{{ (engine_stat.http / engine_stats.max_time )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./simple/stats.html:58: <span style="width: calc(100%*{{ engine_stat.processing / engine_stats.max_time |round(3) }})" class="stacked-bar-chart-serie2"></span>{{- "" -}}
    ./oscar/stats.html:50:  <span style="width: calc(max(2px, 100%*{{ (engine_stat.result_count / engine_stats.max_result_count )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./oscar/stats.html:58:  <span style="width: calc(max(2px, 100%*{{ (engine_stat.http / engine_stats.max_time )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./oscar/stats.html:59:  <span style="width: calc(100%*{{ engine_stat.processing / engine_stats.max_time |round(3) }})" class="stacked-bar-chart-serie2"></span>{{- "" -}}

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
965817f294 [fix] simple theme - generate missing sourceMap file
C&P from searx/static/themes/oscar/gruntfile.js

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00