Commit graph

3040 commits

Author SHA1 Message Date
Adam Tauber
e4b6558339 [enh] add redis offline engine / https://redis.io/
Slightly modified merge of commit [97269be6], [01a8a5814a] and [c8d2b5eb] from
searx.

[97269be6] https://github.com/searx/searx/commit/97269be6
[01a8a581] https://github.com/searx/searx/commit/01a8a581
[c8d2b5eb] https://github.com/searx/searx/commit/c8d2b5eb

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 11:51:25 +02:00
Adam Tauber
e33858d6ef [fix] correct kv template formatting and remove internal data
Slightly modified merge of commit [ea7ccf24] from searx.

[ea7ccf24] https://github.com/searx/searx/commit/ea7ccf24

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 10:53:37 +02:00
Alexandre Flament
925c21ce42
Merge pull request #122 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-06-01 10:03:50 +02:00
Alexandre Flament
3266738e7a
Merge pull request #121 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-06-01 10:03:35 +02:00
Alexandre Flament
54ae95b95b
Merge pull request #120 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-06-01 10:03:22 +02:00
dalf
43d06e8224 Update searx.data - update_ahmia_blacklist.py 2021-06-01 08:01:50 +00:00
dalf
e53ed4742c Update searx.data - update_currencies.py 2021-06-01 08:01:48 +00:00
dalf
666a25314a Update searx.data - update_wikidata_units.py 2021-06-01 08:01:45 +00:00
dalf
9f3a455777 Update searx.data - update_firefox_version.py 2021-06-01 08:01:33 +00:00
Alexandre Flament
6fa114c9ba [mod] settings_default: remove searx.search.max_request_timeout global variable 2021-06-01 08:10:15 +02:00
Alexandre Flament
4b07df62e5 [mod] move all default settings into searx.settings_defaults 2021-06-01 08:10:15 +02:00
Alexandre Flament
6ca74386d6
[fix] mankier: fix paging
See https://github.com/searxng/searxng/pull/111#issuecomment-851641967
2021-05-31 22:33:40 +02:00
Kyle Anthony Williams
d6a2d4f969 [enh] add engine - Docker Hub
Slightly modified merge of commit [1cb1d3ac] from searx [PR 2543]:

      This adds Docker Hub .. as a search engine .. the engine's favicon was
      downloaded from the Docker Hub website with wget and converted to a PNG
      with ImageMagick .. It supports the parsing of URLs, titles, content,
      published dates, and thumbnails of Docker images.

[1cb1d3ac] https://github.com/searx/searx/pull/2543/commits/1cb1d3ac
[PR 2543] https://github.com/searx/searx/pull/2543

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-30 15:18:36 +02:00
Alexandre Flament
1113f7e616 [mod] the bittorent search engines are available only in the files category
related to #101
2021-05-29 16:14:19 +02:00
Alexandre Flament
e239369f1f
Merge pull request #109 from return42/sql-engines
Merge MySQL and PostgreSQL from searx
2021-05-29 08:34:19 +02:00
Alexandre Flament
c22c7cf35b
Merge pull request #111 from return42/mankier
[enh] add engine ManKier (JSON) / ManKier: Linux man pages
2021-05-29 08:33:54 +02:00
Jordan Webb
a8f474b23f [enh] add engine PyPi (XPath)
Add a XPath configuration for PyPI (Python Package Index)

Slightly modified merge of [66d06b05] from searx's [PR 2830].

[PR 2830] https://github.com/searx/searx/pull/2830
[66d06b05] https://github.com/searx/searx/pull/2830/commits/66d06b05

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-29 05:51:33 +02:00
Jordan Webb
52672d887b [enh] add engine ManKier (JSON) / ManKier: Linux man pages
Slightly modified merge of [a881e742aa] from searx's [PR 2829].

[PR 2829] https://github.com/searx/searx/pull/2829
[a881e742aa] https://github.com/searx/searx/pull/2829/commits/a881e742aa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-29 05:29:47 +02:00
Noémi Ványi
87a01a1736 [enh] add MySQL engine
Slightly modified merge of [c00a33fe] from searx.

[c00a33fe] c00a33feee

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 17:36:46 +02:00
Noémi Ványi
324aa96062 [enh] add PostgreSQL engine
Slightly modified merge of [22079ff] from searx.

[22079ff] 22079ffdef

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 17:34:44 +02:00
Allen
a4b0746044 [enh] add metasearch engin dogpile.com (XPath)
Add Dogpile engine [1] / merged from [2]

[1] https://www.dogpile.com
[2] https://github.com/searx/searx/pull/2822
2021-05-28 15:52:28 +02:00
LL Productions FR
15f1d6a80f [enh] add engine Wikimini (fr.wikimini.org)
Online encyclopedia for children (only language fr), merged from [1]

[1] https://github.com/searx/searx/pull/2819
2021-05-28 14:03:21 +02:00
Alexandre Flament
83ccf7b04d
Merge pull request #99 from return42/webapp-misc
[enh] add settings option to enable/disable search formats
2021-05-28 13:26:48 +02:00
Markus Heiser
96b223023a [mod] utils.get_value() - avoidance of a recursion
In a comment [1] dalf suggested to avoid a recursion of get_value()

[1] https://github.com/searxng/searxng/pull/99#discussion_r640833716

Suggested-by: Alexandre Flament <alex@al-f.net>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Markus Heiser
6ed4616da9 [enh] add settings option to enable/disable search formats
Access to formats can be denied by settings configuration::

    search:
        formats: [html, csv, json, rss]

Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Alexandre Flament
b48b4c93d5
Merge pull request #100 from return42/webapp-pylint
[pylint] webapp.py
2021-05-27 17:04:32 +02:00
Markus Heiser
32b5a0ef7b
Merge pull request #93 from return42/genius-misc
Some minor Genius improvements
2021-05-27 14:23:22 +00:00
Markus Heiser
25b5797a0c
Merge pull request #103 from searxng/add-sqlite-engine2
[enh] add offline engine for sqlite database
2021-05-27 14:06:42 +00:00
Alexandre Flament
b1da97d092
Merge pull request #96 from return42/unsplash
lint & fix unsplash engine
2021-05-27 14:58:59 +02:00
Alexandre Flament
2ea34a3c36 [enh] add offline engine for sqlite database
To test & demonstrate this implementation download:

  https://liste.mediathekview.de/filmliste-v2.db.bz2

and unpack into searx/data/filmliste-v2.db, in your settings.yml define a sqlite
engine named "demo"::

    - name : demo
      engine : sqlite
      shortcut: demo
      categories: general
      result_template: default.html
      database : searx/data/filmliste-v2.db
      query_str :  >-
        SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
               COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
               description AS content
          FROM film
         WHERE title LIKE :wildcard OR description LIKE :wildcard
         ORDER BY duration DESC
      disabled : False

Query to test: "!demo concert"

This is a rewrite of the implementation from commit [1]

[1] searx/searx@8e90a21

Suggested-by: @virtadpt searx/searx#2808
2021-05-27 14:27:11 +02:00
Markus Heiser
e1f244b2d5 [pylint] webapp.py - fix all messages from pylint
No functional change, just some linting.

- fix messages from pylint (see below)
- log where general Exceptions are catched (broad-except)
- normalized various indentation
- To avoid clashes with common names, add prefix 'route_' to all @app.route
  decorated functions.

Fixed messages::

    searx/webapp.py:744:0: C0301: Line too long (146/120) (line-too-long)
    searx/webapp.py:756:0: C0301: Line too long (132/120) (line-too-long)
    searx/webapp.py:730:9: W0511: TODO, check if timezone is calculated right (fixme)
    searx/webapp.py:1:0: C0114: Missing module docstring (missing-module-docstring)
    searx/webapp.py:126:8: I1101: Module 'setproctitle' has no 'setthreadtitle' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
    searx/webapp.py:126:36: W0212: Access to a protected member _name of a client class (protected-access)
    searx/webapp.py:131:4: R1722: Consider using sys.exit() (consider-using-sys-exit)
    searx/webapp.py:141:4: R1722: Consider using sys.exit() (consider-using-sys-exit)
    searx/webapp.py:255:38: W0621: Redefining name 'request' from outer scope (line 32) (redefined-outer-name)
    searx/webapp.py:307:4: W0702: No exception type(s) specified (bare-except)
    searx/webapp.py:374:24: W0621: Redefining name 'theme' from outer scope (line 155) (redefined-outer-name)
    searx/webapp.py:420:8: R1705: Unnecessary "else" after "return" (no-else-return)
    searx/webapp.py:544:4: W0621: Redefining name 'preferences' from outer scope (line 917) (redefined-outer-name)
    searx/webapp.py:551:4: W0702: No exception type(s) specified (bare-except)
    searx/webapp.py:566:15: W0703: Catching too general exception Exception (broad-except)
    searx/webapp.py:613:4: R1705: Unnecessary "elif" after "return" (no-else-return)
    searx/webapp.py:690:8: W0621: Redefining name 'search' from outer scope (line 661) (redefined-outer-name)
    searx/webapp.py:661:0: R0914: Too many local variables (22/20) (too-many-locals)
    searx/webapp.py:674:8: R1705: Unnecessary "else" after "return" (no-else-return)
    searx/webapp.py:697:11: W0703: Catching too general exception Exception (broad-except)
    searx/webapp.py:748:4: R1705: Unnecessary "elif" after "return" (no-else-return)
    searx/webapp.py:661:0: R0911: Too many return statements (9/6) (too-many-return-statements)
    searx/webapp.py:661:0: R0912: Too many branches (29/12) (too-many-branches)
    searx/webapp.py:661:0: R0915: Too many statements (74/50) (too-many-statements)
    searx/webapp.py:931:4: W0621: Redefining name 'image_proxy' from outer scope (line 1072) (redefined-outer-name)
    searx/webapp.py:946:4: W0621: Redefining name 'stats' from outer scope (line 1132) (redefined-outer-name)
    searx/webapp.py:917:0: R0914: Too many local variables (34/20) (too-many-locals)
    searx/webapp.py:917:0: R0912: Too many branches (19/12) (too-many-branches)
    searx/webapp.py:917:0: R0915: Too many statements (65/50) (too-many-statements)
    searx/webapp.py:1063:44: W0621: Redefining name 'preferences' from outer scope (line 917) (redefined-outer-name)
    searx/webapp.py:1072:0: R0911: Too many return statements (9/6) (too-many-return-statements)
    searx/webapp.py:1151:4: C0103: Variable name "SORT_PARAMETERS" doesn't conform to '(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*)|([a-z]))$' pattern (invalid-name)
    searx/webapp.py:1297:0: R1721: Unnecessary use of a comprehension (unnecessary-comprehension)
    searx/webapp.py:1303:0: C0103: Argument name "e" doesn't conform to '(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*))$' pattern (invalid-name)
    searx/webapp.py:1303:19: W0613: Unused argument 'e' (unused-argument)
    searx/webapp.py:1338:23: W0621: Redefining name 'app' from outer scope (line 162) (redefined-outer-name)
    searx/webapp.py:1318:0: R0903: Too few public methods (1/2) (too-few-public-methods)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-27 12:30:56 +02:00
Markus Heiser
ae25362205 [pylint] webapp.py - fix all wrong-import-order messages from pylint
pylint message: wrong-import-order
  Respect PEP8 import order (standard imports first, then third-party libraries,
  then local imports).

pylint message: wrong-import-position
  Do not mix code & imports

BTW:

- only one import per line
- replace licence text by SPDX tag

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-26 21:17:19 +02:00
Markus Heiser
85364edaf0 [fix] webapp.py - running without installation is not supported
Remove extension of the sys.path (aka PYTHONPATH).  Running instance directly
from repository's folder is a relict from the early beginning in
2014 (fd651083f) and is no longer supported.

Since commit dd46629 was merged the command line 'searx-run' exists and should
be used.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-26 21:17:19 +02:00
Alexandre Flament
7c76cef746
Merge pull request #94 from return42/pylint-network
Pylint searx.network
2021-05-26 18:35:29 +02:00
Markus Heiser
97d173eda0 [mod] unsplash engine - activated by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 17:41:10 +02:00
Markus Heiser
dc21cb5d4b [fix] unsplash engine - 'searx:result: invalid title:'
- Use result 'alt_description' as title, if not given use
  default title 'unknown'.
- Use result 'description' from unsplash as 'content'

Fix error::

    DEBUG:searx:result: invalid title: {..., 'title': None, 'content': '', 'engine': 'unsplash'}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 17:26:58 +02:00
Markus Heiser
a88e3e4fea [pylint] searx/engines/unsplash.py, add logger & norm indentation
- fix messages from pylint
- add logger and log request URL
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 16:45:32 +02:00
Markus Heiser
2128022f72 [coding-style] searx/network/network.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:48:47 +02:00
Markus Heiser
1499002ceb [coding-style] searx/network/client.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:44:43 +02:00
Markus Heiser
e4211da639 [pylint] searx/network/raise_for_httperror.py
No functional change!

- fix messages from pylint
- add ``global NETWORKS``
- normalized indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:40:10 +02:00
Markus Heiser
44efa911ba [pylint] searx/network/network.py & add global (NETWORKS)
No functional change!

- fix messages from pylint
- add ``global NETWORKS``

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:46 +02:00
Markus Heiser
b595c482d0 [pylint] searx/network/client.py & add global (TRANSPORT_KWARGS)
No functional change!

- fix messages from pylint
- add ``global TRANSPORT_KWARGS``
- normalized python_socks imports

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:37 +02:00
Markus Heiser
8033518899 [pylint] searx/network/__init__.py & add global (THREADLOCAL)
No functional change!

- fix messages from pylint
- add ``global THREADLOCAL``
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:14 +02:00
Markus Heiser
f963759ccc [fix] engine genius should not use the video template
Remove 'template' from result.  Engine genius should
not use the video template.  BTW: fix indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 16:31:14 +02:00
Markus Heiser
3a71d4b175 [pylint] searx/engines/genius.py, add logger & normalized indentation
- pylint searx/engines/genius.py
- add logger and log ignored exceptions
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 16:19:06 +02:00
Markus Heiser
84a943f867 [enh] XPath engine - add time safe-search support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 22:26:18 +02:00
Markus Heiser
6bfe3fd033 [enh] XPath engine - add time range support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 16:49:30 +02:00
Markus Heiser
1933577c8e [enh] XPath engine - add ISO 639-1 {lang} replacement to search-URL
BTW: remove obsolte params['query'] and not needed paging condition.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 15:05:36 +02:00
Markus Heiser
8cd544b2a6 [doc] add documentation about the XPath engine
- pylint searx/engines/xpath.py
- fix indentation of some long lines
- add logging
- add doc-strings

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 11:48:21 +02:00
Markus Heiser
7728e25b11 [refactor] metrics.get_reliabilities() - make code more readable
- init stat values by None
- drop round_or_none
- don't try to get percentage if base is 'None'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-22 15:17:18 +02:00
Alexandre Flament
99aaf86b73 [fix] offline engines: fix templates /stats and /preferences 2021-05-22 15:17:18 +02:00
Alexandre Flament
3014463fed [fix] metrics: processing time = total time if there is no http time
It was previsouly None

Fix /stats
2021-05-22 15:17:18 +02:00
Alexandre Flament
ec83493538 [fix] offline engine: don't crash on time recording 2021-05-22 15:17:18 +02:00
Markus Heiser
6f1446d55f [pylint] searx/search/__init__.py & replace lic-text by SPDX tag
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-21 17:31:22 +02:00
Alexandre Flament
426fadccb3 [mod] remove gc.collect() after each user request 2021-05-21 17:23:18 +02:00
Alexandre Flament
d1cffc55cb [fix] unit test: don't load /etc/searx/settings.yml
Add a new environment variable SEARX_DISABLE_ETC_SETTINGS
to disable loading of /etc/searx/settings.yml

unit tests:
* set SEARX_DISABLE_ETC_SETTINGS to 1
* remove SEARX_SETTINGS_PATH if it exists
2021-05-18 17:23:21 +02:00
Alexandre Flament
d255e5637b
Merge pull request #81 from return42/csp-bar-graph
[fix] make /stats more CSP compliant
2021-05-18 08:55:56 +02:00
Markus Heiser
73d73dcf9b [static] make /stats more CSP compliant- make themes.all
Based on commits

- 0507e185 [fix] bar graph and rename CSS class engine-scores -> engine-score
- 3e9ad7ae [fix] make /stats more CSP compliant - github issue form
- 34859d0e [fix] make /stats more CSP compliant - oscar theme
- 0a6c4884 [fix] make /stats more CSP compliant - simple theme
- cdfb4b7f [fix] make /stats more CSP compliant - bar graph
- 965817f2 [fix] simple theme - generate missing sourceMap file

this patch is generated by::

     make themes.all

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:25:32 +02:00
Markus Heiser
0507e185a5 [fix] bar graph and rename CSS class engine-scores -> engine-score
- drop #main_stats selector in stats.less
- 'engine-score' exists before this PR.
- untabify searx/static/themes/__common__/less/stats.less

for details see comment at: d93bec7638..1204e4f07e (r633571496)

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:24:19 +02:00
Markus Heiser
3e9ad7ae0c [fix] make /stats more CSP compliant - github issue form
Hide textarea from github issue form::

    ./__common__/new_issue.html:6:    <textarea name="body" style="display: none;">{{- '' -}}

BTW: fix indentation.

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 21:23:27 +02:00
Markus Heiser
34859d0e58 [fix] make /stats more CSP compliant - oscar theme
Replace oscar theme's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/__common__/less/stats.less`` ::

    ./oscar/stats.html:29:  <th scope="col" style="width:20rem;">{{ th_sort('name', _("Engine name")) }}</th>
    ./oscar/stats.html:30:  <th scope="col" style="width:7rem; text-align: right;">{{ th_sort('score', _('Scores')) }}</th>
    ./oscar/stats.html:33:  <th scope="col" style="text-align: right;">{{ th_sort('reliability', _('Reliability')) }}</th>
    ./oscar/stats.html:38:  <td style="text-align: right;">
    ./oscar/stats.html:91:  <td style="text-align: right;"> {{ engine_reliabilities.get(engine_stat.name, {}).get('reliablity') }}</td>
    ./oscar/stats.html:109: <tbody style="padding-top: 1rem;">
    ./oscar/stats.html:112: <th scope="row" style="width: 10rem">{{ _('Exception') }}</th><td>{{ error.exception_classname }}</td>
    ./oscar/stats.html:114: <th scope="row" style="width: 10rem">{{ _('Message') }}</th><td>{{ error.log_message }}</td>
    ./oscar/stats.html:116: <th scope="row" style="width: 10rem">{{ _('Percentage') }}</th><td style="width: 10rem">{{ error.percentage }}</td>
    ./oscar/stats.html:121: <span style="border-right: 1px solid gray; padding: 0 1rem 0 0; margin: 0 0 0 0.5rem;">{{ param }}</span>
    ./oscar/stats.html:138: <th scope="col" style="width: 10rem">{{ _('Failed test') }}</th>

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
0a6c488417 [fix] make /stats more CSP compliant - simple theme
Replace simple theme's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/simple/less/stats.less`` ::

    ./simple/stats.html:26:  <table style="max-width: 1280px; margin: 0 auto 0 0;">
    ./simple/stats.html:28:  <th scope="col" style="width:20rem;">{{ th_sort('name', _("Engine name")) }}</th>
    ./simple/stats.html:29:  <th scope="col" style="width:7rem; text-align: right;">{{ th_sort('score', _('Scores')) }}</th>
    ./simple/stats.html:32:  <th scope="col" style="text-align: right;">{{ th_sort('reliability', _('Reliability')) }}</th>
    ./simple/stats.html:37:  <td style="text-align: right;">
    ./simple/stats.html:90:  <td style="text-align: right;"> {{ engine_reliabilities.get(engine_stat.name, {}).get('reliablity') }}</td>
    ./simple/stats.html:106: <table style="max-width: 1280px; margin: 1rem; border: 1px solid gray;">
    ./simple/stats.html:107: <tbody style="padding-top: 1rem;">
    ./simple/stats.html:110: <th scope="row" style="width: 10rem">{{ _('Exception') }}</th><td>{{ error.exception_classname }}</td>
    ./simple/stats.html:112: <th scope="row" style="width: 10rem">{{ _('Message') }}</th><td>{{ error.log_message }}</td>
    ./simple/stats.html:114: <th scope="row" style="width: 10rem">{{ _('Percentage') }}</th><td style="width: 10rem">{{ error.percentage }}</td>
    ./simple/stats.html:119: <span style="border-right: 1px solid gray; padding: 0 1rem 0 0; margin: 0 0 0 0.5rem;">{{ param }}</span>
    ./simple/stats.html:136: <th scope="col" style="width: 10rem">{{ _('Failed test') }}</th>

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
cdfb4b7ff9 [fix] make /stats more CSP compliant - bar graph
Replace bar graph's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/__common__/less/stats.less`` ::

    ./simple/stats.html:49: <span style="width: calc(max(2px, 100%*{{ (engine_stat.result_count / engine_stats.max_result_count )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./simple/stats.html:57: <span style="width: calc(max(2px, 100%*{{ (engine_stat.http / engine_stats.max_time )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./simple/stats.html:58: <span style="width: calc(100%*{{ engine_stat.processing / engine_stats.max_time |round(3) }})" class="stacked-bar-chart-serie2"></span>{{- "" -}}
    ./oscar/stats.html:50:  <span style="width: calc(max(2px, 100%*{{ (engine_stat.result_count / engine_stats.max_result_count )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./oscar/stats.html:58:  <span style="width: calc(max(2px, 100%*{{ (engine_stat.http / engine_stats.max_time )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./oscar/stats.html:59:  <span style="width: calc(100%*{{ engine_stat.processing / engine_stats.max_time |round(3) }})" class="stacked-bar-chart-serie2"></span>{{- "" -}}

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
965817f294 [fix] simple theme - generate missing sourceMap file
C&P from searx/static/themes/oscar/gruntfile.js

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
93594a7b42 [less] update grunt-contrib-less v3.3.0
Upgraded [v3.3.0] otherwise::

`  width: calc(100% - 5rem);`

becomes `width: 95%` once compiled by less version 1.4.1.

[v3.3.0] https://github.com/gruntjs/grunt-contrib-less/releases/tag/v3.0.0

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 17:41:22 +02:00
Markus Heiser
57cdf180fe [fix] engine archive is - search_url has been changed
BTW: set soft_max_redirects from *archive is* by 1, to prevent logging::

    DEBUG:httpx._client:HTTP Request: GET https://archive.is/search/?q=www.python.org "HTTP/2 302 Found"
    DEBUG:httpx._client:HTTP Request: GET https://archive.is/www.python.org "HTTP/2 200 OK"
    DEBUG:searx:archive is: ErrorContext('searx/search/processors/online.py', 110, 'count_error(self.engine_name,', None, '1 redirects, maximum: 0', ('200', 'OK', 'archive.is')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 15:21:53 +02:00
Markus Heiser
ffcebf5e12 [enh] xpath engine - add request parameter 'soft_max_redirects'
Make 'soft_max_redirects' configurable per Xpath engine::

    - name : <engine-name>
      engine : xpath
      soft_max_redirects: 1
      ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 15:04:55 +02:00
Alexandre Flament
90fe232e08 [fix] webapp.py: pylint: disable=assigning-non-slot for request.*
Note: actually, the code should use flask.g.* instead of request.*
2021-05-14 11:15:35 +02:00
Alexandre Flament
d12033e166 [mod] Server-Timing header: add render field
Add a "render" field to the Server-Timing header
This field is the time spent in the searx.webapp.render function (Jinja2 rendering)

See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing
2021-05-06 09:39:52 +02:00
Alexandre Flament
0f4e995ab4 [mod] searx.network.client: the same configuration reuses the same ssl.SSLContext
before there was one ssl.SSLContext per client.

see https://github.com/encode/httpx/issues/978
2021-05-05 20:36:37 +02:00
Markus Heiser
fa0d05c313 [pylint] checker/__main__.py & checker/background.py
Lint files that has been touched by [PR #58]

[PR #58] https://github.com/searxng/searxng/pull/58

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-05 16:47:02 +02:00
Alexandre Flament
8c1a65d32f [mod] multithreading only in searx.search.* packages
it prepares the new architecture change,
everything about multithreading in moved in the searx.search.* packages

previously the call to the "init" function of the engines was done in searx.engines:
* the network was not set (request not sent using the defined proxy)
* it requires to monkey patch the code to avoid HTTP requests during the tests
2021-05-05 13:12:42 +02:00
Markus Heiser
d36adfa59f
Merge pull request #55 from searxng/upd-relialibility-column
[themes] /preferences, reliablity: warning icons & error logs link
2021-05-04 12:20:45 +00:00
Markus Heiser
5253df946b
[mod] option to enable or disable "proxy" button next to each result (#54)
* [mod] option to enable or disable "proxy" button next to each result

Closes: https://github.com/searxng/searxng/issues/51
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Alexandre Flament <alex@al-f.net>
2021-05-04 11:57:14 +02:00
Alexandre Flament
e83862c22f [themes] /preferences, reliablity: warning icons & error logs link
When there is at least one errors or one failed checker test:
* the warning icon is displayed in the reliability column
* the link "View error logs and submit a bug report" is displayed on engine name tooltip.

Before:
* the warning icon was displayed only when one or more checker test(s) failed.
* the link "View error logs and submit a bug report" was not shown when a checker test failed but there were no error.
2021-05-04 10:50:03 +02:00
Markus Heiser
45293270e2
Merge pull request #44 from searxng/dependabot/pip/master/pylint-2.8.2
Bump pylint from 2.7.4 to 2.8.2
2021-05-03 13:58:14 +00:00
dependabot[bot]
2a53f79a2d Bump pylint from 2.7.4 to 2.8.2
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.7.4 to 2.8.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/pylint-2.7.4...v2.8.2)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-03 15:45:30 +02:00
Marc Abonce Seguin
448bfe6005 fix Qwant's fetch_languages function 2021-05-02 17:46:40 -07:00
Alexandre Flament
2756e170ce
Merge pull request #49 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-05-01 10:28:32 +02:00
Alexandre Flament
a3a7be90ff
Merge pull request #50 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-05-01 10:28:22 +02:00
Alexandre Flament
4558c0c027
Merge pull request #48 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-05-01 10:28:11 +02:00
dalf
dd8a5a00dc Update searx.data - update_currencies.py 2021-05-01 08:23:05 +00:00
dalf
de54209f9e Update searx.data - update_ahmia_blacklist.py 2021-05-01 08:23:00 +00:00
dalf
982a5a7d2e Update searx.data - update_wikidata_units.py 2021-05-01 08:22:48 +00:00
dalf
c004712eef Update searx.data - update_firefox_version.py 2021-05-01 08:22:43 +00:00
Alexandre Flament
e5bcc3cbbe
Merge pull request #40 from searxng/link-to-stats
[mod] engine errors: link to the stats to create an github issue
2021-04-29 14:37:12 +02:00
Markus Heiser
8026ed0d37 [mod] UI: engine pref-page, about toolbox: add link to stats
In the preference page, in the 'about' toolbox of an engine, add a link to the
stats page of the engine, if the engine had one or more errors.

Condition is::

    reliabilities[<engine.name>].errors

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-29 11:11:05 +02:00
Alexandre Flament
34bced29ae [mod] engine errors: link to the stats to create an github issue 2021-04-29 11:10:38 +02:00
Michael Ilsaas
0c43cf89ca [fix] URL to solidtorrent result page
Reported-by: https://github.com/searx/searx/pull/2786
2021-04-29 10:40:47 +02:00
Markus Heiser
8291804082 [static] make themes.all - from commit 9e8171e38
Based on commit 9e8171e38 this patch is generated by::

     make themes.all

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 16:22:47 +02:00
Markus Heiser
70cc196e2d [fix] new_issue.html: drop inline style attributes (CSP conformance)
Inline styles are blocked by default with Content Security Policy (CSP).  Move
the inline styles from 'new_issue.html' to::

    searx/static/themes/__common__/less/new_issue.less

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 16:22:37 +02:00
Markus Heiser
7f2dc530b2 [fix] simple theme - remove no longer used stats.less file
File searx/static/themes/simple/less/stats.less is not used (imported) in any
other less file.  I can't say when it's usage was dropped or if it has ever been
used.  ATM this file is without any usage.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 15:42:08 +02:00
Markus Heiser
54d57c8296 [fix] grunt/less/development: oscar light & dark logicodev mixes CSS
closes: https://github.com/searxng/searxng/issues/39

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 15:15:29 +02:00
Alexandre Flament
283ae7bfad [fix] searx.network: fix rare cases where LOOP is None
* searx.network.client.LOOP is initialized in a thread
* searx.network.__init__ imports LOOP which may happen
  before the thread has initialized LOOP

This commit adds a new function "searx.network.client.get_loop()"
to fix this issue
2021-04-27 17:47:36 +02:00
Alexandre Flament
87e914e398
Merge pull request #28 from searxng/mod-processors-error-message
[mod] processors: show identical error messages on /search and /stats
2021-04-27 16:50:41 +02:00
Markus Heiser
924f9afea3 [lint] pylint searx/search/processors files / BTW add some doc-strings
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 15:13:39 +02:00
Alexandre Flament
b1557b5443 [mod] processors: show identical error messages on /search and /stats 2021-04-27 14:20:07 +02:00
Alexandre Flament
b8f5adf500 [mod] searx.metrics.error_recorder: store relative file name instead of the full absolute file name. 2021-04-27 10:42:00 +02:00
Markus Heiser
dc29f1d826 [pylint] tag PYLINT_FILES by comment # lint: pylint
These py files are linted by `test.pylint`, all other files are linted by
`test.pep8`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 20:18:20 +02:00
Markus Heiser
6b92e8c6fd [upd] ./manage pyenv.cmd searx_extra/update/update_pygments.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 19:11:58 +02:00
Markus Heiser
1a402be175 [fix] debug log: UnicodeEncodeError: 'ascii' codec can't encode
The issue exists only in the debug log::

     --- Logging error ---
     Traceback (most recent call last):
       File "/usr/lib/python3.9/logging/__init__.py", line 1086, in emit
	 stream.write(msg + self.terminator)
     UnicodeEncodeError: 'ascii' codec can't encode characters in position 79-89: ordinal not in range(128)
     Call stack:
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
	 return self.wsgi_app(environ, start_response)
       File "/usr/local/searx/searx-src/searx/webapp.py", line 1316, in __call__
	 return self.app(environ, start_response)
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
	 return self.app(environ, start_response)
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
	 response = self.full_dispatch_request()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
	 rv = self.dispatch_request()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
	 return self.view_functions[rule.endpoint](**req.view_args)
       File "/usr/local/searx/searx-src/searx/webapp.py", line 766, in search
	 number_of_results=format_decimal(number_of_results),
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask_babel/__init__.py", line 458, in format_decimal
	 locale = get_locale()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask_babel/__init__.py", line 226, in get_locale
	 rv = babel.locale_selector_func()
       File "/usr/local/searx/searx-src/searx/webapp.py", line 249, in get_locale
	 logger.debug("%s uses locale `%s` from %s", request.url, locale, locale_source)
     Unable to print the message and arguments - possible formatting error.
     Use the traceback above to help find the error.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 12:55:26 +02:00
Alexandre Flament
51494849fc
Merge pull request #23 from searxng/metrics-stats-engines
Metrics stats engines
2021-04-25 14:31:52 +02:00
Alexandre Flament
df41b77121 [mod] /stats : detail per engine
allow to submit a github issue including the technical details
(exceptions, errors, warning, checker result)
2021-04-25 14:28:06 +02:00
Markus Heiser
28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 14:25:08 +02:00
Markus Heiser
c6a5cc019a [brand] searxng is a fork from searx
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 11:42:42 +02:00
Markus Heiser
6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
spongebob33
7528e38c8a add core.ac.uk engine 2021-04-24 08:55:45 +02:00
Alexandre Flament
525bfe7830
Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament
7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament
09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Markus Heiser
c54bf42cb9 [mod] metrics: add to pylint and justify indentations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 06:58:49 +02:00
Alexandre Flament
65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00
Alexandre Flament
3755cf2eba More branding 2021-04-22 21:10:22 +02:00
Alexandre Flament
04b34c70b8 More branding 2021-04-22 20:14:16 +02:00
Alexandre Flament
9d84025e4f searxng brand 2021-04-22 19:57:02 +02:00
Alexandre Flament
d01741c9a2
Merge pull request #15 from return42/add-springer
Add a search engine for Springer Nature
2021-04-22 13:23:31 +02:00
Markus Heiser
213041adc7 [enh] themes - add result.metadata to result sub header
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-22 12:37:18 +02:00
Pierre Chevalier
a80bf1ba97 [enh] Add Springer Nature engine
Springer Nature is a global publisher dedicated to providing service to research
community [1] with official API [2].

To test this PR, first get your API key following this page:

   https://dev.springernature.com/signup

In searx/engines/springer.py at line 24, add this API key.  I left my own key,
commented out in the line aboce.  Feel free to use it, if needed.

[1] https://www.springernature.com/
[2] https://dev.springernature.com/
2021-04-22 12:35:25 +02:00
habsinn
41a2e3785e [enh] add engine using API from "The Art Institute of Chicago" 2021-04-22 12:25:43 +02:00
Alexandre Flament
981cdb7acf
Merge pull request #10 from return42/fix-youtube
[fix] youtube - send CONSENT Cookie to not be redirected
2021-04-22 12:21:31 +02:00
Markus Heiser
9a05ad4c73 [enh] add DOI resolver from sci-hub / replace default DOI
The new sci-hub URLs are comming from @aurora-vasiliev [1].

[1] https://github.com/searx/searx/pull/2706

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 12:13:01 +02:00
Markus Heiser
e9a6ab4015 [fix] youtube - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, youtube requests are redirected to confirm and get a CONSENT
Cookie from https://consent.youtube.com

This patch adds a CONSENT Cookie to the youtube request to avoid redirection.

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Reported-by: https://github.com/searx/searx/issues/2774
2021-04-22 12:09:09 +02:00
Markus Heiser
247c46c6b0 [drop] plugin HTTPS rewrite - rule base is to old (update is to big)
The usefulness of the _HTTPS rewrite_ plugin is questionable:

- the 36 rule files have not been updated since 2015 [1]
- actual there are 23760 rule files in the https-everywhere repo [2]

For the first, we can remove this plugin.  For a complete new implementation, it
might be good to know that there is a project "https-everywhere : Privacy for
Pythons" [3]

related:  https://github.com/return42/searx-next/issues/8

[1] https://github.com/return42/searx-next/tree/d187a1d/searx/plugins/https_rules
[2] https://github.com/EFForg/https-everywhere/tree/master/src/chrome/content/rules
[3] https://github.com/jayvdb/https-everywhere-py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 11:47:16 +02:00
Alexandre Flament
c6d5605d27
Merge pull request #7 from searxng/metrics
Metrics
2021-04-22 08:34:17 +02:00
Alexandre Flament
baff1cbbab fix issues from review 2021-04-21 18:49:13 +02:00
Alexandre Flament
b7848e3422 [fix] searxng fix: sjp engine 2021-04-21 16:31:29 +02:00
Alexandre Flament
7cfd8d900a [mod] oscar: /preferences , engines tab: report engine times
* display the median time instead of the average.
* add a "Reliability" column (sum up the metrics and the checker results).
* the "selected language", "SafeSearch", "Time range" values are displayed as "broken" when the checker tests fail.
2021-04-21 16:24:46 +02:00
Alexandre Flament
c27fef1cde [mod] metrics: add secondary parameter
Some error won't stop the engine:
* additional HTTP redirects for example
* some invalid results

secondary=True allows to flag these errors as not important.
2021-04-21 16:24:46 +02:00
Alexandre Flament
7acd7ffc02 [enh] rewrite and enhance metrics 2021-04-21 16:24:46 +02:00
Alexandre Flament
aae7830d14 [mod] refactoring: processors
Report to the user suspended engines.

searx.search.processor.abstract:
* manages suspend time (per network).
* reports suspended time to the ResultContainer (method extend_container_if_suspended)
* adds the results to the ResultContainer (method extend_container)
* handles exceptions (method handle_exception)
2021-04-21 16:24:46 +02:00
Alexandre Flament
ae5954f2da Merge remote-tracking branch 'return42/fix-preference-save' 2021-04-21 16:24:22 +02:00
Alexandre Flament
624d3ec6db Merge remote-tracking branch 'return42/fix-url-bar-suggestion' 2021-04-21 16:23:52 +02:00
Alexandre Flament
92db0227b1 Merge remote-tracking branch 'dalf/oscar-images' 2021-04-19 09:36:54 +02:00
Alexandre Flament
48720e20a8 Merge remote-tracking branch 'searx/master' 2021-04-19 09:35:12 +02:00
Noémi Ványi
8362257b9a
Merge pull request #2736 from plague-doctor/sjp
Add new engine: SJP - Słownik języka polskiego
2021-04-16 17:30:14 +02:00
Noémi Ványi
e56323d3c8
Merge pull request #2759 from ypid/fix/typo
Fix grammar mistake in debug log output
2021-04-16 17:26:45 +02:00
Plague Doctor
d275d7a35e Code refactoring. 2021-04-16 12:23:27 +10:00
Markus Heiser
f637bfc635 [mod] oscar's "default" template should make use of result.thumbnail
Some engine do have set result.img_src, other return a result.thumbnail.  If
result.img_src is unset and a result.thumbnail is given, show it to the UI.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-15 08:52:11 +02:00
Markus Heiser
062d589f86 [fix] xpath expressions to grap all items from bandcamp's response
I also found some items missing a thumbnail and I used text_extract for content
and title, to remove unneeded whitespaces.

BTW: added bandcamp's favicon

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-15 08:52:11 +02:00
Kyle Anthony Williams
4d3c399ee9 [feat] add bandcamp engine 2021-04-15 08:52:11 +02:00
Alexandre Flament
d14994dc73 [httpx] replace searx.poolrequests by searx.network
settings.yml:

* outgoing.networks:
   * can contains network definition
   * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
     keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
   * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
   * local_addresses can be "192.168.0.1/24" (it supports IPv6)
   * support_ipv4 & support_ipv6: both True by default
     see https://github.com/searx/searx/pull/1034
* each engine can define a "network" section:
   * either a full network description
   * either reference an existing network

* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
2021-04-12 17:25:56 +02:00
Markus Heiser
c4793afadc [fix] https-scheme missing in preferences-page
This patch is an addition to PR #2656 which removed all usage of `base_url` from
the templates, except one was forgotten in the cookie URL of the preferences.

closes: 2740

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12 16:02:16 +02:00
Markus Heiser
2bf297b19f [fix] redirect when saving preferences
Erroneously commit 87e4c4762 droped the 302 redirect.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12 15:29:08 +02:00
Robin Schneider
dfc66ff0f0
Fix grammar mistake in debug log output 2021-04-11 22:12:53 +02:00
Alexandre Flament
eaa694fb7d [enh] replace requests by httpx 2021-04-10 15:38:33 +02:00
Plague Doctor
599ff39ddf Fix conflicts 2021-04-09 06:54:03 +10:00
Noémi Ványi
cc359345a8
Merge pull request #2735 from plague-doctor/wordnik
Add new engine: Wordnik.com
2021-04-08 19:48:13 +02:00
Plague Doctor
6631f11305 Add new engine: SJP 2021-04-08 10:21:54 +10:00
Plague Doctor
7035bed4ee Add new engine: Wordnik.com 2021-04-08 09:58:00 +10:00
Alexandre Flament
c09ff4faf2 [fix] fix PR 2656
SCRIPT_NAME remove trailing slash to avoid infinite redirect
2021-04-07 13:05:55 +02:00
Noémi Ványi
07f5edce3d Add Meilisearch engine
Website: https://www.meilisearch.com/
2021-04-06 21:57:05 +02:00
Noémi Ványi
a477a3a687
Merge pull request #2728 from return42/fix-loader
[fix] settings_loader.py - use update_dict only for mapping types
2021-04-05 21:32:41 +02:00
3nprob
2ca0aa4f29 Remove hubsbpot tracking URL params
More information here:
https://knowledge.hubspot.com/settings/how-do-i-create-a-tracking-url
https://knowledge.hubspot.com/ads/ad-tracking-in-hubspot
https://knowledge.hubspot.com/ctas/calls-to-action-frequently-asked-questions
https://meta.stackexchange.com/questions/263392/what-are-these-very-long-parameters-in-so-careers-feedback-always-welcome-url
2021-04-06 02:11:09 +09:00
Markus Heiser
9c10b15096 [fix] settings_loader.py - use update_dict only for mapping types
I can't set `default_doi_resolver` in `settings.yml` if I'm using
`use_default_settings`.  Searx seems to try to interpret all settings at root
level in `settings.yml` as dict, which is correct except for
`default_doi_resolver` which is at root level and a string::

    File "/usr/lib/python3.9/site-packages/searx/settings_loader.py", line 125, in load_settings
        update_settings(default_settings, user_settings)
    File "/usr/lib/python3.9/site-packages/searx/settings_loader.py", line 61, in update_settings
        update_dict(default_settings[k], v)
    File "/usr/lib/python3.9/site-packages/searx/settings_loader.py", line 48, in update_dict
        for k, v in user_dict.items():
    AttributeError: 'str' object has no attribute 'items'

Signed-off-by: Markus Heiser <markus@darmarit.de>
Suggested-by:  @0xhtml https://github.com/searx/searx/issues/2722#issuecomment-813391659
2021-04-05 16:33:48 +02:00
Alexandre Flament
7089526723
Merge pull request #2656 from return42/fix-url_for
[fix] url_for(..., _external=True) in templates
2021-04-05 14:50:39 +02:00
Markus Heiser
87e4c47621 [fix] url_for(..., _external=True) in templates
The `url_for` function in the template context is not the one from Flask, it is
the one from `webapp`.  The `webapp.url_for_theme` is different from its
namesake of Flask and has it quirks, when called with argument `_external=True`.

The `webapp.url_for_theme` can't handle absolute URLs since it pokes a leading
'/', here is the snippet of the old code::

    url = url_for(endpoint, **values)
    if settings['server']['base_url']:
        if url.startswith('/'):
            url = url[1:]
        url = urljoin(settings['server']['base_url'], url)

Next drawback of (Flask's) `_external=True` is, that it will not return the HTTP
scheme when searx (the Flask app) listens on http and is proxied by a https
server.

To get the right scheme `HTTP_X_SCHEME` is needed by Flask (werkzeug).  Since
this is not provided in every environment (e.g. behind Apache mod_wsgi or the
HTTP header is not fully set for some other reasons) it is recommended to
get *script_name*, *server* and *scheme* from the configured `base_url`.  If
`base_url` is specified, then these values from are given preference over any
Flask's generics.

BTW this patch normalize to use `url_for` in the `opensearch.xml` and drop the
need of `host` and `urljoin` in template's context.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-05 14:34:45 +02:00
Alexandre Flament
7a0fbdecc4 [enh] oscar: image thumbnail layout
Adjust thumbnail sizes to fill the container width
2021-04-05 11:29:17 +02:00
Noémi Ványi
ba90c5a2e0
Merge pull request #2718 from return42/fix-publishedDate
[fix] publishedDate: don't try to get date from empty string or None
2021-04-04 23:00:22 +02:00
Markus Heiser
ebfd0eb2b7 [fix] default_doi_resolver in preferences
Instead of a hard-coded `oadoi.org` default, use the default value from
`settings.yml`.

Fix an issue in the themes: The replacement 'current_doi_resolver' contains the
doi_resolver_url, not the name of the DOI resolver.  Compare return value of::

    searx.plugins.oa_doi_rewrite.get_doi_resolver(...)

Fix a typo in `get_doi_resolver(..)`:  suggested by @kvch:

  *L32 should set doi_resolver not doi_resolvers*

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-04 13:36:33 +02:00
Markus Heiser
c12826c6d5 [fix] publishedDate: don't try to get date from empty string or None
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-04 13:16:38 +02:00
Noémi Ványi
7612e5d9fd
Merge pull request #2712 from searx/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2021-04-03 23:15:43 +02:00
Noémi Ványi
985e8b28fe
Merge pull request #2710 from searx/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-04-03 23:02:44 +02:00
Noémi Ványi
feb2d81874
Merge pull request #2709 from searx/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-04-03 23:02:26 +02:00
Noémi Ványi
9a962c5369
Merge pull request #2711 from searx/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-04-03 23:01:43 +02:00
Noémi Ványi
5ca34ac571
Merge pull request #2713 from searx/update_data_update_languages.py
Update searx.data - update_languages.py
2021-04-03 23:01:00 +02:00
Markus Heiser
169438137f [fix] url bar autocomplete (opensearch suggestions)
Since #2593 is merged the OpenSearch-Format is buggy.  The loop in [1] will
change raw_text_query object and this will change also the value of
`raw_text_query.query` on every `raw_text_query.changeQuery(result)`.

This patch fixes this issue by storing the initial query value in `sug_prefix`.

[1] ac0fdc3b96/searx/webapp.py (L804-L806)

OpenSearch-Format::

    [ "<query>",
      [ "<term 1>", "<term 2>", ... "<term n>" ],
      [ "<content 1>", "<content 2>", ..., "<content n>" ],
      [ "<url 1>", "<url 2>", ..., "<url n>" ]
    ]

- https://www.google.com/support/enterprise/static/gsa/docs/admin/current/gsa_doc_set/xml_reference/query_suggestion.html#1080002
- https://developer.mozilla.org/en-US/docs/Archive/Add-ons/Supporting_search_suggestions_in_search_plugins#implementing_search_suggestion_support_on_the_server

Legacy-Format::

    [ "<term 1>", "<term 2>", ..., "<term n>" ]

- https://www.google.com/support/enterprise/static/gsa/docs/admin/current/gsa_doc_set/xml_reference/query_suggestion.html#1081079

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-03 18:18:50 +02:00
dalf
c0668d248e Update searx.data - update_languages.py 2021-04-01 07:00:46 +00:00
dalf
ad74c42aa1 Update searx.data - update_firefox_version.py 2021-04-01 07:00:39 +00:00
dalf
23dc7ef6d1 Update searx.data - update_wikidata_units.py 2021-04-01 07:00:37 +00:00
dalf
eb5cd7a543 Update searx.data - update_ahmia_blacklist.py 2021-04-01 07:00:33 +00:00
dalf
f804f54ca3 Update searx.data - update_currencies.py 2021-04-01 07:00:29 +00:00
Marc Abonce Seguin
419b907a0b fix dark "expand" button from infobox 2021-03-28 21:54:37 -07:00
Adam Tauber
ae122ea943 [enh] release v1.0.0 2021-03-27 20:30:08 +01:00
Adam Tauber
15f2385a7b [enh] update translations from transifex 2021-03-27 19:10:54 +01:00
Alexandre Flament
bb316d2698
Merge pull request #2682 from dalf/fix-checker
[fix] checker: various bug fixes
2021-03-27 17:44:29 +01:00
Alexandre Flament
725a69616b
Merge pull request #2681 from dalf/fix-wikipedia-title
[fix] wikipedia: remove HTML from the title
2021-03-27 17:43:36 +01:00
Noémi Ványi
9bb312c505 Remove duplicated key from dict in Semantic Scholar 2021-03-27 16:58:32 +01:00
Noémi Ványi
f596f5767b fix Semantic Scholar engine 2021-03-27 16:54:01 +01:00
Adam Tauber
28286cf3f2 [fix] update seznam engine to be compatible with the new website 2021-03-27 15:29:04 +01:00
Alexandre Flament
0b45afd4d7 [fix] checker: various bug fixes
* initialize engine_data (youtube engine)
* don't crash if an engine don't set result['url']
2021-03-25 09:37:37 +01:00
Alexandre Flament
fcfcf662ff [fix] wikipedia: remove HTML from the title
fr.wikipedia.org (and it seems not other wikipedia websites),
adds HTML to api_result['displayTitle'].
(Search for '!wp :fr Braid' for example)

The commit uses api_result['title']
2021-03-25 08:31:39 +01:00
Adam Tauber
06b754ad67 [mod] increase lobste.rs engine timeout to avoid timeouts most of the time 2021-03-25 01:22:36 +01:00
Adam Tauber
0ba71c3644 [fix] make ina engine compatible with the new response json 2021-03-25 01:20:41 +01:00
Adam Tauber
6255b33c9d [fix] rewrite hoogle to use html/xpath instead of json
the json response has been changed and it contains html chunks which is
not compatible with our json engine, so we have to switch to html/xpath
parsing
2021-03-25 01:13:24 +01:00
Adam Tauber
45f0e1a859 [fix] update geektimes.ru url - it redirects to habr.com 2021-03-25 01:02:19 +01:00
Adam Tauber
50ba2b9e87 [fix] update google play movies xpath 2021-03-25 00:55:53 +01:00
Adam Tauber
88657fe9c2 [fix] update google play apps xpath 2021-03-25 00:55:43 +01:00
Adam Tauber
5f450fda74 [enh] add year filter to duckduckgo 2021-03-25 00:25:36 +01:00
Adam Tauber
fd737dc9d8 [fix] remove debug code 2021-03-24 23:54:39 +01:00
Alexandre Flament
d648001688 [mod] preferences: a tooltip is shown when the mouse is over the engine names 2021-03-22 08:22:59 +01:00
Alexandre Flament
6bd01bf81f [mod] oscar: fix the sourcemap URL in *.min.css
Close https://github.com/searx/searx/issues/2670

Note: clean-css contains a bug:
* a multiline comment or URL adds "$stdin" to the sourcemap (see src/less/logicodev/search.less)
* in this case when the user opens the devtools, the browser fails to load this "https://.../$stdin" URL
* it is not a problem and the error appears only when the user actively tries to debug the CSS.
* seems related to https://github.com/jakubpawlowicz/clean-css/issues/593
2021-03-21 18:03:40 +01:00
Alexandre Flament
a48ec0b4bd
Merge pull request #2671 from searx/update-soundcloud
[mod] soundcloud: faster initialization
2021-03-21 15:10:39 +01:00
Alexandre Flament
30c950a2c7
Merge pull request #2660 from dalf/upd-translations
[mod] replace /translations.js with an embedded JSON
2021-03-21 12:39:26 +01:00
Alexandre Flament
38c210d746
[mod] soundcloud: faster initialization
The get_cliend_id() function:
* fetches https://soundcloud.com
* then fetches each referenced javascript URL to get the client id.

This commit fetches the javascript URLs in the reverse order: the client id is in the last javascript URL.
2021-03-21 09:29:53 +01:00
James Higginbotham
ce6eb81a71
Update settings.yml to enable HTTP for yacy
Added a line to the yacy entry to enable HTTP if the local yacy instance isn't using HTTPS. Otherwise, an error will be thrown in the logs: "No connection adapters were found for 'http://localhost:8090/yacysearch.json...'". This is likely related to ticket #2641 that forces HTTPS by default.
2021-03-19 15:06:25 -06:00
Alexandre Flament
2b0dd96bd3 [mod] oscar: remove space
* reduce by 15% the uncompressed output (on average)
* dos2unix searx/templates/oscar/result_templates/files.html
2021-03-17 09:22:05 +01:00
Dr. Rolf Jansen
7a9dc63d74
Merge branch 'master' into conditional-sigusr1 2021-03-16 08:45:57 -03:00
Alexandre Flament
6553c79029 [mod] replace /translations.js by embedded JSON
In webapp.py, there is a new function "get_translations" lists available translations

Close #2064
2021-03-16 11:22:21 +01:00
Alexandre Flament
32cd0d31b3 [mod] upgrade pygments
add searx_extra/update/update_pygments.py to update the css style of the oscar and simple themes.
2021-03-16 09:07:08 +01:00