Commit graph

6531 commits

Author SHA1 Message Date
Markus Heiser
de1e7d12f7 [fix] get_engine_locale: better approximation of 'en' is 'en-US'
Compared to `en-EN` the better approximation of 'en' is 'en-US'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 15:45:07 +02:00
Markus Heiser
ac7776663b [fix] typo in get_engine_locale
Due to a typo in get_engine_locale, a language selection like `!qw :de siemens`
did not work.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 14:35:09 +02:00
Markus Heiser
649d1a144d
Merge pull request #1664 from return42/harden-get_engine_locales
[fix] harden get_engine_locale: handle UnknownLocaleError exceptions
2022-08-14 14:25:14 +02:00
Markus Heiser
ef81d14ccf [fix] harden get_engine_locale: handle UnknownLocaleError exceptions
When a user selects an unknown or invalid locale by using the search syntax:

    !qw siemens :de-TW

Before this patch a UnknownLocaleError exception will be rasied:

```
Traceback (most recent call last):
  File "SearXNG/searx/search/processors/online.py", line 154, in search
    search_results = self._search_basic(query, params)
  File "SearXNG/searx/search/processors/online.py", line 128, in _search_basic
    self.engine.request(query, params)
  File "SearXNG/searx/engines/qwant.py", line 98, in request
    q_locale = get_engine_locale(params['language'], supported_languages, default='en_US')
  File "SearXNG/searx/locales.py", line 216, in get_engine_locale
    locale = babel.Locale.parse(searxng_locale, sep='-')
  File "SearXNG/local/py3/lib/python3.8/site-packages/babel/core.py", line 330, in parse
    raise UnknownLocaleError(input_id)
```

This patch implements a simple exception handling, since e.g. `de-TW` does not
exists `de` will be used to get engines locale.  On invalid terms like `xy-XY`
the default will be returned.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 13:55:42 +02:00
Markus Heiser
2bd3c2079e
Merge pull request #1652 from return42/mod-qwant
Improve qwant engine
2022-08-14 10:58:52 +02:00
Markus Heiser
27385e7898 [mod] qwant - add safesearch option
Closes: https://github.com/searxng/searxng/issues/1640
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:36:14 +02:00
Markus Heiser
6579d6d558 [fix] qwant - API error::locale must be one ..
The request function should not request a language (aka locale) that is not
supported by qwant. Select a locale like zh-TW ends in qwant's API error:

  ERROR searx.engines.qwant news: exception : \
  API error::locale must be one of the following values: \
    en_gb, en_ie, en_us, en_ca, en_my, en_au, en_nz, de_de, de_ch, de_at, fr_fr, \
    fr_be, fr_ch, fr_ca, fr_ad, fc_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx, \
    es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_pt, pt_ad, \
    nl_be, nl_nl

The existing searx.utils.match_language function is unsuitable for this purpose,
it is replaced by function searx.locales.get_engine_locale that is based on the
methods from the babel package.

The quant's _fetch_supported_languages function has been revised to filter out
languages 8aka locales) not supported by qwant.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:36:14 +02:00
Markus Heiser
9ae409a05a [mod] add locale.get_engine_locale to get predictable results
The match_language function sometimes returns incorrect results which is why a
new function get_engine_locale is required.

A bugfix of the match_language is not easily possible, because there is almost
no documentation for it and already the call parameters are undefined.  E.g. the
function processes values like the ones from yahoo::

    "yahoo": [
        "ar",
        ...
        "zh_chs",
        "zh_cht"
     ]

The get_engine_locale has been documented in detail, there is a clear
description of the assumptions as well as the requirements and approximation
rules (read doc-string for more details)::

    Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
    corresponding *engine locales*:

      <engine>: {
          # SearXNG string : engine-string
          'ca-ES'          : 'ca_ES',
          'fr-BE'          : 'fr_BE',
          'fr-CA'          : 'fr_CA',
          'fr-CH'          : 'fr_CH',
          'fr'             : 'fr_FR',
          ...
          'pl-PL'          : 'pl_PL',
          'pt-PT'          : 'pt_PT'
      }

    .. hint::

       The *SearXNG locale* string has to be known by babel!

In the following you will find a comparison:

>>> import babel.languages
>>> from searx.utils import match_language
>>> from searx.locales import get_engine_locale

Assume we have an engine that supports the follwoing locales:

>>> lang_list = {
...     "zh-CN": "zh_CN",
...     "zh-HK": "zh_HK",
...     "nl-BE": "nl_BE",
...     "fr-CA": "fr_CA",
... }

Assumption:

  A. When a user selects a language the results should be optimized according to
     the selected language.

  B. When user selects a language and a territory the results should be
     optimized with first priority on territory and second on language.

----

Example: (Assumption A.)

  A user selects region 'zh-TW' which should end in zh_HK

hint:
  CN is 'Hans' and HK ('Hant') fits better to TW ('Hant')

>>> get_engine_locale('zh-TW', lang_list)
'zh_HK'
>>> lang_list[match_language('zh-TW', lang_list)]
'zh_CN'

----

Example: (Assumption A.)

  A user selects only the language 'zh' which should end in CN

>>> get_engine_locale('zh', lang_list)
'zh_CN'
>>> lang_list[match_language('zh', lang_list)]
'zh_CN'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

hint:
  priority should be on the territory the user selected.  If the user
  prefers 'fr' he will select 'fr' without a region tag.

>>> get_engine_locale('fr-BE', lang_list, default='unknown')
'nl_BE'
>>> match_language('fr-BE', lang_list, fallback='unknown')
'fr-CA'

----

Example: (Assumption A.)

  A user selects only the language 'fr' which should end in fr_CA

>>> get_engine_locale('fr', lang_list)
'fr_CA'
>>> lang_list[match_language('fr', lang_list)]
'fr_CA'

----

The difference in priority on the territory is best shown with a engine that
supports the following locales:

>>> lang_list = {
...     "fr-FR": "fr_FR",
...     "fr-CA": "fr_CA",
...     "en-GB": "en_GB",
...     "nl-BE": "nl_BE",
... }

----

Example: (Assumption A.)

   A user selects only a language

>>> get_engine_locale('en', lang_list)
'en_GB'
>>> match_language('en', lang_list)
'en-GB'

hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR
takes priority ..

>>> get_engine_locale('fr', lang_list)
'fr_FR'
>>> lang_list[match_language('fr', lang_list)]
'fr_FR'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

>>> get_engine_locale('fr-BE', lang_list)
'nl_BE'
>>> lang_list[match_language('fr-BE', lang_list)]
'fr_FR'

----

If the user selects a language and there are two locales like the following:

>>> lang_list = {
...      "fr-BE": "fr_BE",
...      "fr-CH": "fr_CH",
...  }
>>>

>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_BE'

Looks like both functions return the same value, but match_language depends on the
order of the dictionary (which is not predictable):

>>> lang_list = {
...      "fr-CH": "fr_CH",
...      "fr-BE": "fr_BE",
...  }
>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_CH'
>>>

The get_engine_locale selects the locale by looking at the "population percent"
and this percentage has an higher amount in BE (68.%) compared to CH (21%)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:35:55 +02:00
Markus Heiser
75bb8c45d0 [mod] decouple qwant's categories from SearXNG's categories
By using new property `qwant_categ:` the category of qwant is no longer bound to
the category of SearXNG.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:26:54 +02:00
Alexandre Flament
e81c458165
Merge pull request #1656 from searxng/dependabot/pip/master/flask-2.2.2
Bump flask from 2.1.3 to 2.2.2
2022-08-13 18:33:39 +02:00
Markus Heiser
0043190643
Merge pull request #1655 from return42/fix-opensearch
[fix] improve OpenSearch description
2022-08-13 12:36:34 +02:00
Léon Tiekötter
b56848e8d0
Merge pull request #1660 from liimee/eng-neeva
Add neeva engine
2022-08-13 02:53:27 +02:00
Léon Tiekötter
8fa84ee9f4 Fix time range support and add soft_max_redirects
Add custom time_range_url and time_range_map

Set soft_max_redirects = 2 to prevent "ErrorContext('searx/search/processors/online.py', 116, 'count_error(', None, '2 redirects, maximum: 0', ('200', 'OK', 'neeva.com')) True"
2022-08-13 07:35:15 +07:00
ta
d6dfada1a9 add neeva web engine
Neeva is "the world's first ad-free, private search engine" and uses data from Apple, Bing, Yelp and "others".
They claim to crawl "hundreds of millions" of URLs a day (https://twitter.com/Neeva/status/1536447373903335426).
2022-08-13 07:26:36 +07:00
Markus Heiser
561d24cd2c
Merge pull request #1657 from searxng/dependabot/pip/master/selenium-4.4.0
Bump selenium from 4.3.0 to 4.4.0
2022-08-12 18:54:50 +02:00
Markus Heiser
334eccfad6
Merge pull request #1658 from searxng/dependabot/pip/master/setproctitle-1.3.2
Bump setproctitle from 1.3.1 to 1.3.2
2022-08-12 18:54:19 +02:00
Markus Heiser
6481beb479
Merge pull request #1659 from searxng/translations_update
Update translations
2022-08-12 18:51:08 +02:00
searxng-bot
b4fea007f5 [translations] update from Weblate
f2997bfa - 2022-08-12 - Markus Heiser <markus.heiser@darmarit.de>
eeca674f - 2022-08-10 - Edrean Ernst <edrean@allesbeste.com>
7478de6a - 2022-08-11 - Markus Heiser <markus.heiser@darmarit.de>
c4fb9110 - 2022-08-07 - wordpure <wordlesspure@gmail.com>
a5b432e2 - 2022-08-11 - Markus Heiser <markus.heiser@darmarit.de>
eb01d415 - 2022-08-09 - Markus Heiser <markus.heiser@darmarit.de>
f96eb06e - 2022-08-11 - Shopimisrel <shopisrael12@gmail.com>
e7c79191 - 2022-08-08 - ajnasaboobacker <ajnasaboobacker@gmail.com>
f4dbd424 - 2022-08-08 - ajnasaboobacker <ajnasaboobacker@gmail.com>
2022-08-12 07:18:14 +00:00
dependabot[bot]
bf36057c79
Bump setproctitle from 1.3.1 to 1.3.2
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/commits)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-12 07:04:32 +00:00
dependabot[bot]
11dfff766b
Bump selenium from 4.3.0 to 4.4.0
Bumps [selenium](https://github.com/SeleniumHQ/Selenium) from 4.3.0 to 4.4.0.
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/compare/selenium-4.3.0...selenium-4.4.0)

---
updated-dependencies:
- dependency-name: selenium
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-12 07:04:30 +00:00
dependabot[bot]
828073357e
Bump flask from 2.1.3 to 2.2.2
Bumps [flask](https://github.com/pallets/flask) from 2.1.3 to 2.2.2.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/flask/compare/2.1.3...2.2.2)

---
updated-dependencies:
- dependency-name: flask
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-12 07:04:25 +00:00
Markus Heiser
3b0f9c07b2 [fix] improve OpenSearch description
Some HTTP-Clients do have issues with the ``opensearch.xml`` from SearXNG
(related [1][2]) while other OpenSearch descriptions[3] (e.g. from qwant) work
flawles.

Inspired by the OpenSearch description from qwant and with informations from the
specification[4] the ``opensearch.xml`` has been *improved*.

- convert `<Url>` methods from lower case to upper case (`POST`|`GET`)
- add `<moz:SearchForm>` and `xmlns:moz="http://www.mozilla.org/2006/browser/search/"`
- add `<Query role="example" searchTerms="SearXNG" />`  [4]

  OpenSearch description documents should include at least one Query element of
  `role="example"` that is expected to return search results. Search clients may
  use this example query to validate that the search engine is working properly.

- modified `<LongName>` to SearXNG
- modified `<Description>` the word 'hackable' scares uninitiated users and was removed
- add the `type="image/png"` to `<Image>`

Test can be done by::

    make run

Visit http://127.0.0.1:8888/ and add the search engine to your WEB-Browser /
test with different WEB-Browser from desktop and Smartphones (are there any iOS
user here, please test on Safari and Chrome).

[1] https://app.element.io/#/room/#searxng:matrix.org/$xN_abdKhNqUlgXRBrb_9F3pqOxnSzGQ1TG0s0G9hQVw
[2] https://github.com/searxng/searxng/issues/431
[3] https://developer.mozilla.org/en-US/docs/Web/OpenSearch
[4] https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#the-query-element

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-11 19:04:36 +02:00
Markus Heiser
2bfb269f0a
Merge pull request #1618 from return42/drop-pycodestyle
[fix] pycodestyle has been replaced by black in 3c77412d3
2022-08-10 21:58:16 +02:00
Markus Heiser
214f234c6f
Merge pull request #1649 from joxcat/eng-srht
Add sourcehut engine
2022-08-10 21:56:25 +02:00
Markus Heiser
73ffa0c070 Update searx.data - update_engine_descriptions.py
./manage pyenv.cmd python ./searxng_extra/update/update_engine_descriptions.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-10 21:48:34 +02:00
Johan Planchon
e0215e0cc8 add sourcehut engine 2022-08-10 21:48:12 +02:00
Markus Heiser
43c545910b
Merge pull request #1648 from unixfox/google_mobile_ui_parameter
output format protobuf to HTML for google mobile
2022-08-10 20:50:45 +02:00
Markus Heiser
eb02cc77c5 [fix] google - simplify XPath selectors to fetch more results
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-10 18:55:31 +02:00
Markus Heiser
5706d110f2
Merge pull request #1650 from joxcat/eng-librs
Add lib.rs engine
2022-08-10 17:45:28 +02:00
Johan Planchon
90bdb23756 add lib.rs engine 2022-08-10 12:17:55 +02:00
Émilien Devos
b9f16a77db output format protobuf to HTML for google mobile 2022-08-10 09:36:06 +00:00
Léon Tiekötter
07342899e5
Merge pull request #1643 from vojkovic/master
Revert Google XPath.
2022-08-10 04:18:42 +02:00
Brock Vojković
24210fb10b
Revert PR #1633
This reverts the changes made to the Google results XPath in PR #1633.
2022-08-10 03:41:39 +02:00
Markus Heiser
3d6b098a5f
Merge pull request #1633 from tiekoetter/fix-google-results-xpath
[fix] google engine: results XPath
2022-08-09 06:28:03 +02:00
Léon Tiekötter
94b3656b4a [fix] google engine: results XPath
Seems google rolls out changes first on the `google.com` domain and later on the
"language" domains.  By example: yesterday [1] `google.com` did not work but
`google.de` and `google.fr` did work, today they do not work any longer and this
fix is needed on all domains.

Closes: https://github.com/searxng/searxng/issues/1628
[1] https://github.com/searxng/searxng/issues/1628#issuecomment-1208191816
2022-08-09 06:23:59 +02:00
Markus Heiser
925f3ff487
Merge pull request #1627 from allendema/add-marginalia
[enh] Initial Marginalia.nu support (foss)
2022-08-08 06:53:02 +02:00
Markus Heiser
58f630d312
Merge pull request #1623 from return42/mod-yep.com
[mod] engine yep.com: show all 100 results yep.com has
2022-08-08 06:52:15 +02:00
Allen
5b815b3966 [enh] initial Marginalia.nu support (foss)
Currently it uses a public api_key `/public/` [1]

The 'index' parameter selects the search index, corresponding to the drop down
next to the search field in the main GUI.

    0: popular
    1: blogs
    2: big_sites
    3: default
    4: experimental

'experimental' is more up to date and does not exclude other sites, which is the
case with 'big sites' or 'blogs'.

[1] https://api.marginalia.nu/
[2] https://git.marginalia.nu/marginalia/marginalia.nu
[3] https://news.ycombinator.com/item?id=31536626

Closes: https://github.com/searxng/searxng/issues/1620
2022-08-08 06:31:04 +02:00
Markus Heiser
04e3427022
Merge pull request #1626 from return42/fix-minor
[fix] typo in uWSGI cache: searxcache --> searxngcache
2022-08-07 12:53:33 +02:00
Markus Heiser
8b58cd1167 [fix] typo in uWSGI cache: searxcache --> searxngcache
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-07 12:31:09 +02:00
Markus Heiser
647e6187d7 [mod] engine yep.com: show all 100 results yep.com has
yep.com is still in beta, the api.yep.com does not have paging support.  There
is only a 'limit' argument with a maximum of 100 results.

yep.com seems fast; there is nor need for a timeout of 12 sec.

The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.

FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.

Closes: https://github.com/searxng/searxng/issues/1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-06 13:21:28 +02:00
Markus Heiser
63f5121b38 [fix] pycodestyle has been replaced by black in 3c77412d3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-05 10:29:58 +02:00
Markus Heiser
cee586029c
Merge pull request #1614 from searxng/dependabot/pip/master/setproctitle-1.3.1
Bump setproctitle from 1.3.0 to 1.3.1
2022-08-05 10:03:35 +02:00
Markus Heiser
3d0bf840b9
Merge pull request #1617 from searxng/translations_update
Update translations
2022-08-05 09:44:57 +02:00
Markus Heiser
051cf5bc61
Merge pull request #1615 from searxng/dependabot/pip/master/pycodestyle-2.9.1
Bump pycodestyle from 2.8.0 to 2.9.1
2022-08-05 09:33:05 +02:00
searxng-bot
0f02fcc836 [translations] update from Weblate
acf8bd39 - 2022-08-05 - Markus Heiser <markus.heiser@darmarit.de>
4ad75b6e - 2022-08-04 - Markus Heiser <markus.heiser@darmarit.de>
ee8cbee6 - 2022-07-31 - Markus Heiser <markus.heiser@darmarit.de>
87c19313 - 2022-08-01 - Academic tyro <y13593582403@gmail.com>
cbe0de32 - 2022-07-30 - Markus Heiser <markus.heiser@darmarit.de>
45029a17 - 2022-08-04 - Markus Heiser <markus.heiser@darmarit.de>
6eec3795 - 2022-08-03 - Markus Heiser <markus.heiser@darmarit.de>
f8d8f31f - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
b3fb365f - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
aaeabbc9 - 2022-08-03 - Lakatos Tamás <tomimost@gmail.com>
6c71c501 - 2022-08-03 - Markus Heiser <markus.heiser@darmarit.de>
f7b5ba19 - 2022-08-01 - Markus Heiser <markus.heiser@darmarit.de>
850e7fa0 - 2022-08-04 - Mico Hautaluoma <m@mha.fi>
0cb696fc - 2022-07-31 - Markus Heiser <markus.heiser@darmarit.de>
04c3785f - 2022-08-02 - Markus Heiser <markus.heiser@darmarit.de>
b500f2ad - 2022-08-01 - Edrean Ernst <edrean@allesbeste.com>
0b576b83 - 2022-08-01 - GooGuJiang <gu@gmoe.cc>
0adeb6e2 - 2022-08-01 - Edrean Ernst <edrean@allesbeste.com>
0b025f17 - 2022-07-31 - PRATYAY MUSTAFI <pratyaymustafi@gmail.com>
2022-08-05 07:18:24 +00:00
dependabot[bot]
27cdbeb9c4
Bump pycodestyle from 2.8.0 to 2.9.1
Bumps [pycodestyle](https://github.com/PyCQA/pycodestyle) from 2.8.0 to 2.9.1.
- [Release notes](https://github.com/PyCQA/pycodestyle/releases)
- [Changelog](https://github.com/PyCQA/pycodestyle/blob/main/CHANGES.txt)
- [Commits](https://github.com/PyCQA/pycodestyle/compare/2.8.0...2.9.1)

---
updated-dependencies:
- dependency-name: pycodestyle
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-05 07:06:16 +00:00
dependabot[bot]
c6b5bc5c69
Bump setproctitle from 1.3.0 to 1.3.1
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.3.0...version-1.3.1)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-05 07:06:12 +00:00
Markus Heiser
225f8caaf9
Merge pull request #1608 from allendema/brave-time_range
[enh] time range support for brave
2022-08-04 06:48:50 +02:00
Allen
58369e41d6 [wip] brave time range support 2022-08-04 04:47:25 +02:00