Commit graph

21 commits

Author SHA1 Message Date
Markus Heiser
9b575a997b [fix] doc of locales.get_engine_locale() / zh-classical is missleading
Wikipedia's zh-classical is not zh_Hant (see doc-string of engines.wikipedia).
Fixed the example in the doc-string of locales.get_engine_locale() to 'zh_TW'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-17 08:48:57 +02:00
Markus Heiser
16f0db4493 [mod] replace utils.match_language by locales.match_locale
This patch replaces the *full of magic* ``utils.match_language`` function by a
``locales.match_locale``.  The ``locales.match_locale`` function is based on the
``locales.build_engine_locales`` introduced in 9ae409a0 [1].

In the past SearXNG did only support a search by a language but not in a region.
This has been changed a long time ago and regions have been added to SearXNG
core but not to the engines.  The ``utils.match_language`` was the function to
handle the different aspects of language/regions in SearXNG core and the
supported *languages* in the engine.  The ``utils.match_language`` did it with
some magic and works good for most use cases but fails in some edge case.

To replace the concurrence of languages and regions in the SearXNG core the
``locales.build_engine_locales`` was introduced in 9ae409a0 [1].  With the last
patches all engines has been migrated to a ``fetch_traits`` and a
language/region concept that is based on ``locales.build_engine_locales``.

To summarize: there is no longer a need for the ``locales.match_language``.

[1] https://github.com/searxng/searxng/pull/1652

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
6e5f22e558 [mod] replace engines_languages.json by engines_traits.json
Implementations of the *traits* of the engines.

Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*.  Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.

To load traits from the persistence::

    searx.enginelib.traits.EngineTraitsMap.from_data()

For new traits new properties can be added to the class::

    searx.enginelib.traits.EngineTraits

.. hint::

   Implementation is downward compatible to the deprecated *supported_languages
   method* from the vintage implementation.

   The vintage code is tagged as *deprecated* an can be removed when all engines
   has been ported to the *traits method*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
dependabot[bot]
70fff2ebf1 Bump flask-babel from 2.0.0 to 3.0.0
Bumps [flask-babel](https://github.com/python-babel/flask-babel) from 2.0.0 to 3.0.0.
- [Release notes](https://github.com/python-babel/flask-babel/releases)
- [Changelog](https://github.com/python-babel/flask-babel/blob/master/CHANGELOG)
- [Commits](https://github.com/python-babel/flask-babel/compare/v2.0.0...v3.0.0)

---
updated-dependencies:
- dependency-name: flask-babel
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-20 09:38:30 +00:00
Alexandre FLAMENT
d8dee21277 searx.locale: add Dhivehi language 2022-11-05 21:26:15 +00:00
Alexandre FLAMENT
20d951fc5c searx.locales: improve support for languages not supported by babel
* refactor get_translations() to rely on ADDITIONAL_TRANSLATIONS and LOCALE_BEST_MATCH
* update RTL_LOCALES for languages in ADDITIONAL_TRANSLATIONS
2022-11-05 21:26:07 +00:00
Markus Heiser
fbf07237ff [fix] and improve docs generated from source code.
Fix::

    searx/locales.py:docstring of searx.locales.get_engine_locale:17: \
      WARNING: Definition list ends without a blank line; unexpected unindent.

Improvement: don't show default values in the generated documentation whe it is
more a mess than a usefull information (`:meta hide-value:`).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-18 12:44:12 +02:00
Markus Heiser
de1e7d12f7 [fix] get_engine_locale: better approximation of 'en' is 'en-US'
Compared to `en-EN` the better approximation of 'en' is 'en-US'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 15:45:07 +02:00
Markus Heiser
ac7776663b [fix] typo in get_engine_locale
Due to a typo in get_engine_locale, a language selection like `!qw :de siemens`
did not work.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 14:35:09 +02:00
Markus Heiser
ef81d14ccf [fix] harden get_engine_locale: handle UnknownLocaleError exceptions
When a user selects an unknown or invalid locale by using the search syntax:

    !qw siemens :de-TW

Before this patch a UnknownLocaleError exception will be rasied:

```
Traceback (most recent call last):
  File "SearXNG/searx/search/processors/online.py", line 154, in search
    search_results = self._search_basic(query, params)
  File "SearXNG/searx/search/processors/online.py", line 128, in _search_basic
    self.engine.request(query, params)
  File "SearXNG/searx/engines/qwant.py", line 98, in request
    q_locale = get_engine_locale(params['language'], supported_languages, default='en_US')
  File "SearXNG/searx/locales.py", line 216, in get_engine_locale
    locale = babel.Locale.parse(searxng_locale, sep='-')
  File "SearXNG/local/py3/lib/python3.8/site-packages/babel/core.py", line 330, in parse
    raise UnknownLocaleError(input_id)
```

This patch implements a simple exception handling, since e.g. `de-TW` does not
exists `de` will be used to get engines locale.  On invalid terms like `xy-XY`
the default will be returned.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 13:55:42 +02:00
Markus Heiser
9ae409a05a [mod] add locale.get_engine_locale to get predictable results
The match_language function sometimes returns incorrect results which is why a
new function get_engine_locale is required.

A bugfix of the match_language is not easily possible, because there is almost
no documentation for it and already the call parameters are undefined.  E.g. the
function processes values like the ones from yahoo::

    "yahoo": [
        "ar",
        ...
        "zh_chs",
        "zh_cht"
     ]

The get_engine_locale has been documented in detail, there is a clear
description of the assumptions as well as the requirements and approximation
rules (read doc-string for more details)::

    Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
    corresponding *engine locales*:

      <engine>: {
          # SearXNG string : engine-string
          'ca-ES'          : 'ca_ES',
          'fr-BE'          : 'fr_BE',
          'fr-CA'          : 'fr_CA',
          'fr-CH'          : 'fr_CH',
          'fr'             : 'fr_FR',
          ...
          'pl-PL'          : 'pl_PL',
          'pt-PT'          : 'pt_PT'
      }

    .. hint::

       The *SearXNG locale* string has to be known by babel!

In the following you will find a comparison:

>>> import babel.languages
>>> from searx.utils import match_language
>>> from searx.locales import get_engine_locale

Assume we have an engine that supports the follwoing locales:

>>> lang_list = {
...     "zh-CN": "zh_CN",
...     "zh-HK": "zh_HK",
...     "nl-BE": "nl_BE",
...     "fr-CA": "fr_CA",
... }

Assumption:

  A. When a user selects a language the results should be optimized according to
     the selected language.

  B. When user selects a language and a territory the results should be
     optimized with first priority on territory and second on language.

----

Example: (Assumption A.)

  A user selects region 'zh-TW' which should end in zh_HK

hint:
  CN is 'Hans' and HK ('Hant') fits better to TW ('Hant')

>>> get_engine_locale('zh-TW', lang_list)
'zh_HK'
>>> lang_list[match_language('zh-TW', lang_list)]
'zh_CN'

----

Example: (Assumption A.)

  A user selects only the language 'zh' which should end in CN

>>> get_engine_locale('zh', lang_list)
'zh_CN'
>>> lang_list[match_language('zh', lang_list)]
'zh_CN'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

hint:
  priority should be on the territory the user selected.  If the user
  prefers 'fr' he will select 'fr' without a region tag.

>>> get_engine_locale('fr-BE', lang_list, default='unknown')
'nl_BE'
>>> match_language('fr-BE', lang_list, fallback='unknown')
'fr-CA'

----

Example: (Assumption A.)

  A user selects only the language 'fr' which should end in fr_CA

>>> get_engine_locale('fr', lang_list)
'fr_CA'
>>> lang_list[match_language('fr', lang_list)]
'fr_CA'

----

The difference in priority on the territory is best shown with a engine that
supports the following locales:

>>> lang_list = {
...     "fr-FR": "fr_FR",
...     "fr-CA": "fr_CA",
...     "en-GB": "en_GB",
...     "nl-BE": "nl_BE",
... }

----

Example: (Assumption A.)

   A user selects only a language

>>> get_engine_locale('en', lang_list)
'en_GB'
>>> match_language('en', lang_list)
'en-GB'

hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR
takes priority ..

>>> get_engine_locale('fr', lang_list)
'fr_FR'
>>> lang_list[match_language('fr', lang_list)]
'fr_FR'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

>>> get_engine_locale('fr-BE', lang_list)
'nl_BE'
>>> lang_list[match_language('fr-BE', lang_list)]
'fr_FR'

----

If the user selects a language and there are two locales like the following:

>>> lang_list = {
...      "fr-BE": "fr_BE",
...      "fr-CH": "fr_CH",
...  }
>>>

>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_BE'

Looks like both functions return the same value, but match_language depends on the
order of the dictionary (which is not predictable):

>>> lang_list = {
...      "fr-CH": "fr_CH",
...      "fr-BE": "fr_BE",
...  }
>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_CH'
>>>

The get_engine_locale selects the locale by looking at the "population percent"
and this percentage has an higher amount in BE (68.%) compared to CH (21%)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:35:55 +02:00
Alexandre Flament
f90aed7ef9 locales.py: add support for Papiamento 2022-07-08 10:00:20 +02:00
Markus Heiser
ef4239c68a [doc] fix some leftovers from ad964562c
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-14 16:31:41 +02:00
Markus Heiser
ad964562ce [fix] move locale code from webapp.py to locales.py and fix #1303
To improve modularization this patch:

- moves *locale* related implementation from the webapp.py application to the
  locale.py module.

- The initialization of the locales is now done in the application (webapp) and
  is no longer done while importing searx.locales.

In the searx.locales module a new dictionary named `LOCALE_BEST_MATCH` has been
added.  In this dictionary we can map languages without a translation to
languages we have a translation for.

To fix #1303 zh-HK has been mapped to zh-Hant-TW (we do not need additional
translations of traditional Chinese)

Closes: https://github.com/searxng/searxng/issues/1303
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-12 10:52:26 +02:00
Alexandre FLAMENT
550f70b1aa Add support for the Silesian language 2022-05-06 09:40:45 +00:00
Markus Heiser
3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:26:22 +01:00
Alexandre Flament
a1d1aec6e2 [mod] locale: use hyphen everywhere except for Babel 2021-10-12 21:06:20 +02:00
Markus Heiser
443bf35e09 [pylint] fix global-variable-not-assigned issues
If there is no write access, there is no need for global.  Remove global
statement if there is no assignment.

global-variable-not-assigned:
  Using global for names but no assignment is done Used when a variable is
  defined through the "global" statement but no assignment to this variable is
  done.

In Pylint 2.11 the global-variable-not-assigned checker now catches global
variables that are never reassigned in a local scope and catches (reassigned)
functions [1][2]

[1] https://pylint.pycqa.org/en/latest/whatsnew/2.11.html
[2] https://github.com/PyCQA/pylint/issues/1375

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17 10:14:27 +02:00
Alexandre Flament
0d20e5dfe3 [mod] searx/locales.py: language names based on Unicode CLDR
rename "oc" to "Occitan":
* https://github.com/unicode-org/cldr/blob/35.1/seed/main/oc.xml#L115
* https://oc.wikipedia.org/wiki/Occitan

see https://github.com/searxng/searxng/pull/247#issuecomment-892382001
2021-08-04 09:50:34 +02:00
Markus Heiser
809bf1a105 [mod] pylint & document searx.locales (settings.yml: remove locales)
- Add ``# lint: pylint`` header to pylint this python file.
- Fix issues reported by pylint.
- Add source code documentation of modul searx.locales

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-04 07:54:53 +02:00
Alexandre Flament
f30d01ffab [mod] settings.yml: remove locales
There are detected from the searx/translations directory
2021-08-03 15:44:45 +02:00