Commit graph

40 commits

Author SHA1 Message Date
Bnyro
5ec7df3480 [mod] engine duckduckgo definitions: support for answer source 2023-08-18 19:07:14 +02:00
Markus Heiser
c80e82a855 [mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the DuckDuckGo (DDG) engines including a
improved language and region handling based on the enigne.traits_v1 data.

- DDG Lite
- DDG Instant Answer API
- DDG Images
- DDG Weather

docs/src/searx.engine.duckduckgo.rst:
  Online documentation of the DDG engines (make docs.live)

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - "duckduckgo" (WEB),
  - "duckduckgo images" and
  - "duckduckgo weather"

  and remove data from obsolete data type "supported_languages".

searx/autocomplete.py:
  Reversed engineered Autocomplete from DDG.  Supports DDG's languages.

searx/engines/duckduckgo.py:
  - fetch_traits():  Fetch languages & regions from DDG.

  - get_ddg_lang(): Get DDG's language identifier from SearXNG's locale.  DDG
    defines its languages by region codes.  DDG-Lite does not offer a language
    selection to the user, only a region can be selected by the user.

  - Cache ``vqd`` value: The vqd value depends on the query string and is needed
    for the follow up pages or the images loaded by a XMLHttpRequest (DDG
    images).  The ``vqd`` value of a search term is stored for 10min in the
    redis DB.

  - DDG Lite engine: reversed engineered request method with improved Language
    and region support and better ``vqd`` handling.

searx/engines/duckduckgo_definitions.py: DDG Instant Answer API
  The *instant answers* API does not support languages, or at least we could not
  find out how language support should work.  It seems that most of the features
  are based on English terms.

searx/engines/duckduckgo_images.py: DDG Images
  Reversed engineered request method.  Improved language and region handling
  based on cookies and the enigne.traits_v1 data.  Response: add image format to
  the result list

searx/engines/duckduckgo_weather.py: DDG Weather
  Improved language and region handling based on cookies and the
  enigne.traits_v1 data.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
dba8977b09 [mod] DuckDuckGo: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the DuckDuckGo engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
Markus Heiser
8df1f0c47e [mod] add 'Accept-Language' HTTP header to online processores
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).

- add new engine option: send_accept_language_header

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 17:01:59 +02:00
Markus Heiser
d84226bf63 [fix] issues reported by pylint
Fix pylint issues from commit (3d96a983)

    [format.python] initial formatting of the python code

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 10:16:20 +01:00
Markus Heiser
3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:26:22 +01:00
Markus Heiser
f0059b80ed [pylint] engines: drop no longer needed 'missing-function-docstring'
Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914168470
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 13:26:59 +02:00
Markus Heiser
cd033b5416 [fix] drop useless pylint: disable=undefined-variable
Since 7b235a1 (see line 591) it is no longer needed to disable
'undefined-variable' for names defined in::

   PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 10:26:15 +02:00
Markus Heiser
aecfb2300d [mod] one logger per engine - drop obsolete logger.getChild
Remove the no longer needed `logger = logger.getChild(...)` from engines.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-06 18:05:46 +02:00
Markus Heiser
343570f7fb [pylint] searx/engines/duckduckgo_definitions.py
BTW: normalize indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-14 09:22:29 +02:00
Alexandre Flament
a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
2021-01-14 20:57:17 +01:00
Alexandre Flament
28cc644f0a [fix] duckduckgo_definitions: fix relative image URL
ddg returns relative URL to https://duckduckgo.com/
2020-12-06 10:14:09 +01:00
Alexandre Flament
b00d108673 [mod] pylint: numerous minor code fixes 2020-12-01 15:21:19 +01:00
Alexandre Flament
3038052c79 [mod] remove unused import
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url  # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
2020-11-14 14:11:02 +01:00
Alexandre Flament
ca593728af [mod] duckduckgo_definitions: display only user friendly attributes / URL
various bug fixes
2020-10-28 08:09:25 +01:00
Alexandre Flament
2006eb4680 [mod] move extract_text, extract_url to searx.utils 2020-10-02 18:13:56 +02:00
Dalf
1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Markus Heiser
30ad0c666d duckduckgo_definitions: remove the debug message
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-12-02 14:26:11 +01:00
Markus Heiser
b6d9f5aa71 [fix] duckduckgo_definition issues reported by 'manage.sh test'
Fix this error while travis build::

  /home/travis/build/asciimoo/searx/searx/engines/duckduckgo_definitions.py:21:44: E225 missing whitespace around operator

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-11-29 19:59:35 +01:00
Markus Heiser
4998e9ec85 [fix] duckduckgo_definitions - where 'AnswerType' is 'calc'
Do not try to get text when 'AnswerType' is 'calc'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-11-29 19:20:14 +01:00
Marc Abonce Seguin
772c048d01 refactor engine's search language handling
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
2018-03-27 00:08:03 -06:00
Adam Tauber
52e615dede [enh] py3 compatibility 2017-05-15 12:02:30 +02:00
marc
af35eee10b tests for _fetch_supported_languages in engines
and refactor method to make it testable without making requests
2016-12-15 00:40:21 -06:00
marc
f62ce21f50 [mod] fetch supported languages for several engines
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
2016-12-13 19:58:10 -06:00
marc
149802c569 [enh] add supported_languages on engines and auto-generate languages.py 2016-12-13 19:32:00 -06:00
a01200356
a44faa7716 [fix] compile regex in ddg_definitions 2016-04-19 12:46:15 -05:00
a01200356
6dca14e95d [enh] multilingual duckduckgo_definitions 2016-04-17 16:34:46 -05:00
Cqoicebordel
8c2a5f0492 DDG Definitions' unit tests 2015-02-09 18:28:08 +01:00
Adam Tauber
2181c4384e [mod] purge local html_to_text 2015-01-01 14:14:56 +01:00
dalf
ffcec383b7 [fix] pep8 : duckduckgo_definitions and wikidata engines 2014-12-07 16:36:20 +01:00
dalf
cac1761a54 [enh] infoboxes : if the result doesn't contain anything except one link, use the normal result template 2014-10-11 15:49:50 +02:00
Dalf
6bfd566353 [enh] add infoboxes and answers 2014-09-28 16:51:41 +02:00
asciimoo
b2492c94f4 [fix] pep/flake8 compatibility 2014-01-20 02:31:20 +01:00
rhapsodhy
74ca542a02 [enh] set up special css classes for highlighting results + highlight definitions with dark grey border 2014-01-11 03:31:54 +00:00
Dalf
e88cf0a0a8 [mod] minor fixes (duckduck_definitions : if a ddg bang is in the query, avoid a useless redirect) 2014-01-05 13:50:17 +01:00
asciimoo
74b6be3991 [enh] engine cfg compatibilty 2013-10-23 23:55:37 +02:00
asciimoo
d793c2733c [enh] engine types 2013-10-15 19:11:43 +02:00
asciimoo
cfba30f1ba [fix] comments removed 2013-10-15 01:19:51 +02:00
asciimoo
e91e4e7fdb [enh] duckduckgo definitions 2013-10-14 23:54:33 +02:00