Commit graph

1699 commits

Author SHA1 Message Date
jazzzooo
102502a4f0 [fix] engine - duckduckgo vqd edge-case 2023-09-20 20:05:06 +02:00
Markus Heiser
043dcbf7c5 [fix] engine qwant (web-lite) - ignore advertising adds
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 17:06:56 +02:00
Émilien (perso)
ad725ce7d7
wikipedia wikidata infobox + disable wikisource (#2806)
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 10:31:02 +02:00
Bnyro
efd3a2d6d1 [feat] engine: implementation of internet archive scholar 2023-09-18 18:12:00 +02:00
jazzzooo
223b3487c3 [fix] spelling 2023-09-18 16:20:27 +02:00
Markus Heiser
a9b6963971 [fix] engine - qwant delivers only 5 pages maximum
all qwant engines (incl qwant-lite) delivers only 5 pages maximum

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18 01:43:43 +02:00
jazzzooo
da1446c5ed [fix] engine - qwant wrong error type 2023-09-18 01:43:43 +02:00
Markus Heiser
7398d525c8 [fix] qwant: subsequent fix of commit d9dbcedeb
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 17:52:56 +02:00
Markus Heiser
d9dbcedeb6 [feat] implementation of qwant lite for web search
Related: https://github.com/searxng/searxng/issues/2719
Replace: https://github.com/searxng/searxng/pull/2748
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 16:53:25 +02:00
Bnyro
b4e0d2eedc [feat] engine: implemenation of moviepilot (de) 2023-09-17 14:30:56 +02:00
jazzzooo
7dfcc3386e [fix] tagesschau videos 2023-09-16 18:40:26 +02:00
jazzzooo
ec540a967a [fix] brave.videos 2023-09-15 22:00:09 +02:00
jazzzooo
27477f51fd [fix] brave.news 2023-09-15 22:00:09 +02:00
Justas Zabulionis
41ef73ca3a [fix] rumble redirect 2023-09-14 19:21:21 +02:00
Justas Zabulionis
be888810ba [fix] pubmed content being None 2023-09-14 18:40:15 +02:00
Justas Zabulionis
92d39de410 [fix] solidtorrents redirects 2023-09-14 18:03:21 +02:00
Justas Zabulionis
cf8a6cf6db [fix] solidtorrents pagination 2023-09-14 18:03:21 +02:00
Justas Zabulionis
8172f89075 [fix] solidtorrents 2023-09-14 18:03:21 +02:00
jazzzooo
74600c028d [fix] engine - Crossref
Crossref was broken on result types journal-issue and component .. The old code
had lots of assumptions, and broke during parsing.  Now the assumptions are more
explicit and checked them with the API.
2023-09-14 17:39:23 +02:00
Bnyro
3568a3cafb [feat] odysee: implement fetch_traits for language support 2023-09-13 21:41:33 +02:00
Bnyro
09c61dabc9 [mod] odysee: time range support 2023-09-13 21:41:33 +02:00
jazzzooo
b98907e91f [fix] engine - piped.music incorrect timestamps 2023-09-13 21:39:37 +02:00
jazzzooo
6039dbf211 [fix] engine - invidious thumbnails 2023-09-13 11:37:42 +02:00
jazzzooo
b2fd6304bf [fix] engine - openstreetmap currency rendering 2023-09-13 10:56:52 +02:00
jazzzooo
54a3e03b45 [fix] engine - openstreetmap currency matching 2023-09-12 20:57:05 +02:00
Bnyro
64d9587ac8 [feat] new engine: svgrepo 2023-09-12 20:38:36 +02:00
jazzzooo
b189578b6b [fix] engine - brave 2023-09-12 11:31:43 +02:00
Bnyro
f182abd6f8 [mod] library of congress: fix engine 2023-09-11 19:42:31 +02:00
Bnyro
e73a6f5d14 [fix] engine deviantart: review of the result-scrapper
The deviantart site changed and hence deviantart is currently unusable.
2023-09-11 13:22:36 +02:00
Alexandre Flament
d07c006aed Replace chompjs with pure Python code
The new implementation is good enough for the current usage (brave)
2023-09-09 13:02:36 +02:00
Bnyro
9e83c0dedc [feat] engine: implementation of Yummly
Co-authored-by: Markus Heiser <markus.heiser@damarit.de>
2023-09-08 11:47:13 +02:00
Bnyro
a3d7e9c285 [mod] utils.py: add markdown_to_text helper function 2023-09-08 11:47:13 +02:00
Hackurei
1f21ac7d62 [feat] engine: implementation of bilibili https://www.bilibili.com 2023-09-05 22:53:03 +02:00
Markus Heiser
696c35d2c3 [fix] engine - duckduckgo_images / determination of vqd value incorrect
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-05 22:24:51 +02:00
bonswouar
4fb26cd96d [fix] engine duckduckgo weather api changes 2023-09-05 16:55:00 +02:00
Markus Heiser
01be9e0e20 [fix] engine: wikicommons - don't quoute ':|' in URL parameters
From [1]: It seems to be because of [2] For some reason it gets url encoded
twice, resulting in

- ``filetype%253Abitmap%257Cdrawing+birds`` instead of
- ``filetype:bitmap%7Cdrawing+birds``

[1] https://github.com/searxng/searxng/issues/2707
[2] https://github.com/searxng/searxng/blob/master/searx/engines/wikicommons.py#L43

Closes: #2707
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-03 18:45:39 +02:00
Markus Heiser
4f8895c6de [fix] follow-up of 4da7003ae / add missing review from @Bnyro
[1] https://github.com/searxng/searxng/pull/2656#pullrequestreview-1607956209

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-02 09:26:45 +02:00
Hackurei
4da7003ae0 [feat] engine: implementation of odysee 2023-09-02 09:14:12 +02:00
Bnyro
9c4e9d3814 [feat] implementation of Wikimedia commons for images 2023-09-01 18:39:24 +02:00
Alexandre Flament
faa4280e1a [mod] bing: resolve redirect without additional requests
Remove the usage of searx.network.multi_requests
The results from Bing contains the target URL encoded in base64
See the u parameter, remove the first two character "a1", and done.

Also add a comment the check of the result_len / pageno
( from https://github.com/searx/searx/pull/1387 )
2023-08-29 07:39:06 +02:00
Markus Heiser
b0d2cd5ca9 [doc] add documentation of Mwmbl engine & autocompleter
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-27 17:25:26 +02:00
Bnyro
19300a5659 [mod] engine mwmbl: add link to official api docs 2023-08-27 17:25:26 +02:00
Alexandre Flament
e16c007c22 [fix] openstreetmap engine
It seems there is an API change:
extratags can be either a dictionnary or None.

This commit avoid crash when extratags is None

Test query "!osm gare du nord"
2023-08-27 11:49:16 +02:00
Markus Heiser
0647f83a3e [fix] google engine: don't overspecify the search query to Google
The method EngineTraits.get_region(..) returns engine's region string
that **best fits** to SearXNG's locale.  This means it returns a
region (country) if only a language is set in the locale.  By example the method
returns for a locale tag `es` a region `ES`.

Google's search parameter `cr` restricts search results to documents originating
in a particular country / in case of a locale tag (language) as described above,
this argument should be unset in the query send to Google.

Closes: https://github.com/searxng/searxng/issues/2672
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-26 07:47:07 +02:00
Markus Heiser
4b42644579 [fix] engine google_video: google has changed the layout of the rsponse
Closes: https://github.com/searxng/searxng/issues/2664
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-22 08:34:04 +02:00
Bnyro
c59ae91b76 [feat] engine: implementation of mwmbl 2023-08-19 18:23:42 +02:00
Markus Heiser
c741fc6f00 [mod] currency_convert: support for showing the answer source url
Show URL of the ddg-search page, not the URL of a (generic) Javascript.  The
latter one is not usefull for the user.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-18 19:07:14 +02:00
Markus Heiser
e2744520f8 [mod] google: support for showing the answer source url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-18 19:07:14 +02:00
Bnyro
5ec7df3480 [mod] engine duckduckgo definitions: support for answer source 2023-08-18 19:07:14 +02:00
Bnyro
64bc98b5fb [mod] brave: support for showing the answer source url 2023-08-18 19:07:14 +02:00
Markus Heiser
9100a48541 [mod] improve seekr engines and add documentation
Tis patch adds some more fields to the result items and changed paging to the
``nextResultSet`` given in seekr's JSON response.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-15 16:17:42 +02:00
Bnyro
2bab658d39 [feat] engine: implementation of seekr for news, images and videos 2023-08-15 16:17:42 +02:00
Bnyro
e25d1c7288 [feat] engine: implementation of German news, Tagesschau
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 20:27:54 +02:00
Bnyro
834e1c3f12 [mod] engine lemmy: increase thumbnail quality to align with theme 2023-08-10 12:58:40 +02:00
Markus Heiser
c381fc001f [mod] settings: remove lemmy from categ 'general' & enable by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 12:58:40 +02:00
Markus Heiser
fda111c0c9 [mod] engine lemmy: add more info fields to the result items
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 12:58:40 +02:00
Bnyro
224f2250ae [feat] engine: support for lemmy communities, posts, comments and users 2023-08-10 12:58:40 +02:00
Bnyro
9f82c39610 [mod] engine google_play: raise error on unsupported category 2023-08-10 12:35:24 +02:00
Bnyro
0a99dc85b9 [mod] engine brave: raise error on unsupported category 2023-08-10 12:35:24 +02:00
allendema_searxng_pi
c00c0c5434 [mod] remove discontinued petalsearch engines 2023-08-09 07:17:40 +02:00
Markus Heiser
b8352eca0c [mod] brave engines: add fetch_traits() / improve language support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-08 16:21:45 +02:00
Markus Heiser
460bbe5b81 [mod] implement brave (WEB) engine to replace XPath configuration
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-08 16:21:45 +02:00
Bnyro
d151497db3 [feat] engine: brave - support for news 2023-08-08 16:21:45 +02:00
Bnyro
cae06f2781 [feat] engine: brave - support for videos 2023-08-08 16:21:45 +02:00
Bnyro
73364e158e [feat] engine: brave - support for images 2023-08-08 16:21:45 +02:00
Markus Heiser
1d0abb7157 [doc] engine bt4g: add documentation to docs/dev/engines/online/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-06 09:30:48 +02:00
Emilien Devos
0fc8f99ecc [feat] new engine: bt4g added & enabled and disable by default btdigg
Disable btdigg because on most SearXNG instances, SearXNG is blocked by btdigg
due to cloudflare too many requests.

This impementation did not parse the HTML page because there is an API in
XML (RSS).  The RSS feed provides fewer data like amount of seeders/leechers and
the files in the torrent file. It's a tradeoff for a "stable" engine as the XML
from RSS content will change way less than the HTML page.

Closes: https://github.com/searxng/searxng/issues/2553
2023-08-06 09:30:48 +02:00
Markus Heiser
db522cf76d [mod] engine: wikimedia - improve results, add addition settings & doc
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-04 19:06:50 +02:00
Bnyro
7d8c20c80d [feat] new engine: wikispecies 2023-08-04 19:06:50 +02:00
Markus Heiser
1b030d4b41 [doc] engine: Yacy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 19:58:51 +02:00
zutto
ca518c6803 add option to change yacy search mode 2023-08-03 19:58:51 +02:00
Markus Heiser
203f1f0928 [fix] engine piped: 'invalid content'
SearXNG does not allow a None value in the content field of a result item.

If the key (shortDescription, uploaderName) in the JSON response from piped
exists but is set to None, SearXNG ignores this result item::

  DEBUG   searx    : result: invalid content: { ..,  'content': None,  ..}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Markus Heiser
207fcc0c8c [mod] engine piped: add paging support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Markus Heiser
ef5831cd84 [mod] engine piped: split into two dedicated engiens for video & music
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Markus Heiser
7aa95d2d52 [doc] engine piped: add documentation to docs/dev/engines/online/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Bnyro
636bfdac68 [feat] engine: implementation of Piped 2023-08-03 16:23:36 +02:00
Paolo Basso
cada89ee36 [feat] engine: re-enables z-library (zlibrary-global.se)
- re-enables z-library as the new domain zlibrary-global.se is now available
  from the open web.   The announcement of the domain:

    https://www.reddit.com/r/zlibrary/comments/13whe08/mod_note_zlibraryglobalse_domain_is_officially/

  It is an official domain, it requires to log in to the "personal" subdomain
  only to download files, but the search works.

- changes the result template of zlibrary to paper.html, filling the appropriate fields
- implements language filtering for zlibrary
- implement zlibrary custom filters (engine traits)
- refactor and document the zlibrary engine
2023-07-07 21:36:51 +02:00
Markus Heiser
5720844fcd [doc] rearranges Settings & Engines docs for better readability
We have built up detailed documentation of the *settings* and the *engines* over
the past few years.  However, this documentation was still spread over various
chapters and was difficult to navigate in its entirety.

This patch rearranges the Settings & Engines documentation for better
readability.

To review new ordered docs::

   make docs.clean docs.live

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-07-01 22:45:19 +02:00
Markus Heiser
87e7926ae9 [fix] engine: Anna's Archive - grep results from '.js-scroll-hidden' elements
The renderuing of the WEB page is very strange; except the firts position all
other positions of Anna's result page are enclosed in SGML comments.  These
cooments are *uncommented* by some JS code, see query of the class
'.js-scroll-hidden' in Anna's HTML template [1].

[1] https://annas-software.org/AnnaArchivist/annas-archive/-/blob/main/allthethings/templates/macros/md5_list.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-29 09:32:57 +02:00
Markus Heiser
e2df6b77a3 [mod] engine: Anna's Archive - additionl settings (content, sort, ext)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-29 09:32:57 +02:00
Markus Heiser
eafc2906f1 [mod] engine: Anna's Archive - fetch search arguments from search form
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-29 09:32:57 +02:00
Paolo Basso
7adb9090e5 [mod] engine: Anna's Archive - add language support 2023-06-29 09:32:57 +02:00
Paolo Basso
e5637fe7b9 [feat] engine: implementation of Anna's Archive
Anna's Archive [1] is a free non-profit online shadow library metasearch engine
providing access to a variety of book resources (also via IPFS), created by a
team of anonymous archivists [2].

[1] https://annas-archive.org/
[2] https://annas-software.org/AnnaArchivist/annas-archive
2023-06-29 09:32:57 +02:00
Paolo Basso
401561cb58 [mod] engine torznab - refactor & option to hide links
- torznab engine using types and clearer code
- torznab option to hide torrent and magnet links.
- document the torznab engine
- add myself to authors

Closes: https://github.com/searxng/searxng/issues/1124
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-28 10:03:44 +02:00
Markus Heiser
da7c30291d [fix] Google API changed
It seems that Google is rolling out a modified WEB API [1][2].

In the past there was only the UI language in the `hl` argument but nowadays it
seems a combination of the UI language and the "search region" is mixed in this
argument and the `gl` argument has been removed.  I'm very surprised that google
is starting to mix the parameters of the UI with the parameters of the search
index.

This patch modifies the get_google_info(..) function.  Beside Google-WEB this
function is also used by other Google services, here are some examples to test
region & language of ..

- Google-WEB:    `!go dragon boat :en-CA`
- Google-News:   `!gon dragon boat :en-CA`
- Google-Videos: `!gov bmw :en-CA`
- Goolge-Images  `!goi bmw :en-CA`

- [1] https://github.com/searxng/searxng/issues/2515#issuecomment-1606294635
- [2] https://github.com/searxng/searxng/issues/2515#issuecomment-1607150817

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-26 18:28:09 +02:00
Markus Heiser
e8706fb738 [fix] engine & network issues / documentation and type annotations
This patch fixes some quirks and issues related to the engines and the network.
Each engine has its own network and this network was broken for the following
engines[1]:

- archlinux
- bing
- dailymotion
- duckduckgo
- google
- peertube
- startpage
- wikipedia

Since the files have been touched anyway, the type annotaions of the engine
modules has also been completed so that error messages from the type checker are
no longer reported.

Related and (partial) fixed issue:

- [1] https://github.com/searxng/searxng/issues/762#issuecomment-1605323861
- [2] https://github.com/searxng/searxng/issues/2513
- [3] https://github.com/searxng/searxng/issues/2515

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-25 13:58:26 +02:00
pankaj
4900c091a6 use logger.warning
logger.warn() is depricated.
logger.warning is already being used in some files.
2023-05-19 19:35:29 +05:30
Markus Heiser
caebd297e9 [fix] engine ddg: minor change in the API of ddg
Closes: https://github.com/searxng/searxng/issues/2419
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-12 18:58:49 +02:00
Markus Heiser
9b575a997b [fix] doc of locales.get_engine_locale() / zh-classical is missleading
Wikipedia's zh-classical is not zh_Hant (see doc-string of engines.wikipedia).
Fixed the example in the doc-string of locales.get_engine_locale() to 'zh_TW'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-17 08:48:57 +02:00
Markus Heiser
f1b6351ae1 [fix] engine: google play movies
Closes: https://github.com/searxng/searxng/pull/1746
Closes: https://github.com/searxng/searxng/issues/1599

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-16 19:15:44 +02:00
Markus Heiser
27369ebec2 [fix] searxng_extra/update/update_engine_descriptions.py (part 1)
Follow up of #2269

The script to update the descriptions of the engines does no longer work since
PR #2269 has been merged.

searx/engines/wikipedia.py
==========================

1. There was a misusage of zh-classical.wikipedia.org:

   - `zh-classical` is dedicate to classical Chinese [1] which is not
     traditional Chinese [2].

   - zh.wikipedia.org has LanguageConverter enabled [3] and is going to
     dynamically show simplified or traditional Chinese according to the
     HTTP Accept-Language header.

2. The update_engine_descriptions.py needs a list of all wikipedias.  The
   implementation from #2269 included only a reduced list:

   - https://meta.wikimedia.org/wiki/Wikipedia_article_depth
   - https://meta.wikimedia.org/wiki/List_of_Wikipedias

searxng_extra/update/update_engine_descriptions.py
==================================================

Before PR #2269 there was a match_language() function that did an approximation
using various methods.  With PR #2269 there are only the types in the data model
of the languages, which can be recognized by babel.  The approximation methods,
which are needed (only here) in the determination of the descriptions, must be
replaced by other methods.

[1] https://en.wikipedia.org/wiki/Classical_Chinese
[2] https://en.wikipedia.org/wiki/Traditional_Chinese_characters
[3] https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter

Closes: https://github.com/searxng/searxng/issues/2330
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 16:03:59 +02:00
Markus Heiser
23ac964e35 [fix] Bing-WEB: use <span class='algoSlug_icon'> for the description
On some result items from Bing-WEB the `<span class='algoSlug_icon'>` tag is the
only tag that contains a description.  The issue can be reproduced by [1]::

    !bi vmware

[1] https://github.com/searxng/searxng/issues/1764#issuecomment-1417990531

Reported-by: @AlyoshaVasilieva
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-08 09:43:04 +02:00
Markus Heiser
2ffd446e5c [mod] clarify the difference of the default category and subgrouping
This PR does no functional change it is just an attempt to make more clear in
the code, what a default category is and what a subcategory is.  The previous
name 'others' leads to confusion with the **category 'other'**.

If a engine is not assigned to a category, the default is assigned::

    DEFAULT_CATEGORY = 'other'

If an engine has only one category and this category is shown as tab in the user
interface, this engine has no further subgrouping::

    NO_SUBGROUPING = 'without further subgrouping'

Related:

- https://github.com/searxng/searxng/issues/1604
- https://github.com/searxng/searxng/pull/1545

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-07 11:03:25 +02:00
Markus Heiser
5234e45010 [fix] Gigablast.com has been erased
[1] https://www.reddit.com/r/searchengines/comments/128wdcp/gigablastcom_has_been_erased/

Closes: https://github.com/searxng/searxng/issues/2321
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-06 08:22:57 +02:00
Markus Heiser
a762172bf7 [fix] engine ddg: quote !bangs in a request send to ddg
Closes: https://github.com/searxng/searxng/issues/392
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03 09:52:16 +02:00
Markus Heiser
0430662189 [fix] engine google-News: fix decoding of URLs (part 2)
Follow up of 8de8070ed to fix the issue reported by AlyoshaVasilieva [1].

[1] https://github.com/searxng/searxng/issues/1959#issuecomment-1493300574

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-02 19:19:59 +02:00
Markus Heiser
8de8070ed9 [fix] engine google-News: fix decoding of URLs
Google-News returns internal links where the origin URL is encoded in a
base64 (RFC 2045 aka URL-safe) string.

Closes: https://github.com/searxng/searxng/issues/1959
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01 19:33:13 +02:00
Markus Heiser
509afbbb84 [fix] engine seznam: fix issues reported by black & pylint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-31 17:25:39 +02:00
Venca24
c8d78355ff [fix] engine seznam 2023-03-31 16:11:27 +02:00
Markus Heiser
270ad18897 [fix] engine flickr: adapt to the new data model from flicker's response
Closes: https://github.com/searxng/searxng/issues/1879
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-30 21:04:53 +02:00
Markus Heiser
2b8dfab33f [fix] engine gigablast: add &userid=<User ID>&code=<Feed Code>
Gigablast's API does block unauthorized request[1].

[1] https://gigablast.com/searchfeed.html

Closes: https://github.com/searxng/searxng/issues/1454
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-29 16:18:02 +02:00
Markus Heiser
6f9e678346 [fix] engine: google has changed the layout of its response
Since 28. March google has changed its response, this patch fixes the google
engine to scrap out the results & images from the new designed response.

closes: https://github.com/searxng/searxng/issues/2287

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-28 14:39:16 +02:00
Markus Heiser
4d4aa13e1f [mod] remove obsolete EngineTraits.supported_languages
All engines has been migrated from ``supported_languages`` to the
``fetch_traits`` concept.  There is no longer a need for the obsolete code that
implements the ``supported_languages`` concept.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
96a2eec3b5 [mod] Archlinux Wiki: improved request API & upgrade to data_type: traits_v1
re-implementation of the Archlinux Wiki:

- fetch_traits(): fetch languages, wiki URLs and title arguments
- add content field to the result list
- add documentation

Wikis from wiki.archlinux.fr, wiki.archlinux.ro, archtr.org/wiki do no longer
exists (has been merged in the main wiki).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
057e9bc1d1 [mod] SepiaSearch: re-engineered & upgrade to data_type: traits_v1
- fetch_traits() SepiaSearch and Peertube are using identical languages.
  Replace module's dictionary `supported_languages` by `engine.traits.languages`
  (data_type: `traits_v1`).
- fixed code to pass pylint
- request(): add argument boostLanguages
- response(): is replaced by peertube's video_response() function, which adds
  metadata from channel name, host & tags

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
8a8c584fec [mod] Dailymotion: improved request API & upgrade to data_type: traits_v1
- fetch_traits(): fetch locales (and languages) from dailymotion API
- removed obsolete data-type "supported_languages"
- add documentation
- improved argument list of the HTTP request:
  - add argument: family_filter_map
  - add conditional argument: localization
    Don't add localization and country arguments if the user does select a
    language (:de, :en, ..)
- improve code quality (mainly improve readability)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
2499899554 [mod] Google: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.

When ever possible the implementations of the Google engines try to make use of
the async REST APIs.  The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - Google (WEB),
  - Google images,
  - Google news,
  - Google scholar and
  - Google videos

  and remove data from obsolete data type "supported_languages".

  A traits.custom type that maps region codes to *supported_domains* is fetched
  from https://www.google.com/supported_domains

searx/autocomplete.py:
  Reversed engineered autocomplete from Google WEB.  Supports Google's languages and
  subdomains.  The old API suggestqueries.google.com/complete has been replaced
  by the async REST API: https://{subdomain}/complete/search?{args}

searx/engines/google.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - always use the async REST API (formally known as 'use_mobile_ui')
  - use *supported_domains* from traits
  - improved the result list by fetching './/div[@data-content-feature]'
    and parsing the type of the various *content features* --> thumbnails are
    added

searx/engines/google_images.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - if exists, freshness_date is added to the result
  - issue 1864: result list has been improved a lot (due to the new cr parameter)

searx/engines/google_news.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
    *supported_domains* is not needed but a ceid list has been added.
  - different region handling compared to Google WEB
  - fixed for various languages & regions (due to the new ceid parameter) /
    avoid CONSENT page
  - Google News do no longer support time range
  - result list has been fixed: XPath of pub_date and pub_origin

searx/engines/google_videos.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - add paging support
  - implement a async request ('asearch': 'arc' & 'async':
    'use_ac:true,_fmt:html')
  - simplified code (thanks to '_fmt:html' request)
  - issue 1359: fixed xpath of video length data

searx/engines/google_scholar.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - request(): include patents & citations
  - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
  - hardening XPath to iterate over results
  - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
  - issue 1769 fixed: new request implementation is no longer incompatible

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
c80e82a855 [mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the DuckDuckGo (DDG) engines including a
improved language and region handling based on the enigne.traits_v1 data.

- DDG Lite
- DDG Instant Answer API
- DDG Images
- DDG Weather

docs/src/searx.engine.duckduckgo.rst:
  Online documentation of the DDG engines (make docs.live)

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - "duckduckgo" (WEB),
  - "duckduckgo images" and
  - "duckduckgo weather"

  and remove data from obsolete data type "supported_languages".

searx/autocomplete.py:
  Reversed engineered Autocomplete from DDG.  Supports DDG's languages.

searx/engines/duckduckgo.py:
  - fetch_traits():  Fetch languages & regions from DDG.

  - get_ddg_lang(): Get DDG's language identifier from SearXNG's locale.  DDG
    defines its languages by region codes.  DDG-Lite does not offer a language
    selection to the user, only a region can be selected by the user.

  - Cache ``vqd`` value: The vqd value depends on the query string and is needed
    for the follow up pages or the images loaded by a XMLHttpRequest (DDG
    images).  The ``vqd`` value of a search term is stored for 10min in the
    redis DB.

  - DDG Lite engine: reversed engineered request method with improved Language
    and region support and better ``vqd`` handling.

searx/engines/duckduckgo_definitions.py: DDG Instant Answer API
  The *instant answers* API does not support languages, or at least we could not
  find out how language support should work.  It seems that most of the features
  are based on English terms.

searx/engines/duckduckgo_images.py: DDG Images
  Reversed engineered request method.  Improved language and region handling
  based on cookies and the enigne.traits_v1 data.  Response: add image format to
  the result list

searx/engines/duckduckgo_weather.py: DDG Weather
  Improved language and region handling based on cookies and the
  enigne.traits_v1 data.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
e9afc4f8ce [mod] Startpage: reversed engineered & upgrade to data_type: traits_v1
One reason for the often seen CAPTCHA of the Startpage requests are the
incomplete requests SearXNG sends to startpage.com: this patch is a complete new
implementation of the ``request()`` function, reversed engineered from the
Startpage's search form.  The new implementation:

- use traits of data_type: traits_v1 and drop deprecated data_type: supported_languages
- adds time-range support
- adds save-search support
- fix searxng/searxng/issues 1884
- fix searxng/searxng/issues 1081 --> improvements to avoid CAPTCHA

In preparation for more categories (News, Images, Videos ..) from Startpage, the
variable ``startpage_categ`` was set up.  The default value is ``web`` and other
categories from Startpage are not yet implemented.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
858aa3e604 [mod] wikipedia & wikidata: upgrade to data_type: traits_v1
BTW this fix an issue in wikipedia: SearXNG's locales zh-TW and zh-HK are now
using language `zh-classical` from wikipedia (and not `zh`).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
e0a6ca96cc [doc] add a description of bing engines (web, news, video, images)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
15eaf0f15f [mod] bing_news: use async API & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
ff80e7637e [mod] bing_images: use async API & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
bc21d28298 [mod] bing_videos: use async API & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
d0f465e6fa [mod] bing: add time_range support & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
7daf4f95ef [mod] Wikipedia: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Wikipedia engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
f78f908383 [mod] Google: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Google engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
dba8977b09 [mod] DuckDuckGo: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the DuckDuckGo engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
ef143729a0 [mod] yahoo: fetch engine traits (data_type: traits_v1)
Implements a fetch_traits function for the Yahoo engine.

.. note::

   Includes migration of the request methode from 'supported_languages' to
   'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
c1ae2ef57c [mod] qwant: fetch engine traits (data_type: traits_v1)
Implements a fetch_traits function for the Qwant engines.

.. note::

   Includes migration of the request methode from 'supported_languages' to
   'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
fc0c775030 [mod] Dailymotion: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Dailymotion engine.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
61383edb27 [mod] Startpage: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Startpage engine.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
d3aa690a7a [mod] bing: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Bing engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
a7fe22770a [mod] Peertube: re-engineered & upgrade to data_type: traits_v1
- fetch_traits(): Fetch languages from peertube's search-index source code.

  [mod] Include migration of the request methode from 'supported_languages'
        to 'traits' (EngineTraits) object.
  [fix] old supported_languages_url is no longer valid since the sources
        has been moved to a different path.

- fixed code to pass pylint
- request(): complete re-implementation based on the API docs [1]
- response(): complete re-implementation, adds serveral fields missed before
- add source code documentation

[1] https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
6e5f22e558 [mod] replace engines_languages.json by engines_traits.json
Implementations of the *traits* of the engines.

Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*.  Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.

To load traits from the persistence::

    searx.enginelib.traits.EngineTraitsMap.from_data()

For new traits new properties can be added to the class::

    searx.enginelib.traits.EngineTraits

.. hint::

   Implementation is downward compatible to the deprecated *supported_languages
   method* from the vintage implementation.

   The vintage code is tagged as *deprecated* an can be removed when all engines
   has been ported to the *traits method*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Solirs
ac169a0f75 Pass black formatting test 2023-03-21 00:41:36 +01:00
Solirs
e26bce33d4 WIKIDATA: Add description for results 2023-03-21 00:14:54 +01:00
Alexandre Flament
3e9cddc606
rollback test 2023-03-15 19:55:20 +01:00
Alexandre Flament
41ed0ef0c7
test 2023-03-15 19:53:53 +01:00
Markus Heiser
4c06837a50 [mod] make python code pylint 2.16.1 compliant
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 13:59:21 +01:00
Markus Heiser
257dc7d6c4 [fix-2146] set different HTTP Referer header to DuckDuckGo requests
For what ever reasons, ddg-lite don't like the Referer

  https://lite.duckduckgo.com/

In an interactive session in the WEB browser the the Reverer has exactly this
value, but ddg-lite don't like this value when the request is build up by
SearXNG.  The new value is:

  https://google.com/

What fakes a user comes from a google link.

Related: https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2146

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-03 08:45:51 +01:00
Alexandre Flament
9d102fb08f
Merge pull request #2132 from dalf/update_pr_1967
search.suspended_time settings: bug fixes
2023-01-29 20:48:43 +01:00
Alexandre Flament
bfca63c536 wikipedia engine: update _fetch_supported_languages
the layout https://meta.wikimedia.org/wiki/List_of_Wikipedias has changed
2023-01-29 10:01:58 +00:00
Alexandre Flament
8256de2fe8 peertube engine: update _fetch_supported_languages
There is now an API to get the list of supported languages
https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getLanguages
2023-01-29 10:01:54 +00:00
Alexandre Flament
37addec69e search.suspended_time settings: bug fixes
* fix type in settings.yml: replace suspend_times by suspended_times
* always use delay defined in settings.yml:
  * HTTP status 402 and 403: read the value from settings.yml instead of using the hardcoded value of 1 day.
  * startpage engine: CAPTCHA suspend the engine for one day instead of one week
2023-01-28 10:24:14 +00:00
Ahmad Alkadri
7fc8d72889 [fix] bing: parsing result; check to see if the element contains links
This patch is to hardening the parsing of the bing response:

1. To fix [2087] check if the selected result item contains a link, otherwise
   skip result item and continue in the result loop.  Increment the result
   pointer when a result has been added / the enumerate that counts for skipped
   items is no longer valid when result items are skipped.

   To test the bugfix use:   ``!bi :all cerbot``

2. Limit the XPath selection of result items to direct children nodes (list
   items ``li``) of the ordered list (``ol``).

   To test the selector use: ``!bi :en pontiac aztek wiki``

   .. in the result list you should find the wikipedia entry on top,
   compare [2068]

[2087] https://github.com/searxng/searxng/issues/2087
[2068] https://github.com/searxng/searxng/issues/2068
2023-01-09 15:08:24 +01:00
ahmad-alkadri
9ee99423fe [fix] Bing-Web engine: XPath to get the wikipedia result
Modify the XPath selector to get the wikipedia result plus small fixes.

About result content: especially with the Wikipedia result, we'd get several
paragraph elements, only the first paragraph would be taken and displayed on the
search result
2023-01-08 09:11:16 +01:00
Rudis Muiznieks
128b8c7f0a
Add HTTP Referer header to DuckDuckGo requests
closes #2080
2023-01-06 16:07:37 -06:00
Rudis Muiznieks
6804ff048d
Fix: add trailing slash to duckduckgo url
Close #1854
2022-12-22 07:49:58 -06:00
Alexandre Flament
269326063a Fix: don't crash when engine or name is missing in settings.yml
SearXNG crashes if the engine or name fields are missing.
With this commit, the app displays an error in the log and keeps loading.

Close #1951
2022-12-04 23:43:59 +01:00
Émilien Devos
46ad32343a Switch back to protobuf for raw HTML 2022-11-11 07:39:48 +00:00
ngosang
78be4b4c70 Fix Google search engine.
- Fix broken links. Resolves #1794
- Fix missing results. Resolves #1829
2022-11-11 07:34:19 +01:00
Alexandre Flament
8f19bdaf17
Merge pull request #1882 from fehho/metacpan
Add MetaCPAN engine
2022-11-07 21:54:11 +01:00
fehho
fe351c2802 Add MetaCPAN engine 2022-11-07 08:07:06 -06:00
Vasilis Gerakaris
947b62c9d5
Fix floating point format in DDG weather humidity
Fixes #1836
2022-10-20 11:44:17 +03:00
Alexandre FLAMENT
035bc507ec [fix] startpage engine 2022-10-14 18:27:53 +00:00
Alexandre Flament
a3148e5115
Merge pull request #1814 from return42/fix-typos
[fix] typos / reported by @kianmeng in searx PR-3366
2022-09-28 09:22:02 +02:00
Alexandre Flament
0e00af9c26
Merge pull request #1810 from return42/fix-1809
[fix] springer: unsupported operand type(s) for +: 'NoneType' and 'str'
2022-09-28 09:20:03 +02:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
Markus Heiser
52023e3d6e [fix] doc of the paper.html template (isbn, issn)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-25 15:46:29 +02:00
Markus Heiser
0052887929 [fix] springer: unsupported operand type(s) for +: 'NoneType' and 'str'
- fix issue reported #1809
- filter out `None` value from issn and isbn list
- add comments (from publicationName)
- add publisher

Closes: https://github.com/searxng/searxng/issues/1809
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-25 15:25:55 +02:00
Markus Heiser
e36b023508 [mod] core.ac.uk: add cetgory 'scientific publications'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-24 16:16:22 +02:00
Alexandre Flament
16443d4f4a [mod] core.ac.uk: try multiple ways to get url
If the url is not found, using:
* the DOI
* the downloadUrl
* the ARK id
2022-09-24 15:02:39 +02:00
Markus Heiser
c76830d8a8 [mod] core.ac.uk: use paper.html template
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-24 13:19:33 +02:00
Markus Heiser
3ff2ad939d [fix] ERROR searx.engines.core.ac.uk: list index out of range
Some result items from core.ac.uk do not have an URL::

  Traceback (most recent call last):
  File "searx/search/processors/online.py", line 154, in search
    search_results = self._search_basic(query, params)
  File "searx/search/processors/online.py", line 142, in _search_basic
    return self.engine.response(response)
  File "SearXNG/searx/engines/core.py", line 73, in response
    'url': source['urls'][0].replace('http://', 'https://', 1),

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-24 13:19:33 +02:00
Alexandre Flament
d6446be38f [mod] science category: various update of about PR 1705 2022-09-23 20:52:55 +02:00
Alexandre FLAMENT
e36f85b836 Science category: update the engines
* use the paper.html template
* fetch more data from the engines
* add crossref.py
2022-09-23 20:45:58 +02:00
Alexandre Flament
bef3984d03
Merge pull request #1728 from liimee/eng-ddw
add duckduckgo weather engine
2022-09-23 18:14:09 +02:00
Alexandre Flament
d3fec1388c
Merge pull request #1624 from liimee/eng-wttr
Add wttr.in engine
2022-09-23 18:13:37 +02:00
Alexandre Flament
1a7b6872b5
Merge pull request #1792 from unixfox/google-images-internal-api
use the internal API for google images
2022-09-21 19:50:38 +02:00
Markus Heiser
cf7ee67f71 [mod] google-images: slightly improvements of the engine
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-21 18:59:55 +02:00
Emilien Devos
df5f8d0e8e use the internal API for google images 2022-09-20 22:52:38 +02:00
Markus Heiser
dcf1d408a5 [fix] google-news: origin result does not have a content area
The google news are in a rework, the content area of a news item has been
removed.

Closes: https://github.com/searxng/searxng/issues/1790
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-20 20:18:43 +02:00
Markus Heiser
fbf07237ff [fix] and improve docs generated from source code.
Fix::

    searx/locales.py:docstring of searx.locales.get_engine_locale:17: \
      WARNING: Definition list ends without a blank line; unexpected unindent.

Improvement: don't show default values in the generated documentation whe it is
more a mess than a usefull information (`:meta hide-value:`).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-18 12:44:12 +02:00
Alexandre FLAMENT
dd0887be18 xpath engine: change raise_for_httperror to no_result_for_http_status
no_result_for_http_status contains a list of HTTP status.
These HTTP status are seen an empty result list.
In other cases an exception is thrown as usual.

Previously raise_for_httperror were ignoring all HTTP error,
which make defective engines invisible in the stats.
2022-09-04 09:07:28 +02:00
Markus Heiser
a15dfa5ee1 [fix] engine woxikon.de - don't raise exception on empty result list
Woxikon expects a word in German, so with query "foo" the site finds nothing and
respons a 404:

    httpx.HTTPStatusError: Client error '404 Not Found' \
      for url 'https://synonyme.woxikon.de/synonyme/foo.php'

[1] https://github.com/searxng/searxng/issues/1543#issuecomment-1193317054

Closes: https://github.com/searxng/searxng/issues/1543
Suggested-by: @allendema [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-04 09:07:28 +02:00
Markus Heiser
8e9fb0b435
Merge pull request #1647 from return42/deepl-engine
[mod] add deepl translation engine
2022-09-02 14:09:22 +02:00
ta
85b5293e40 simplify infobox result 2022-08-31 18:29:50 +07:00
ta
12f7d4a46b add duckduckgo weather engine 2022-08-31 17:29:32 +07:00
Alexandre Flament
56000d5162
Merge pull request #1699 from liimee/eng-app-store
add apple app store engine
2022-08-27 07:43:23 +02:00
Alexandre Flament
44bc94c36e
Merge pull request #1700 from liimee/eng-ddm
add apple maps engine
2022-08-27 07:36:16 +02:00
ta
5057007270 remove thumbnail from results 2022-08-27 06:23:30 +07:00
ta
525946d7dd add poi's website and phone number, doesn't crash when there is no displayMapRegion, query the token on the first request 2022-08-27 06:17:58 +07:00
ta
5dce299b22 add apple maps engine 2022-08-25 17:05:40 +07:00
ta
cef7bbab22 get the not cropped version of the thumbnail when the image height is not too important 2022-08-24 18:33:11 +07:00
ta
78bff4618c add safesearch support 2022-08-24 18:31:04 +07:00
ta
bcae7ae4e3 add developer info as author 2022-08-24 17:50:38 +07:00
ta
e5c1b64b1d add the apple app store engine
The Apple App Store is the digital app distribution platform for iOS & iPadOS.
2022-08-24 17:27:36 +07:00
ta
040e24f9ad support playing videos directly 2022-08-24 16:48:31 +07:00
ta
79d06509c1 add tags as suggestions 2022-08-23 05:18:35 +07:00
ta
d22f469010 use invalid-name instead of C0103 for pylint 2022-08-22 18:27:35 +07:00
ta
dd9127492f add 9gag engine
9GAG is a social media website where users upload and share user-generated images and videos
2022-08-22 17:35:07 +07:00
ta
e64cca8c3f don't raise error when nothing was found 2022-08-22 17:04:29 +07:00
M Asenov
faa32d5773 fixed xpath selector for appropriate results 2022-08-21 20:08:00 +01:00
Alexandre Flament
5ed40af3ba
Merge pull request #1661 from liimee/eng-tw
Add twitter engine
2022-08-21 15:21:18 +02:00
Markus Heiser
77a0f33819 [fix] engine duden - don't raise exception on empty result list
Duden expects a word in German, so with query "amazing" the site finds nothing
and respons a 404:

    httpx.HTTPStatusError: Client error '404 Not Found' for url\
      'https://www.duden.de/suchen/dudenonline/amazing'

[1] https://github.com/searxng/searxng/issues/1543#issuecomment-1193317054

Suggested-by: @allendema [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-20 08:41:03 +02:00
ta
05851978cf add explanation of token 2022-08-17 19:45:42 +07:00
ta
c8acd4a3b6 add profile image to user results 2022-08-17 14:30:59 +07:00
ta
b6fd7cd571 add thumbnail to results if available 2022-08-17 14:25:22 +07:00
Markus Heiser
27385e7898 [mod] qwant - add safesearch option
Closes: https://github.com/searxng/searxng/issues/1640
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:36:14 +02:00
Markus Heiser
6579d6d558 [fix] qwant - API error::locale must be one ..
The request function should not request a language (aka locale) that is not
supported by qwant. Select a locale like zh-TW ends in qwant's API error:

  ERROR searx.engines.qwant news: exception : \
  API error::locale must be one of the following values: \
    en_gb, en_ie, en_us, en_ca, en_my, en_au, en_nz, de_de, de_ch, de_at, fr_fr, \
    fr_be, fr_ch, fr_ca, fr_ad, fc_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx, \
    es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_pt, pt_ad, \
    nl_be, nl_nl

The existing searx.utils.match_language function is unsuitable for this purpose,
it is replaced by function searx.locales.get_engine_locale that is based on the
methods from the babel package.

The quant's _fetch_supported_languages function has been revised to filter out
languages 8aka locales) not supported by qwant.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:36:14 +02:00
Markus Heiser
75bb8c45d0 [mod] decouple qwant's categories from SearXNG's categories
By using new property `qwant_categ:` the category of qwant is no longer bound to
the category of SearXNG.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:26:54 +02:00
ta
96ea355a1f add twitter engine 2022-08-14 08:39:41 +07:00
Markus Heiser
eb02cc77c5 [fix] google - simplify XPath selectors to fetch more results
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-10 18:55:31 +02:00
Émilien Devos
b9f16a77db output format protobuf to HTML for google mobile 2022-08-10 09:36:06 +00:00
Thomas Renard
d4acbcfe63 [mod] add deepl translation engine
This implements the Deepl Translation engine. It works nearly like lingva but
directly to the deepl API.  This api only needs a to-lang, from-lang is a fake
by now.

There is a free option to use [1].

[1] https://www.deepl.com/pro-api?cta=header-pro-api for registering a free account.
2022-08-10 09:14:36 +02:00
Brock Vojković
24210fb10b
Revert PR #1633
This reverts the changes made to the Google results XPath in PR #1633.
2022-08-10 03:41:39 +02:00
Léon Tiekötter
94b3656b4a [fix] google engine: results XPath
Seems google rolls out changes first on the `google.com` domain and later on the
"language" domains.  By example: yesterday [1] `google.com` did not work but
`google.de` and `google.fr` did work, today they do not work any longer and this
fix is needed on all domains.

Closes: https://github.com/searxng/searxng/issues/1628
[1] https://github.com/searxng/searxng/issues/1628#issuecomment-1208191816
2022-08-09 06:23:59 +02:00
liimee
8c318562e2
add description and wikidata ID to wttr.in engine 2022-08-07 14:57:10 +07:00
ta
8aa018db95 add wttr.in engine 2022-08-07 13:04:18 +07:00