Commit graph

1255 commits

Author SHA1 Message Date
Martin Fischer
576e19dad1 [fix] add default for "about" engine property
Fixes #732.
2022-01-10 08:40:06 +01:00
Markus Heiser
4fc5e5299c [fix] ccengine engine - avoid unwanted redirects
api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:

    /v1/images? -->/ v1/images/?

otherwise it redirects, here is the debug log:

    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
    DEBUG   searx.network.openverse       : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
    WARNING searx.engines.openverse       : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-07 14:14:31 +01:00
Léon Tiekötter
37baf46ece [fix] Rename ccengine engine to openverse
The CC engine was merged with WordPress and renamed to Openverse

Source: https://wordpress.org/news/2021/05/welcome-to-openverse/
2022-01-07 13:06:05 +01:00
Léon Tiekötter
4be6deb0a1 [fix] ccengine engine
Change domain to api.openverse.engineering
2022-01-07 13:01:37 +01:00
Markus Heiser
ced656606f
Merge pull request #709 from return42/drop-etools
[fix] drop etools engine module
2022-01-07 11:18:47 +01:00
Markus Heiser
5dd3442f83 [fix] drop etools engine module
The implementation of the etools engine is poor.  No date-range support, no
language support and it is broken by a CAPTCHA.

etools is a metasearch engine, the major search engines it supports (google,
bing, wikipedia, Yahoo) are already available in SeaarXNG.

While etools does support several engines we currently don't support directly,
support for them should be added directly to SearXNG if there is demand.

In practice: in SearXNG the worse etools results will be mixed with good results
from other engines we have (as long as there is no captcha).

At best case, what we win with etools is in e.g. results from de.ask.com in a
query from a german request .. in all other cases worse results are bubble up in
SearXNG's result list.

[1] https://github.com/searxng/searxng/issues/696#issuecomment-1005855499

Closes: https://github.com/searxng/searxng/issues/696
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-07 10:41:09 +01:00
Martin Fischer
e12525a1fa
Merge pull request #708 from not-my-profile/pref-refactor
Refactor `preferences`
2022-01-07 09:45:23 +01:00
Léon Tiekötter
3ab826de22 Drop microsoft academic engine
Microsoft academic was discontinued on 2021-12-31.

Source: https://www.microsoft.com/en-us/research/project/academic/articles/microsoft-academic-to-expand-horizons-with-community-driven-approach/
2022-01-07 01:35:13 +01:00
Martin Fischer
bb06758a7b [refactor] add type hints & remove Setting._post_init
Previously the Setting classes used a horrible _post_init
hack that prevented proper type checking.
2022-01-06 14:21:14 +01:00
Alexandre Flament
aedd6279b3
Merge pull request #634 from not-my-profile/powered-by
Introduce `categories_as_tabs` & group engines in tabs
2022-01-06 09:22:02 +01:00
Alexandre Flament
d3ecadd3f8
Merge pull request #679 from dalf/brand-searxng
searxng.org: update setup.py & settings.yml
2022-01-05 19:07:53 +01:00
Martin Fischer
d01e8aa8cc [mod] introduce searx.engines.Engine for type hinting 2022-01-05 11:03:44 +01:00
Martin Fischer
1e195f5b95 [mod] move group_engines_in_tab to searx.webutils 2022-01-05 11:03:44 +01:00
Martin Fischer
5d74bf3820 [enh] move dictionaries, Erowid & IMDb out of general category
The general category is the category that is searched by default.
From a privacy standpoint it doesn't make sense to send all general
queries to specialized search engines that cannot deal with those
queries anyway.
2022-01-05 11:03:44 +01:00
Martin Fischer
ab90e2ac49 [enh] show categories not in any tab category in "Other" preferences tab
Previously we didn't have a good place to put search engines that don't
fit into any of the tab categories. This commit automatically puts
search engines that don't belong to any tab category in an "other"
category, that is only displayed in the user preferences (and not above
search results).
2022-01-05 11:03:44 +01:00
Martin Fischer
b02f762687 [enh] add more categories 2022-01-05 11:00:11 +01:00
Martin Fischer
8e9ad1ccc2 [enh] introduce categories_as_tabs
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.

This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs  can still be searched with !bangs.
2022-01-03 07:01:49 +01:00
Martin Fischer
df34b1ddcf [enh] settings.yml: allow granular overwrites for about 2022-01-03 07:01:49 +01:00
Alexandre Flament
d83aa2b0d2
Merge pull request #613 from return42/pylint-bing-images
[pylint] Bing (Images) engine
2022-01-02 22:00:55 +01:00
Alexandre Flament
76cbfbbdda reference docs.searxng.org 2022-01-02 21:18:29 +01:00
Markus Heiser
61ce0c2244 [fix] bing engines: fetch_supported_languages
The Request to and the Response from https://www.bing.com/account/general has
been changed.

[1] https://github.com/searxng/searxng/pull/672#discussion_r777104919

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-01 17:31:38 +01:00
Markus Heiser
dc4f1f705d [pylint] Bing (Images) engine
Fix remarks from pylint and remove obsolete try/except block

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-28 14:43:39 +01:00
Markus Heiser
6d7a38a912 [pylint] Bing (Videos) engine
Fix remarks from pylint and remove obsolete try/except block

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-28 14:33:05 +01:00
Markus Heiser
d84226bf63 [fix] issues reported by pylint
Fix pylint issues from commit (3d96a983)

    [format.python] initial formatting of the python code

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 10:16:20 +01:00
Markus Heiser
3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:26:22 +01:00
Markus Heiser
fcdc2c2cd2 [format.python] disable py code formatting for some hunks of code
Disable the python code formatting from python-black, where the readability of
code suffers by formatting.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:16:03 +01:00
Martin Fischer
e28c6bda35 [doc] introduce about.language and sort engines by it 2021-12-21 09:58:51 +01:00
Markus Heiser
7a215e07e7
Merge pull request #611 from return42/fix-bing
[fix] bing engine: fix paging support, show inital page.
2021-12-20 10:08:52 +01:00
Markus Heiser
2af50c2588 [pylint] Reddit engine
Add Reddit engine to pylint process

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-18 17:59:47 +01:00
Markus Heiser
6b85607274 [fix] bing engine: fix paging support, show inital page.
Follow up queries for the pages needed to be fixed.

- Split search-term in one for initial query and one for following queries.
- Set some headers in HTTP requests, bing needs for paging support.
- IMO //div[@class="sa_cc"] does no longer match in a bing response.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-18 13:50:38 +01:00
Markus Heiser
b2177e5916 [pylint] Bing (Web) engine
Fix remarks from pylint and improved code-style.  In preparation for a bug-fix
of the Bing (Web) engine I add this engine to the pylint-list.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-18 13:40:36 +01:00
Markus Heiser
f41734a543 [fix] engine bing-news: replace the http:// by https://
BTW: add bing_news to the pylint process

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-17 13:25:50 +01:00
Markus Heiser
8cc7c880ae
Merge pull request #587 from dalf/fix-gigablast
[fix] gigablast engine
2021-12-12 15:58:13 +01:00
Markus Heiser
b5c9cc4ff3
Merge pull request #586 from dalf/remove-yggtorrent
[del] remove yggtorrent
2021-12-07 07:00:47 +01:00
Alexandre Flament
1a6207574e [fix] gigablast engine
fetch extra params after 3000 seconds
2021-12-06 22:55:15 +01:00
Alexandre Flament
fbc2a6ab4b [del] remove yggtorrent
yggtorrent is behind cloudflare now
close #580
2021-12-06 21:59:51 +01:00
Alexandre Flament
037cb7dd3d [fix] imdb: don't crash when there is no result 2021-12-06 21:49:18 +01:00
Markus Heiser
6e06618e0c [fix] google-videos engine: ignore news articles
In the video search, google also sometimes includes news.  E.g. in the DE
language when you search for `!gov paris`, google adds an article from a german
newspaper (FAZ), I assume these are sponsored link (not tagged advertisement?)

Those links do not have an image / this patch ignores *video links* wqithout an
image ID.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-26 17:11:20 +01:00
Markus Heiser
1ce09df9aa [fix] google video engine - rework of the HTML parser
The google video response has been changed slightly, a rework of the parser was
needed.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-26 01:14:17 +01:00
Markus Heiser
488ace1da9 [fix] google engine - suggestion
BTW: google no longer offers *spelling suggestions*

Closes: https://github.com/searxng/searxng/issues/442
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-25 19:42:03 +01:00
Markus Heiser
5b28c9109f [fix] google images: @href index 0 not found
Sometimes there is no href in the `<a ..>` tag of a *link_node* [1].

[1] https://github.com/searxng/searxng/issues/532

Reported-by: @TheEssem
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-21 09:55:59 +01:00
Markus Heiser
4c82ac7670 [drop] engine digg - https://digg.com/api is no longer available
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-19 15:00:22 +01:00
Tom
e1d60051ca
[fix] Qwant search query string
Search string: "!qwant time"
Resulting request URL: https://api.qwant.com/v3/search/web?q=q=time&count=10&offset=0&device=desktop&safesearch=1&locale=en_US
Notice the double "q="

Resulting request URL after fix: https://api.qwant.com/v3/search/web?q=time&count=10&offset=0&device=desktop&safesearch=1&locale=en_US
2021-11-17 18:13:54 +01:00
MrPaulBlack
41494d9f47 [fix] make reddit only in social media category avail.
fix https://github.com/searxng/searxng/issues/470
2021-11-01 20:37:17 +01:00
Alexandre Flament
64b29ad838 [mod] microsoft academic: increase timeout to 6 seconds
also avoid a crash when there is no result

close #433
2021-10-26 12:26:43 +02:00
Markus Heiser
713814547a [fix] yahoo engine - don't lump all search suggestions together
Closes: https://github.com/searxng/searxng/issues/421
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-21 07:51:05 +00:00
Markus Heiser
f63ffbb22b [fix] engine - yahoo: rewrite and fix issues
Languages are supported by mapping the language to a domain.  If domain is not
found in :py:obj:`lang2domain` URL ``<lang>.search.yahoo.com`` is used.

BTW: fix issue reported at https://github.com/searx/searx/issues/3020

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-16 20:05:26 +00:00
Markus Heiser
38a157b56f [pylint] engines: yahoo fix several issues reported from pylint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-16 20:05:26 +00:00
MrPaulBlack
00b0394e19 [fix] language param for qwant 2021-10-14 16:11:44 +00:00
Noémi Ványi
4cc1ee8565 [fix] qwant engine - only get results from categories
Reported-by: https://github.com/searx/searx/issues/3014
Cherry-picked: https://github.com/searx/searx/commit/3bcca43
2021-10-12 18:42:50 +00:00