mirrors/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-12-25 10:40:29 +00:00
Code Issues Projects Releases Wiki Activity
7396 commits 6 branches 17 tags 132 MiB
Commit graph

670 commits

Author SHA1 Message Date
Alexandre FLAMENT
e92755d358 Initialize Redis in searx/webapp.py 
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"

The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.

It worked before because SearXNG initialized the Redis connection when the limiter started.

In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
 2022-11-05 17:45:52 +01:00
Alexandre Flament
32e8c2cf09 searx.network: add "verify" option to the networks 
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.

SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
 2022-10-14 13:59:22 +00:00
Mohamed Elashri
8d5653e60d
Merge branch 'searxng:master' into master 2022-09-30 23:06:54 +00:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366 
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-09-27 18:32:14 +02:00
Mohamed Elashri
5832c70680
correct sci-hub links/ add .ru and remove other 3rd party domains. 2022-09-24 11:03:57 -04:00
Markus Heiser
caebafdd06 [fix] typo in crossref settings: disable --> disabled 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-09-24 08:12:36 +02:00
Alexandre Flament
d6446be38f [mod] science category: various update of about PR 1705 2022-09-23 20:52:55 +02:00
Alexandre FLAMENT
e36f85b836 Science category: update the engines 
* use the paper.html template
* fetch more data from the engines
* add crossref.py
 2022-09-23 20:45:58 +02:00
Alexandre Flament
bef3984d03
Merge pull request #1728 from liimee/eng-ddw 
add duckduckgo weather engine
 2022-09-23 18:14:09 +02:00
Alexandre Flament
d3fec1388c
Merge pull request #1624 from liimee/eng-wttr 
Add wttr.in engine
 2022-09-23 18:13:37 +02:00
Alexandre FLAMENT
33b43763b9 Brave engine: fix BrotliDecoderDecompressStream error 2022-09-18 22:08:38 +00:00
LencoDigitexer
3f72a79088 add yandex to autocomplete backends settings 2022-09-09 23:50:58 +03:00
Markus Heiser
ad8ffd222c [mod] option 'ui: cache_url:' to configure internet cache or archive 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-09-04 09:42:20 +02:00
Alexandre FLAMENT
dd0887be18 xpath engine: change raise_for_httperror to no_result_for_http_status 
no_result_for_http_status contains a list of HTTP status.
These HTTP status are seen an empty result list.
In other cases an exception is thrown as usual.

Previously raise_for_httperror were ignoring all HTTP error,
which make defective engines invisible in the stats.
 2022-09-04 09:07:28 +02:00
Markus Heiser
a15dfa5ee1 [fix] engine woxikon.de - don't raise exception on empty result list 
Woxikon expects a word in German, so with query "foo" the site finds nothing and
respons a 404:

    httpx.HTTPStatusError: Client error '404 Not Found' \
      for url 'https://synonyme.woxikon.de/synonyme/foo.php'

[1] https://github.com/searxng/searxng/issues/1543#issuecomment-1193317054

Closes: https://github.com/searxng/searxng/issues/1543
Suggested-by: @allendema [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-09-04 09:07:28 +02:00
Markus Heiser
8e9fb0b435
Merge pull request #1647 from return42/deepl-engine 
[mod] add deepl translation engine
 2022-09-02 14:09:22 +02:00
Émilien Devos
fcccf39030
Disable brave by default 
Brave is too unstable and will often not work by default. As seen in many issues: https://github.com/searxng/searxng/issues?q=is%3Aissue++sort%3Aupdated-desc+brave+label%3Abug+
 2022-08-31 15:47:56 +02:00
ta
12f7d4a46b add duckduckgo weather engine 2022-08-31 17:29:32 +07:00
Alexandre FLAMENT
341ad46303 settings.yml: set default values for result_proxy 
* initialize result_proxy with searx/settings_defaults.py
* allow result_proxy.key to be a string

this commit supersedes #1522
 2022-08-28 09:27:53 +00:00
Alexandre Flament
56000d5162
Merge pull request #1699 from liimee/eng-app-store 
add apple app store engine
 2022-08-27 07:43:23 +02:00
Alexandre Flament
44bc94c36e
Merge pull request #1700 from liimee/eng-ddm 
add apple maps engine
 2022-08-27 07:36:16 +02:00
ta
5dce299b22 add apple maps engine 2022-08-25 17:05:40 +07:00
ta
e5c1b64b1d add the apple app store engine 
The Apple App Store is the digital app distribution platform for iOS & iPadOS.
 2022-08-24 17:27:36 +07:00
ta
dd9127492f add 9gag engine 
9GAG is a social media website where users upload and share user-generated images and videos
 2022-08-22 17:35:07 +07:00
Alexandre Flament
5ed40af3ba
Merge pull request #1661 from liimee/eng-tw 
Add twitter engine
 2022-08-21 15:21:18 +02:00
Markus Heiser
75bb8c45d0 [mod] decouple qwant's categories from SearXNG's categories 
By using new property `qwant_categ:` the category of qwant is no longer bound to
the category of SearXNG.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-08-14 10:26:54 +02:00
ta
96ea355a1f add twitter engine 2022-08-14 08:39:41 +07:00
Léon Tiekötter
8fa84ee9f4 Fix time range support and add soft_max_redirects 
Add custom time_range_url and time_range_map

Set soft_max_redirects = 2 to prevent "ErrorContext('searx/search/processors/online.py', 116, 'count_error(', None, '2 redirects, maximum: 0', ('200', 'OK', 'neeva.com')) True"
 2022-08-13 07:35:15 +07:00
ta
d6dfada1a9 add neeva web engine 
Neeva is "the world's first ad-free, private search engine" and uses data from Apple, Bing, Yelp and "others".
They claim to crawl "hundreds of millions" of URLs a day (https://twitter.com/Neeva/status/1536447373903335426).
 2022-08-13 07:26:36 +07:00
Johan Planchon
e0215e0cc8 add sourcehut engine 2022-08-10 21:48:12 +02:00
Johan Planchon
90bdb23756 add lib.rs engine 2022-08-10 12:17:55 +02:00
Thomas Renard
d4acbcfe63 [mod] add deepl translation engine 
This implements the Deepl Translation engine. It works nearly like lingva but
directly to the deepl API.  This api only needs a to-lang, from-lang is a fake
by now.

There is a free option to use [1].

[1] https://www.deepl.com/pro-api?cta=header-pro-api for registering a free account.
 2022-08-10 09:14:36 +02:00
Markus Heiser
925f3ff487
Merge pull request #1627 from allendema/add-marginalia 
[enh] Initial Marginalia.nu support (foss)
 2022-08-08 06:53:02 +02:00
Markus Heiser
58f630d312
Merge pull request #1623 from return42/mod-yep.com 
[mod] engine yep.com: show all 100 results yep.com has
 2022-08-08 06:52:15 +02:00
Allen
5b815b3966 [enh] initial Marginalia.nu support (foss) 
Currently it uses a public api_key `/public/` [1]

The 'index' parameter selects the search index, corresponding to the drop down
next to the search field in the main GUI.

    0: popular
    1: blogs
    2: big_sites
    3: default
    4: experimental

'experimental' is more up to date and does not exclude other sites, which is the
case with 'big sites' or 'blogs'.

[1] https://api.marginalia.nu/
[2] https://git.marginalia.nu/marginalia/marginalia.nu
[3] https://news.ycombinator.com/item?id=31536626

Closes: https://github.com/searxng/searxng/issues/1620
 2022-08-08 06:31:04 +02:00
Markus Heiser
8b58cd1167 [fix] typo in uWSGI cache: searxcache --> searxngcache 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-08-07 12:31:09 +02:00
ta
8aa018db95 add wttr.in engine 2022-08-07 13:04:18 +07:00
Markus Heiser
647e6187d7 [mod] engine yep.com: show all 100 results yep.com has 
yep.com is still in beta, the api.yep.com does not have paging support.  There
is only a 'limit' argument with a maximum of 100 results.

yep.com seems fast; there is nor need for a timeout of 12 sec.

The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.

FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.

Closes: https://github.com/searxng/searxng/issues/1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-08-06 13:21:28 +02:00
Allen
58369e41d6 [wip] brave time range support 2022-08-04 04:47:25 +02:00
Markus Heiser
8df1f0c47e [mod] add 'Accept-Language' HTTP header to online processores 
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).

- add new engine option: send_accept_language_header

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-08-01 17:01:59 +02:00
Markus Heiser
4231a5770b [fix] sjp engine - convert enginename to a latin1 compliance name 
The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-24 21:10:55 +02:00
Solirs
6d646129c3 [mod] add tor_check plugin - convenient tor checking trough searxng 2022-07-19 07:34:54 +02:00
Markus Heiser
4e05197444
Merge pull request #1475 from return42/Emojipedia 
[mod] Add engine for Emojipedia
 2022-07-15 09:30:40 +02:00
Jay
10edcbe3c2 [mod] Add engine for Emojipedia 
Emojipedia is an emoji reference website which documents the meaning and
common usage of emoji characters in the Unicode Standard.  It is owned by Zedge
since 2021. Emojipedia is a voting member of The Unicode Consortium.[1]

Cherry picked from @james-still [2[3] and slightly modified to fit SearXNG's
quality gates.

[1] https://en.wikipedia.org/wiki/Emojipedia
[2] 2fc01eb20f
[3] https://github.com/searx/searx/pull/3278
 2022-07-15 09:26:44 +02:00
Brock Vojković
84e2a3bd3f Add infinite scroll as a setting in settings.yml 2022-07-09 17:33:25 +00:00
Alexandre Flament
df837d8b1b
Merge pull request #1428 from return42/fix-center_aligment 
fix typo and document preference 'center_alignment' in the 'ui' section
 2022-07-07 09:43:12 +02:00
ta
14756a2674 [mod] Adds Lingva translate engine 
Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.
 2022-07-04 19:06:45 +02:00
Markus Heiser
241bde5ae3 [fix] typo: add missing 'n' in center_aligment --> center_alignment 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-03 17:35:54 +02:00
Alexandre Flament
f8f239fe1f Donation link: default value to searxng.org, can be hidden or custom 
Add a new setting: general.donation_url

By default the value is https://docs.searxng.org/donate.html

When the value is false, the link is hidden

When the value is true, the link goes to the infopage donation,
the administrator can create a custom page.
 2022-07-02 11:29:21 +02:00
Markus Heiser
ed1ff4b8bc
Merge pull request #1357 from liimee/eng-3 
Add curlie.org engine
 2022-06-22 18:29:36 +02:00
ta
7c457fcf7a add curlie engine 2022-06-22 11:46:33 +07:00
ta
8883aed132 [fix] google play apps engine: implement engines/google_play_apps.py 2022-06-18 16:02:39 +02:00
Alexandre Flament
8a85d1b0c7 Theme: add a preference to center the results (Oscar) (2nd edition) 
related to
* #1169
* #1281
* #1286
 2022-06-18 08:54:24 +02:00
Alexandre Flament
a7b0b2ecbf
Merge pull request #1329 from samsaptidev/feature/privacypolicy_url 
Add privacypolicy_url option
 2022-06-18 07:26:40 +02:00
Léon Tiekötter
6bdc66b70e
[fix] mojeek engine: XPath 
Update the XPath of the Mojeek engine.
 2022-06-17 09:32:53 +02:00
samsaptidev
31005595c9
Add privacypolicy_url option 2022-06-16 11:56:24 +02:00
ta
e9cc6ab0e7 add pub.dev engine 2022-06-15 12:55:37 +02:00
Allen
fd9a13a3e5 [enh] Initial no paging support for Yep.com 
Upstream example query:
https://yep.com/web?q=test

https://yep.com/about
 2022-06-11 14:17:44 +02:00
Maciej Błędkowski
98c4156f12
Fixed typo in settings.yml 2022-06-09 17:08:14 +02:00
Alexandre Flament
9723cef99a Revert "Theme: add a preference to center the results (Oscar)" 
This reverts commit 4237f5fd50.
 2022-06-05 23:03:26 +02:00
Alexandre Flament
4237f5fd50 Theme: add a preference to center the results (Oscar) 
related to #1169
 2022-06-05 19:59:49 +02:00
Allen
43dc9eb7d6 [enh] Initial Petalsearch Images support 
Upstream example query:

  https://petalsearch.com/search?query=test&channel=image&ps=50&pn=1&region=de-de&ss_mode=off&ss_type=normal

Depending on locale it will internally use some/all results from other
engines. See:

  https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#general-indexing-search-engines
 2022-06-02 14:32:37 +02:00
Léon Tiekötter
167f635924
[engine] petal search news 
Add Petal Search News to SearXNG via XPath
 2022-05-29 22:49:41 +02:00
Léon Tiekötter
f09e8abd76
[engine] petal search 
Add Petal Search to SearXNG via XPath
 2022-05-29 22:23:30 +02:00
Léon Tiekötter
7c8903c809
[fix] mojeek engine: xpath 2022-05-28 18:09:25 +02:00
Allen
13f4dcafc1
[fix] [new] Brave - Update xpath 
Last merge request was messed up with git.
 2022-05-26 18:44:11 +02:00
Alexandre Flament
c24864a9ad Enable duckduckgo by default 
see https://github.com/searxng/searxng/pull/1219#discussion_r879543174
 2022-05-24 23:33:12 +02:00
Alexandre Flament
cea7b71d14 Disable the bing engine 
see
* https://github.com/searxng/searxng/discussions/1209
* https://github.com/searxng/searxng/issues/941
* https://github.com/searxng/searxng/pull/1219
 2022-05-24 23:33:08 +02:00
Allen
6ef24ffd94
[enh] Add alexandria.org engine 
engine: json_engine
official_api_documentation: https://github.com/alexandria-org/alexandria-api/raw/master/README.md
 2022-05-15 11:10:15 +02:00
Markus Heiser
444b1e70db [mod] add setting: search.autocomplete_min 
Minimun characters to type before autocompleter starts.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-05-07 17:58:30 +02:00
Markus Heiser
e8541b6006 [theme] peel out oscar from SearXNG development 
This is the first step of removing oscar theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-04-30 13:20:27 +02:00
Léon Tiekötter
02026bcc76
[fix] library genesis engine: xpath and url 
Changed the xpath to use the official download mirror.

Changed the URL to libgen.fun as this is the official instance. (https://en.wikipedia.org/wiki/Library_Genesis)
 2022-04-27 13:11:34 +02:00
Allen
624b58764a
[enh] Add pagination to Brave 
Also added ```&spellcheck=1``` because now it is disabled by default, not returning any ```suggestion_xpath```.
 2022-04-23 14:06:53 +02:00
Allen
f039873e1a
[mod] Make a default domain for ebay 
Which can be changed.
 2022-04-17 10:37:12 +02:00
Vojtěch Fošnár
de4af2fefd [enh] add seznam autocomplete 2022-04-14 03:02:05 +02:00
Austin Huang
19fa0095a0
(fix) satisfy the linter, and btw reduce timeout 2022-04-01 09:23:24 -04:00
Austin Huang
934ae4e086
(feat) add jisho.org 
Closes #1016
 2022-03-31 14:45:39 -04:00
Alexandre Flament
dec04c0ed6
Merge pull request #999 from return42/fix-doc-settings2 
[docs] document settings.yml: search.languages
 2022-03-26 17:09:07 +01:00
Markus Heiser
a521d4985a [docs] document settings.yml: search.languages 
Requested-by: @dalf https://github.com/searxng/searxng/pull/996#discussion_r830858139
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-26 13:32:42 +01:00
Markus Heiser
7653e8bf16
Merge pull request #995 from return42/fix-932 
[interim fix] static files can't be delivered by HTTP.
 2022-03-25 06:56:11 +01:00
Markus Heiser
32a35b4654 [docs] improve settings.yml documentation (admin/engines/settings) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-21 08:48:06 +01:00
Markus Heiser
fd5fcdedce [interim fix] static files can't be delivered by HTTP. 
Since PR 932 [1][2] static files can't be delivered by HTTP server any longer.

This patch makes the hash paramter in the URL of static files:

    /static/themes/simple/css/searxng.min.css?5fde34a74bc438c7b56ec8c6501e131cc9914bd8

optional.  By default the hash parameter is disabled.

HINT:

  Instances that do not deliver static files by their HTTP server and have a
  long expire time [3] should enable this option.

----

This is only a interim solution, on the long run:

    make static.build.commit

creates files including the file name:

    css/searxng-5fde34a74bc438c7b56ec8c6501e131cc9914bd8.min.css

and a mapping.json with this content[4]

[1] https://github.com/searxng/searxng/issues/964
[2] https://github.com/searxng/searxng/pull/932#issuecomment-1067039518
[3] 5583336440
[4] https://github.com/searxng/searxng/pull/932#issuecomment-1067216426

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-20 16:28:14 +01:00
Markus Heiser
e4a2bfba38 [fix] settings.yml: ui.results_on_new_tab 
Closes: https://github.com/searxng/searxng/issues/987
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-19 23:06:21 +01:00
Markus Heiser
20f4538e13 [fix] engine: Semantic Scholar (Science) // rework & fix 
Closes: https://github.com/searxng/searxng/issues/939
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-05 11:53:41 +01:00
Alexandre Flament
8230603f48
Merge pull request #916 from dalf/pref_infinite_scroll2 
Convert the infinite_scroll plugin as a preference (second version)
 2022-02-21 12:49:04 +01:00
Alexandre Flament
56e34947a6 [mod] infinite_scroll as preference 
* oscar theme: code from searx/plugins/infinite_scroll.py
* simple theme: new implementation

Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
 2022-02-20 22:58:51 +01:00
Markus Heiser
5ec9eb8030 [mod] switch default theme from oscar to simple 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-20 18:36:41 +01:00
Markus Heiser
ae8b174e93 [fix] replace peer.tube by peertube.biz 
More peertube instances are listed at [1]

[1] https://instances.joinpeertube.org/instances

Closes: https://github.com/searxng/searxng/issues/881
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-13 22:48:09 +01:00
Markus Heiser
f2e16904a5 [mod] disable deezer engine by default 
To play content from deezer a account is needed, the majority of the SearXNG
won't have.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-07 22:58:05 +01:00
Markus Heiser
e2ec6b4211 [fix] invidious engine: store random base_url in param 
Two different threads ( = two different user queries) can call the request
function in a row and then the response function.  The namespace will be same
since this is the same engine.

To keep exactly the same value ``base_url`` must be stored in params and then
retrieve using ``resp.search_params["base_url"]``.

Suggested-by: @dalf https://github.com/searxng/searxng/pull/862#discussion_r799324861
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-04 15:42:06 +01:00
Markus Heiser
d92b3d96fd [fix] solidtorrents engine: JSON API no longer exists 
The API endpoint, we where using does not exist anymore.  This patch is a
rewrite that parses the HTML page.

Related: https://github.com/paulgoio/searxng/issues/17
Closes: https://github.com/searxng/searxng/issues/858

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-04 14:53:37 +01:00
Alexandre Flament
f79b0fce06 [enh] limiter plugin 
can replace filtron:
* rate limite the number of request per IP and per (IP, User-Agent)
* block some bots

use Redis
data stored in Redis never contains the IP addresses, only HMAC using the secret_key

Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-02 09:15:59 +01:00
Alexandre Flament
ebd3013a1a [mod] tineye engine: minor changes 
* remove "disable: false" in settings.yml
* use the json() method from httpx.Response (faster character encoding detection)
 2022-01-30 20:49:22 +01:00
Markus Heiser
a6b879f19c [mod] tineye engine: set engine_type to 'online_url_search' 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-01-30 16:30:52 +01:00
Allen
880555e263 [enh] engine - add Tineye reverse image search 
Other optional parameter ..

`&sort=crawl_date`
    can be appended to search_string to sort results by date.

`&domain=example.org`
    can be implemented to search_string to get results from just one domain.

Public instances could get relatively fast timed-out for 3600s.

--

Merged from @allendema's commit [1] and slightly modfied / see [2].

Related-to: [1] 455b2b4460
Related-to: [2] https://github.com/searx/searx/pull/3040
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-01-28 09:06:44 +01:00
Markus Heiser
ccffab3a43
Merge pull request #813 from MarcAbonce/change-wolfram-category 
Move WolframAlpha away from Science category
 2022-01-26 14:10:19 +01:00
Markus Heiser
10c9b9775d [mod] improve setup of invidious engine 
- My experience is, that a timeout of 5 sec is not need, I got fast response
  less than a second.

- https://invidious.tube/ redirects to http://ww25.invidious.tube/
  - in SearXNG defaults the http protocol is unsafe and raise an error
  - https://ww25.invidious.tube has SSL_ERROR_UNSAFE_NEGOTIATION

Related-to: https://github.com/searxng/searxng/issues/821
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-01-25 19:02:11 +01:00
Marc Abonce Seguin
1311745f24 move WolframAlpha away from Science category 2022-01-23 23:24:02 -06:00
Markus Heiser
977e9a4330
Merge pull request #686 from return42/lib_redis 
Add redis DB and connector
 2022-01-11 19:55:14 +01:00
Léon Tiekötter
37baf46ece [fix] Rename ccengine engine to openverse 
The CC engine was merged with WordPress and renamed to Openverse

Source: https://wordpress.org/news/2021/05/welcome-to-openverse/
 2022-01-07 13:06:05 +01:00