Commit graph

4831 commits

Author SHA1 Message Date
searxng-bot
96ccec1aa6 [translations] update from Weblate
acdcdace8 - 2023-11-01 - Ivan Gabaldon <admin@inetol.net>
31a1602a9 - 2023-10-29 - return42 <markus.heiser@darmarit.de>
a1d1070d3 - 2023-10-29 - return42 <markus.heiser@darmarit.de>
2023-11-03 10:01:28 +01:00
Markus Heiser
b5a8ddfec8
Revert "[fix] limit maximum page number of a search query to page 50." (#2979)
This reverts commit 7e2e335dd1.
2023-11-01 16:55:20 +00:00
Markus Heiser
d13a8f6453 [mod] document server:public_instance & remove it out of the botdetection
- the option server:public_instance lacks some documentation
- the processing of this option belongs in the limiter and not
  in botdetection module

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
Markus Heiser
fd814aac86 [mod] isolation of botdetection from the limiter
This patch was inspired by the discussion around PR-2882 [2].  The goals of this
patch are:

1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
   enabled (Redis is available).

This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.

Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin.  The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.

[1] https://github.com/searxng/searxng/pull/2882
[2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891

To test:

- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
  - http://0.0.0.0:8000/admin/searx.limiter.html
  - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
sev
b05a15540e Check public_instance in simple theme
Fix #2975
2023-10-31 10:40:19 +01:00
Markus Heiser
7e2e335dd1 [fix] limit maximum page number of a search query to page 50.
To test this PR run a local instance and try to query page 51:

    http://127.0.0.1:8888/search?q=foo&pageno=51

A parameter exception will be raised:

    searx.exceptions.SearxParameterException: Invalid value "51" for parameter pageno

And the client will receive a HTTP 400 (Bad request).

Closes https://github.com/searxng/searxng/issues/2972

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-29 16:17:25 +01:00
dalf
b63841f0a1 Update searx.data - update_engine_traits.py 2023-10-29 09:04:24 +01:00
dalf
a9d778ee0f Update searx.data - update_wikidata_units.py 2023-10-29 08:45:15 +01:00
dalf
92efb941d8 Update searx.data - update_firefox_version.py 2023-10-29 08:44:59 +01:00
dalf
fa494b9be3 Update searx.data - update_currencies.py 2023-10-29 08:44:35 +01:00
dalf
1b92ea4e70 Update searx.data - update_engine_descriptions.py 2023-10-29 08:44:17 +01:00
dalf
55e7cb3e33 Update searx.data - update_ahmia_blacklist.py 2023-10-29 08:17:29 +01:00
searxng-bot
6af8644ead [translations] update from Weblate
4e5e5db44 - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
e1a8d3508 - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
84bddfb89 - 2023-10-26 - return42 <markus.heiser@darmarit.de>
d67a4114d - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
62fe8e328 - 2023-10-26 - return42 <markus.heiser@darmarit.de>
6e37ab975 - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
2cdab3247 - 2023-10-25 - SomeTr <SomeTr@users.noreply.translate.codeberg.org>
cf7ea7234 - 2023-10-23 - clsty <celestial.y@outlook.com>
0ea313893 - 2023-10-23 - return42 <markus.heiser@darmarit.de>
22151e440 - 2023-10-23 - return42 <markus.heiser@darmarit.de>
e4eaf42b6 - 2023-10-22 - clsty <celestial.y@outlook.com>
2023-10-27 09:23:23 +02:00
Markus Heiser
ef56e1d684 [fix] HTMLParser: undocumented not implemented method
In python versions <py3.10 there is an issue with an undocumented method
HTMLParser.error() [1][2] that was deprecated in Python 3.4 and removed
in Python 3.5.

To be compatible to higher versions (>=py3.10) an error method is implemented
which throws an AssertionError exception like the higher Python versions do [3].

[1] https://github.com/python/cpython/issues/76025
[2] https://bugs.python.org/issue31844
[3] https://github.com/python/cpython/pull/8562

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-22 10:35:02 +02:00
searxng-bot
01b5b9cb8e [translations] update from Weblate
2325f1583 - 2023-10-20 - return42 <markus.heiser@darmarit.de>
5090c6a8e - 2023-10-18 - return42 <markus.heiser@darmarit.de>
3a38219d8 - 2023-10-18 - return42 <markus.heiser@darmarit.de>
94a9f4164 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
bdbeb4b30 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
f9b483f48 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
7f1ca1997 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
c5a701dc4 - 2023-10-14 - alexgabi <alexgabi@disroot.org>
2023-10-20 10:44:11 +02:00
Markus Heiser
b013cbb4cc [build] /static 2023-10-13 12:56:29 +02:00
Bnyro
df2e41c150 [fix] search.js: crash on homepage when setting form listeners 2023-10-13 12:56:29 +02:00
Émilien (perso)
4280318fc5 fixing results parsing brave 2023-10-13 11:47:30 +02:00
searxng-bot
ac82daf58d [translations] update from Weblate
74e401e68 - 2023-10-09 - return42 <markus.heiser@darmarit.de>
897dd8db1 - 2023-10-09 - return42 <markus.heiser@darmarit.de>
6ed046a90 - 2023-10-09 - tentsbet <remendne@pentrens.jp>
815ecb336 - 2023-10-09 - return42 <markus.heiser@darmarit.de>
65d9a0c2f - 2023-10-07 - return42 <markus.heiser@darmarit.de>
3ec249ef9 - 2023-10-07 - return42 <markus.heiser@darmarit.de>
53dc6c108 - 2023-10-07 - return42 <markus.heiser@darmarit.de>
2023-10-13 09:20:54 +02:00
Hackurei
efada7cba2 [fix] hackernews keyerror problem 2023-10-13 08:16:47 +02:00
Hackurei
af071121de [fix] imgur - incorrect wikidata id 2023-10-12 09:14:00 +02:00
Markus Heiser
14323d683f [fix] ddg-lite & ddg-extra: don't send empty vqd value
DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-12 08:52:28 +02:00
Markus Heiser
3388441917 [fix] ddg-lite vqd value: some search terms do not have a vqd value
Some search terms do not have results and therefore no vqd value

BTW: remove a leftover from 9197efa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10 09:12:30 +02:00
Markus Heiser
9197efa2a7 [fix] duckduckgo lite engine: set HTTP header 'Referer'
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].

From reverse engineering:

- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
  does not like this referer anymore!

- The 'Referer' header is only set on second and follow up pages but not on the
  first page

- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.

Related to 'Referer' header & ddg requests:

[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081

Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10 08:40:53 +02:00
Bnyro
fa5b2a7948 [mod] yacy: use official instance by default and fix crashes 2023-10-09 20:50:24 +02:00
Alex Balgavy
5d53aac20f [mod] add hotkeys option to settings.yml
The change in the hotkey mechanism introduced in 317db5b04 does not allow
configuration via `settings.yml`.  This commit adds that functionality.

Closes: #2898
2023-10-09 18:13:00 +02:00
Hackurei
ff78b1a902 [feat] implement hackernews engine - news.ycombinator.com 2023-10-09 14:00:04 +02:00
Aine
213cb74378 [fix] matrixrooms add proper MRS integration
Related:

- https://github.com/searxng/searxng/issues/2918
2023-10-09 13:25:13 +02:00
Bnyro
48cb58bd2e [feat] duckduckgo: support for videos and news 2023-10-09 06:53:43 +02:00
Bnyro
c3ab49cd90 [fix] kickass: crash when no results 2023-10-07 11:48:23 +02:00
Bnyro
f22daf8b47 [mod] piped: always show video length if available 2023-10-07 11:45:46 +02:00
Bnyro
ce270961e8 [feat] engine: implementation of mastodon 2023-10-06 10:58:23 +02:00
searxng-bot
48f8ac48d1 [translations] update from Weblate
68d743281 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
42f091b7f - 2023-10-05 - return42 <markus.heiser@darmarit.de>
2479c0d7b - 2023-10-05 - ghose <correo@xmgz.eu>
a4e6cd592 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
9d4e5f5c3 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
b79d44775 - 2023-10-05 - gallegonovato <fran-carro@hotmail.es>
746291184 - 2023-10-06 - return42 <markus.heiser@darmarit.de>
f24d7e8b1 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
6140911f9 - 2023-10-05 - Fjuro <ifjuro@proton.me>
2023-10-06 09:19:00 +02:00
Markus Heiser
fd1422a670 [mod] engine - simplify region & lang handling, make filters configurable
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-05 10:55:08 +02:00
Bnyro
3e2ae756f0 [feat] engine: implementation of radio-browser.info 2023-10-05 10:55:08 +02:00
Markus Heiser
d56b31d674 [fix] limiter / botdetection: remove http_connection method
Related:

- https://github.com/searxng/searxng/issues/2892#issuecomment-1742153932

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 20:24:22 +02:00
Jinyuan Huang
e509cb7c45 [typo] solved a typo in yahoo error message. 2023-10-01 08:29:06 +02:00
Jinyuan Huang
d4d9f2073e [fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866
Co-authored-by: Blair Noctis <n@sail.ng>
2023-10-01 08:29:06 +02:00
Bnyro
fe9386b58d [fix] emojipedia: fix engine 2023-10-01 08:19:45 +02:00
Markus Heiser
32a4ea350e [fix] Revision of the Bing engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 08:01:38 +02:00
jazzzooo
079636c079 [fix] engine - bing fix search, pagination, remove safesearch 2023-10-01 08:01:38 +02:00
Bnyro
5ce1792432 [feat] engine: implementation of pinterest 2023-09-30 15:01:45 +02:00
Bnyro
6096457e4d [fix] matrixrooms.info: pagination not working properly 2023-09-30 14:51:07 +02:00
Markus Heiser
e1a8b8189f [fix] engine - moviepilot instead of thumbnail use img_src
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.

Related:

- b4e0d2eedc (r128785388)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-30 11:29:19 +02:00
Bnyro
159629c588 [mod] tagesschau: add option to only use tagesschau urls 2023-09-30 11:00:11 +02:00
Bnyro
2ca60a19fc [feat] engine: implementation of matrixrooms.info 2023-09-30 09:09:23 +02:00
Bnyro
d1a3fc5be6 [feat] engine: implementation of tootfinder 2023-09-29 23:00:30 +02:00
Bnyro
fc4a20f734 [mod] add movies category for tmdb, imdb and moviepilot 2023-09-29 22:37:51 +02:00
Émilien (perso)
0330648b98 [mod] warn to info for failed ping limiter 2023-09-29 22:20:56 +02:00
Bnyro
26fed56d51 [mod] settings.yml: remove plugin settings for plugins that don't exist anymore 2023-09-29 11:26:49 +02:00
searxng-bot
9a065832e0 [translations] update from Weblate
80bbf74ef - 2023-09-27 - return42 <markus.heiser@darmarit.de>
5d554ef71 - 2023-09-27 - BernieHuang2008 <berniehuang2008@163.com>
771fd1b90 - 2023-09-27 - return42 <markus.heiser@darmarit.de>
d412012c8 - 2023-09-27 - y0nei <y0nei@proton.me>
1ac171d0a - 2023-09-27 - return42 <markus.heiser@darmarit.de>
a004c40cc - 2023-09-27 - return42 <markus.heiser@darmarit.de>
47e729f89 - 2023-09-27 - return42 <markus.heiser@darmarit.de>
f85b28cc4 - 2023-09-26 - Fjuro <ifjuro@proton.me>
75ce5d581 - 2023-09-27 - return42 <markus.heiser@darmarit.de>
40ce06009 - 2023-09-24 - nonepork <ptohrek@gmail.com>
21f343f71 - 2023-09-25 - tentsbet <remendne@pentrens.jp>
4c80f80fd - 2023-09-22 - return42 <markus.heiser@darmarit.de>
ef3c859e4 - 2023-09-22 - artnay <jiri.gronroos@iki.fi>
2023-09-29 09:12:12 +02:00
dalf
ae2762eb78 Update searx.data - update_firefox_version.py 2023-09-29 06:58:11 +02:00
dalf
18da3d65c5 Update searx.data - update_ahmia_blacklist.py 2023-09-29 06:57:40 +02:00
dalf
4a66fa3e7d Update searx.data - update_wikidata_units.py 2023-09-29 06:56:50 +02:00
dalf
9be6e20256 Update searx.data - update_engine_traits.py 2023-09-29 06:56:10 +02:00
dalf
d821ec44dd Update searx.data - update_currencies.py 2023-09-29 06:55:04 +02:00
dalf
cdee5aa3b9 Update searx.data - update_engine_descriptions.py 2023-09-29 06:53:05 +02:00
Bnyro
75587d03a6 [fix] categories.html: can't select social media category using search on category select 2023-09-28 10:45:49 +02:00
jazzzooo
e37d775fa2 [fix] engine - currency fix and simplify 2023-09-28 08:29:38 +02:00
Jinyuan Huang
ae28d429c9 [fix] bilibili new api used 2023-09-28 08:24:51 +02:00
jazzzooo
1a66d74673 [fix] engine - kickass update url, fix parsing, use multiple mirrors 2023-09-27 10:19:41 +02:00
Markus Heiser
0c39064d60 Update searx.data - update_engine_traits.py
Update searx.data - update_engine_traits.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Markus Heiser
b428ccc5a0 [fix] engine brave - fetch traits (modified settings menu)
Brave has changed it settings menu fundamental.  Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:

  https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Markus Heiser
3a456b1282 [fix] engine annas archive - fetch traits (modified xpath selectors)
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Jinyuan Huang
1f835a9f1f solve #2839 2023-09-25 23:01:14 +02:00
jazzzooo
3c6549a17f [fix] update user agent 2023-09-25 22:46:22 +02:00
Emilien Devos
ffec8d1b4c add warning about enabling public_instance setting 2023-09-25 22:31:14 +02:00
Emilien Devos
47721a3485 add new parameter called server.public_instance
for enabling by default advanced limiter functions
in the future allow us to add features just for the public instances
2023-09-25 22:31:14 +02:00
Jinyuan Huang
1df4588279 [build] /static 2023-09-25 17:45:21 +02:00
Jinyuan Huang
98366bcf90 [fix] settings icon invisble on mobile devices 2023-09-25 17:45:21 +02:00
Alexandre Flament
19383921d7 /config: expose limiter configuration 2023-09-24 11:01:11 +02:00
Émilien (perso)
1851f27154
[mod] remove twitter (#2843) 2023-09-24 08:32:53 +00:00
Markus Heiser
597c68b4aa [doc] move dosc of botdetection from developer to admin section
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-23 14:00:03 +02:00
Bnyro
75c9de02d1 [feat] engine: implementation of imgur 2023-09-22 20:50:53 +02:00
Bnyro
fb72f71f0a [fix] internet archive scholar: crash when there's no title 2023-09-22 18:49:39 +02:00
searxng-bot
0660d20e1d [translations] update from Weblate
f27a3bbd4 - 2023-09-21 - return42 <markus.heiser@darmarit.de>
a97336d0a - 2023-09-21 - return42 <markus.heiser@darmarit.de>
91e8946b3 - 2023-09-19 - return42 <markus.heiser@darmarit.de>
19f53731e - 2023-09-20 - ghose <correo@xmgz.eu>
daf10bd75 - 2023-09-19 - gallegonovato <fran-carro@hotmail.es>
ab004b8b8 - 2023-09-19 - return42 <markus.heiser@darmarit.de>
cdedd99c9 - 2023-09-19 - return42 <markus.heiser@darmarit.de>
2023-09-22 18:05:02 +02:00
Bnyro
95885e7f9f [build] /static 2023-09-22 17:54:55 +02:00
Bnyro
8b0b6731fc [fix] search.less: start page search view on mobile devices 2023-09-22 17:54:55 +02:00
Markus Heiser
71358e9c67 Revert "[fix] engine - duckduckgo vqd edge-case"
This reverts commit 102502a4f0.
2023-09-22 09:31:25 +02:00
Bnyro
51236ae47a [feat] engine: implementation of chefkoch.de 2023-09-21 17:23:59 +02:00
jazzzooo
8bcca0e620 [fix] engine - brave don't show ads 2023-09-21 16:55:39 +02:00
jazzzooo
b729542a66 [fix] engine - google images error when no results 2023-09-21 16:38:37 +02:00
Bnyro
f0df53673e [mod] search-syntax.md: add tutorial about the feeling lucky feature 2023-09-21 15:17:23 +02:00
Bnyro
cc2e0537a3 [feat] engine: implementation of google icons/material design icons 2023-09-21 15:16:49 +02:00
Bnyro
c999cfb422 [feat] engine: implementation of wallhaven 2023-09-21 14:25:43 +02:00
jazzzooo
102502a4f0 [fix] engine - duckduckgo vqd edge-case 2023-09-20 20:05:06 +02:00
Markus Heiser
d013f51a25 [fix] Search on category select without JavaScript PR #2740
Small addendum to #2740; search_on_category_select is now no longer a plugin.

Related: https://github.com/searxng/searxng/pull/2740#issuecomment-1725437584
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 21:03:02 +02:00
Markus Heiser
0ce10046c4 [build] /static 2023-09-19 20:15:51 +02:00
ormai
a76c8e4101 Improve the layout of the search page on mobile 2023-09-19 20:15:51 +02:00
Émilien (perso)
33496e7f51 Switch back to qwant web normal API 2023-09-19 17:07:22 +02:00
Markus Heiser
043dcbf7c5 [fix] engine qwant (web-lite) - ignore advertising adds
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 17:06:56 +02:00
Markus Heiser
3ac7c40b6a [build] /static 2023-09-19 11:15:25 +02:00
Emilien Devos
33e722f83b better error message when no results found 2023-09-19 11:10:48 +02:00
Émilien (perso)
ad725ce7d7
wikipedia wikidata infobox + disable wikisource (#2806)
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 10:31:02 +02:00
Bnyro
dcee823345 [feat] implement feeling lucky feature 2023-09-19 09:40:57 +02:00
Markus Heiser
71508abcbf [build] /static 2023-09-19 08:06:06 +02:00
Jinyuan Huang
2ec77ef813 [mod] reformatted keyBindingLayouts in JS client (default, vim) 2023-09-19 08:06:06 +02:00
Markus Heiser
317db5b04f [mod] preferences: implement drop-down menu for hotkeys (default, vim)
Replace the on/off checkbox of the vim-hotkeys in the preferences by a drop-down
menu.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 08:06:06 +02:00
Markus Heiser
09935e2897 [build] /static 2023-09-18 21:29:11 +02:00
Bnyro
a55e0ac553 [feat] search on category select without JS
Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-09-18 21:29:11 +02:00