Commit graph

795 commits

Author SHA1 Message Date
Markus Heiser
007a615ffa [mod] donation_url: disable by default
SearXNG's donation campaign has been ended.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-15 09:19:17 +02:00
Markus Heiser
45529f51a1
Merge pull request #2347 from return42/mod-lang-detection
If language recognition fails use the Accept-Language
2023-04-25 15:46:26 +02:00
Markus Heiser
f1b6351ae1 [fix] engine: google play movies
Closes: https://github.com/searxng/searxng/pull/1746
Closes: https://github.com/searxng/searxng/issues/1599

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-16 19:15:44 +02:00
Markus Heiser
8adbc4fcec [mod] settings.yml: enable language detection by default_lang (auto)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 22:24:59 +02:00
Markus Heiser
5234e45010 [fix] Gigablast.com has been erased
[1] https://www.reddit.com/r/searchengines/comments/128wdcp/gigablastcom_has_been_erased/

Closes: https://github.com/searxng/searxng/issues/2321
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-06 08:22:57 +02:00
Markus Heiser
2b8dfab33f [fix] engine gigablast: add &userid=<User ID>&code=<Feed Code>
Gigablast's API does block unauthorized request[1].

[1] https://gigablast.com/searchfeed.html

Closes: https://github.com/searxng/searxng/issues/1454
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-29 16:18:02 +02:00
Markus Heiser
2499899554 [mod] Google: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.

When ever possible the implementations of the Google engines try to make use of
the async REST APIs.  The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - Google (WEB),
  - Google images,
  - Google news,
  - Google scholar and
  - Google videos

  and remove data from obsolete data type "supported_languages".

  A traits.custom type that maps region codes to *supported_domains* is fetched
  from https://www.google.com/supported_domains

searx/autocomplete.py:
  Reversed engineered autocomplete from Google WEB.  Supports Google's languages and
  subdomains.  The old API suggestqueries.google.com/complete has been replaced
  by the async REST API: https://{subdomain}/complete/search?{args}

searx/engines/google.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - always use the async REST API (formally known as 'use_mobile_ui')
  - use *supported_domains* from traits
  - improved the result list by fetching './/div[@data-content-feature]'
    and parsing the type of the various *content features* --> thumbnails are
    added

searx/engines/google_images.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - if exists, freshness_date is added to the result
  - issue 1864: result list has been improved a lot (due to the new cr parameter)

searx/engines/google_news.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
    *supported_domains* is not needed but a ceid list has been added.
  - different region handling compared to Google WEB
  - fixed for various languages & regions (due to the new ceid parameter) /
    avoid CONSENT page
  - Google News do no longer support time range
  - result list has been fixed: XPath of pub_date and pub_origin

searx/engines/google_videos.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - add paging support
  - implement a async request ('asearch': 'arc' & 'async':
    'use_ac:true,_fmt:html')
  - simplified code (thanks to '_fmt:html' request)
  - issue 1359: fixed xpath of video length data

searx/engines/google_scholar.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - request(): include patents & citations
  - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
  - hardening XPath to iterate over results
  - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
  - issue 1769 fixed: new request implementation is no longer incompatible

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
a7fe22770a [mod] Peertube: re-engineered & upgrade to data_type: traits_v1
- fetch_traits(): Fetch languages from peertube's search-index source code.

  [mod] Include migration of the request methode from 'supported_languages'
        to 'traits' (EngineTraits) object.
  [fix] old supported_languages_url is no longer valid since the sources
        has been moved to a different path.

- fixed code to pass pylint
- request(): complete re-implementation based on the API docs [1]
- response(): complete re-implementation, adds serveral fields missed before
- add source code documentation

[1] https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Solirs
35fbb3578b Increase timeout for gentoo wiki engine 2023-02-28 13:54:44 +01:00
Markus Heiser
5820dc78ce [doc] slight improvements to the doc of the settings (base_url)
Closes: https://github.com/searxng/searxng/issues/2190

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17 12:08:58 +01:00
Markus Heiser
52f6bc745b
Merge pull request #2188 from ahmad-alkadri/fix/petalsearch
Fix the petalsearch engine
2023-02-15 13:57:28 +01:00
Ahmad Alkadri
f6af59899b Fix petalsearch and remove petalsearch news 2023-02-14 18:43:55 +01:00
Markus Heiser
7d446dfdb2 [mod] disbale engine tineye by default
Tineye becomes active as soon as a https:// signature is found in the search
term, but most of the time a reverse image search is not requested when a URL is
specified, often the URL is just from a C&P.

The frequent requests to tineye lead in the end to the SearXNG instance being
blocked by tineye and the user seeing unexpected error messages.

BTW: many maintainers have disabled this engine in their local SearXNG settings.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-14 08:27:19 +01:00
Markus Heiser
3abff182ea [fix] remove engine neeva from settings.yml
Engine is broken and can't by used any longer as a simple XPath engine.
@allendema tested a engines/neeva.py version using json from the dom, but
without luck: There was some kind of captcha for pagination.

[1] https://github.com/searxng/searxng/issues/2007#issuecomment-1426061698

Closes: https://github.com/searxng/searxng/issues/2007
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 18:46:37 +01:00
wibyweb
6707354bc8 [mod] engine wiby: add pagination
Suggested by: @wibyweb in searx https://github.com/searx/searx/pull/3465

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 15:31:24 +01:00
Alexandre Flament
37addec69e search.suspended_time settings: bug fixes
* fix type in settings.yml: replace suspend_times by suspended_times
* always use delay defined in settings.yml:
  * HTTP status 402 and 403: read the value from settings.yml instead of using the hardcoded value of 1 day.
  * startpage engine: CAPTCHA suspend the engine for one day instead of one week
2023-01-28 10:24:14 +00:00
Alexandre Flament
13b0c251c4
Merge pull request #2100 from nexryai/master
Add goo engine
2023-01-15 23:08:28 +01:00
Léon Tiekötter
0cedb1c6d8 Add search.suspended_times settings
Make suspended_time changeable in settings.yml
Allow different values to be set for different exceptions.

Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-01-15 09:00:32 +00:00
nexryai
4e7bb1bf9a
Add goo engine 2023-01-12 16:28:09 +09:00
Milad-Laly
cf4db4be37 [fix] Mojeek Xpath showing suggestions and searches + add lang support 2023-01-09 09:33:47 +01:00
Markus Heiser
ed901ab18e [mod] improve 'Autodetect search language' plugin
- Add documentation to the plugin
- Harmonize FastText language model with SearXNG's language model

Reosurces::

    import fasttext                                    # --> +10 MB
    fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB

Suggested-by: @dalf

- To speed up and simplify the deployment use fasttext-wheel instead of fasttext
- Building numpy on the Alpine Linux of docker-images takes ages --> install
  py3-numpy from Alpines package manager (apk)
- Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu
  libc) --> patch Dockerfile and build from fastetxt:

     sed -i s/fasttext-wheel/fasttext/ requirements.txt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-11 11:26:07 +01:00
ArtikusHG
9925a20950 [mod] new plugin: Autodetect search language 2022-12-10 13:11:47 +01:00
Ryan Draga
408200c87e [fix] disabling zlibrary due to z-lib.org domain seizure 2022-11-10 21:18:21 +01:00
Alexandre Flament
8f19bdaf17
Merge pull request #1882 from fehho/metacpan
Add MetaCPAN engine
2022-11-07 21:54:11 +01:00
fehho
fe351c2802 Add MetaCPAN engine 2022-11-07 08:07:06 -06:00
Alexandre FLAMENT
e92755d358 Initialize Redis in searx/webapp.py
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"

The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.

It worked before because SearXNG initialized the Redis connection when the limiter started.

In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
2022-11-05 17:45:52 +01:00
Alexandre Flament
32e8c2cf09 searx.network: add "verify" option to the networks
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.

SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
2022-10-14 13:59:22 +00:00
Mohamed Elashri
8d5653e60d
Merge branch 'searxng:master' into master 2022-09-30 23:06:54 +00:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
Mohamed Elashri
5832c70680
correct sci-hub links/ add .ru and remove other 3rd party domains. 2022-09-24 11:03:57 -04:00
Markus Heiser
caebafdd06 [fix] typo in crossref settings: disable --> disabled
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-24 08:12:36 +02:00
Alexandre Flament
d6446be38f [mod] science category: various update of about PR 1705 2022-09-23 20:52:55 +02:00
Alexandre FLAMENT
e36f85b836 Science category: update the engines
* use the paper.html template
* fetch more data from the engines
* add crossref.py
2022-09-23 20:45:58 +02:00
Alexandre Flament
bef3984d03
Merge pull request #1728 from liimee/eng-ddw
add duckduckgo weather engine
2022-09-23 18:14:09 +02:00
Alexandre Flament
d3fec1388c
Merge pull request #1624 from liimee/eng-wttr
Add wttr.in engine
2022-09-23 18:13:37 +02:00
Alexandre FLAMENT
33b43763b9 Brave engine: fix BrotliDecoderDecompressStream error 2022-09-18 22:08:38 +00:00
LencoDigitexer
3f72a79088 add yandex to autocomplete backends settings 2022-09-09 23:50:58 +03:00
Markus Heiser
ad8ffd222c [mod] option 'ui: cache_url:' to configure internet cache or archive
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-04 09:42:20 +02:00
Alexandre FLAMENT
dd0887be18 xpath engine: change raise_for_httperror to no_result_for_http_status
no_result_for_http_status contains a list of HTTP status.
These HTTP status are seen an empty result list.
In other cases an exception is thrown as usual.

Previously raise_for_httperror were ignoring all HTTP error,
which make defective engines invisible in the stats.
2022-09-04 09:07:28 +02:00
Markus Heiser
a15dfa5ee1 [fix] engine woxikon.de - don't raise exception on empty result list
Woxikon expects a word in German, so with query "foo" the site finds nothing and
respons a 404:

    httpx.HTTPStatusError: Client error '404 Not Found' \
      for url 'https://synonyme.woxikon.de/synonyme/foo.php'

[1] https://github.com/searxng/searxng/issues/1543#issuecomment-1193317054

Closes: https://github.com/searxng/searxng/issues/1543
Suggested-by: @allendema [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-04 09:07:28 +02:00
Markus Heiser
8e9fb0b435
Merge pull request #1647 from return42/deepl-engine
[mod] add deepl translation engine
2022-09-02 14:09:22 +02:00
Émilien Devos
fcccf39030
Disable brave by default
Brave is too unstable and will often not work by default. As seen in many issues: https://github.com/searxng/searxng/issues?q=is%3Aissue++sort%3Aupdated-desc+brave+label%3Abug+
2022-08-31 15:47:56 +02:00
ta
12f7d4a46b add duckduckgo weather engine 2022-08-31 17:29:32 +07:00
Alexandre FLAMENT
341ad46303 settings.yml: set default values for result_proxy
* initialize result_proxy with searx/settings_defaults.py
* allow result_proxy.key to be a string

this commit supersedes #1522
2022-08-28 09:27:53 +00:00
Alexandre Flament
56000d5162
Merge pull request #1699 from liimee/eng-app-store
add apple app store engine
2022-08-27 07:43:23 +02:00
Alexandre Flament
44bc94c36e
Merge pull request #1700 from liimee/eng-ddm
add apple maps engine
2022-08-27 07:36:16 +02:00
ta
5dce299b22 add apple maps engine 2022-08-25 17:05:40 +07:00
ta
e5c1b64b1d add the apple app store engine
The Apple App Store is the digital app distribution platform for iOS & iPadOS.
2022-08-24 17:27:36 +07:00
ta
dd9127492f add 9gag engine
9GAG is a social media website where users upload and share user-generated images and videos
2022-08-22 17:35:07 +07:00
Alexandre Flament
5ed40af3ba
Merge pull request #1661 from liimee/eng-tw
Add twitter engine
2022-08-21 15:21:18 +02:00
Markus Heiser
75bb8c45d0 [mod] decouple qwant's categories from SearXNG's categories
By using new property `qwant_categ:` the category of qwant is no longer bound to
the category of SearXNG.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:26:54 +02:00
ta
96ea355a1f add twitter engine 2022-08-14 08:39:41 +07:00
Léon Tiekötter
8fa84ee9f4 Fix time range support and add soft_max_redirects
Add custom time_range_url and time_range_map

Set soft_max_redirects = 2 to prevent "ErrorContext('searx/search/processors/online.py', 116, 'count_error(', None, '2 redirects, maximum: 0', ('200', 'OK', 'neeva.com')) True"
2022-08-13 07:35:15 +07:00
ta
d6dfada1a9 add neeva web engine
Neeva is "the world's first ad-free, private search engine" and uses data from Apple, Bing, Yelp and "others".
They claim to crawl "hundreds of millions" of URLs a day (https://twitter.com/Neeva/status/1536447373903335426).
2022-08-13 07:26:36 +07:00
Johan Planchon
e0215e0cc8 add sourcehut engine 2022-08-10 21:48:12 +02:00
Johan Planchon
90bdb23756 add lib.rs engine 2022-08-10 12:17:55 +02:00
Thomas Renard
d4acbcfe63 [mod] add deepl translation engine
This implements the Deepl Translation engine. It works nearly like lingva but
directly to the deepl API.  This api only needs a to-lang, from-lang is a fake
by now.

There is a free option to use [1].

[1] https://www.deepl.com/pro-api?cta=header-pro-api for registering a free account.
2022-08-10 09:14:36 +02:00
Markus Heiser
925f3ff487
Merge pull request #1627 from allendema/add-marginalia
[enh] Initial Marginalia.nu support (foss)
2022-08-08 06:53:02 +02:00
Markus Heiser
58f630d312
Merge pull request #1623 from return42/mod-yep.com
[mod] engine yep.com: show all 100 results yep.com has
2022-08-08 06:52:15 +02:00
Allen
5b815b3966 [enh] initial Marginalia.nu support (foss)
Currently it uses a public api_key `/public/` [1]

The 'index' parameter selects the search index, corresponding to the drop down
next to the search field in the main GUI.

    0: popular
    1: blogs
    2: big_sites
    3: default
    4: experimental

'experimental' is more up to date and does not exclude other sites, which is the
case with 'big sites' or 'blogs'.

[1] https://api.marginalia.nu/
[2] https://git.marginalia.nu/marginalia/marginalia.nu
[3] https://news.ycombinator.com/item?id=31536626

Closes: https://github.com/searxng/searxng/issues/1620
2022-08-08 06:31:04 +02:00
Markus Heiser
8b58cd1167 [fix] typo in uWSGI cache: searxcache --> searxngcache
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-07 12:31:09 +02:00
ta
8aa018db95 add wttr.in engine 2022-08-07 13:04:18 +07:00
Markus Heiser
647e6187d7 [mod] engine yep.com: show all 100 results yep.com has
yep.com is still in beta, the api.yep.com does not have paging support.  There
is only a 'limit' argument with a maximum of 100 results.

yep.com seems fast; there is nor need for a timeout of 12 sec.

The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.

FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.

Closes: https://github.com/searxng/searxng/issues/1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-06 13:21:28 +02:00
Allen
58369e41d6 [wip] brave time range support 2022-08-04 04:47:25 +02:00
Markus Heiser
8df1f0c47e [mod] add 'Accept-Language' HTTP header to online processores
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).

- add new engine option: send_accept_language_header

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 17:01:59 +02:00
Markus Heiser
4231a5770b [fix] sjp engine - convert enginename to a latin1 compliance name
The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-24 21:10:55 +02:00
Solirs
6d646129c3 [mod] add tor_check plugin - convenient tor checking trough searxng 2022-07-19 07:34:54 +02:00
Markus Heiser
4e05197444
Merge pull request #1475 from return42/Emojipedia
[mod] Add engine for Emojipedia
2022-07-15 09:30:40 +02:00
Jay
10edcbe3c2 [mod] Add engine for Emojipedia
Emojipedia is an emoji reference website which documents the meaning and
common usage of emoji characters in the Unicode Standard.  It is owned by Zedge
since 2021. Emojipedia is a voting member of The Unicode Consortium.[1]

Cherry picked from @james-still [2[3] and slightly modified to fit SearXNG's
quality gates.

[1] https://en.wikipedia.org/wiki/Emojipedia
[2] 2fc01eb20f
[3] https://github.com/searx/searx/pull/3278
2022-07-15 09:26:44 +02:00
Brock Vojković
84e2a3bd3f Add infinite scroll as a setting in settings.yml 2022-07-09 17:33:25 +00:00
Alexandre Flament
df837d8b1b
Merge pull request #1428 from return42/fix-center_aligment
fix typo and document preference 'center_alignment' in the 'ui' section
2022-07-07 09:43:12 +02:00
ta
14756a2674 [mod] Adds Lingva translate engine
Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.
2022-07-04 19:06:45 +02:00
Markus Heiser
241bde5ae3 [fix] typo: add missing 'n' in center_aligment --> center_alignment
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-03 17:35:54 +02:00
Alexandre Flament
f8f239fe1f Donation link: default value to searxng.org, can be hidden or custom
Add a new setting: general.donation_url

By default the value is https://docs.searxng.org/donate.html

When the value is false, the link is hidden

When the value is true, the link goes to the infopage donation,
the administrator can create a custom page.
2022-07-02 11:29:21 +02:00
Markus Heiser
ed1ff4b8bc
Merge pull request #1357 from liimee/eng-3
Add curlie.org engine
2022-06-22 18:29:36 +02:00
ta
7c457fcf7a add curlie engine 2022-06-22 11:46:33 +07:00
ta
8883aed132 [fix] google play apps engine: implement engines/google_play_apps.py 2022-06-18 16:02:39 +02:00
Alexandre Flament
8a85d1b0c7 Theme: add a preference to center the results (Oscar) (2nd edition)
related to
* #1169
* #1281
* #1286
2022-06-18 08:54:24 +02:00
Alexandre Flament
a7b0b2ecbf
Merge pull request #1329 from samsaptidev/feature/privacypolicy_url
Add privacypolicy_url option
2022-06-18 07:26:40 +02:00
Léon Tiekötter
6bdc66b70e
[fix] mojeek engine: XPath
Update the XPath of the Mojeek engine.
2022-06-17 09:32:53 +02:00
samsaptidev
31005595c9
Add privacypolicy_url option 2022-06-16 11:56:24 +02:00
ta
e9cc6ab0e7 add pub.dev engine 2022-06-15 12:55:37 +02:00
Allen
fd9a13a3e5 [enh] Initial no paging support for Yep.com
Upstream example query:
https://yep.com/web?q=test

https://yep.com/about
2022-06-11 14:17:44 +02:00
Maciej Błędkowski
98c4156f12
Fixed typo in settings.yml 2022-06-09 17:08:14 +02:00
Alexandre Flament
9723cef99a Revert "Theme: add a preference to center the results (Oscar)"
This reverts commit 4237f5fd50.
2022-06-05 23:03:26 +02:00
Alexandre Flament
4237f5fd50 Theme: add a preference to center the results (Oscar)
related to #1169
2022-06-05 19:59:49 +02:00
Allen
43dc9eb7d6 [enh] Initial Petalsearch Images support
Upstream example query:

  https://petalsearch.com/search?query=test&channel=image&ps=50&pn=1&region=de-de&ss_mode=off&ss_type=normal

Depending on locale it will internally use some/all results from other
engines. See:

  https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#general-indexing-search-engines
2022-06-02 14:32:37 +02:00
Léon Tiekötter
167f635924
[engine] petal search news
Add Petal Search News to SearXNG via XPath
2022-05-29 22:49:41 +02:00
Léon Tiekötter
f09e8abd76
[engine] petal search
Add Petal Search to SearXNG via XPath
2022-05-29 22:23:30 +02:00
Léon Tiekötter
7c8903c809
[fix] mojeek engine: xpath 2022-05-28 18:09:25 +02:00
Allen
13f4dcafc1
[fix] [new] Brave - Update xpath
Last merge request was messed up with git.
2022-05-26 18:44:11 +02:00
Alexandre Flament
c24864a9ad Enable duckduckgo by default
see https://github.com/searxng/searxng/pull/1219#discussion_r879543174
2022-05-24 23:33:12 +02:00
Alexandre Flament
cea7b71d14 Disable the bing engine
see
* https://github.com/searxng/searxng/discussions/1209
* https://github.com/searxng/searxng/issues/941
* https://github.com/searxng/searxng/pull/1219
2022-05-24 23:33:08 +02:00
Allen
6ef24ffd94
[enh] Add alexandria.org engine
engine: json_engine
official_api_documentation: https://github.com/alexandria-org/alexandria-api/raw/master/README.md
2022-05-15 11:10:15 +02:00
Markus Heiser
444b1e70db [mod] add setting: search.autocomplete_min
Minimun characters to type before autocompleter starts.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-07 17:58:30 +02:00
Markus Heiser
e8541b6006 [theme] peel out oscar from SearXNG development
This is the first step of removing oscar theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-30 13:20:27 +02:00
Léon Tiekötter
02026bcc76
[fix] library genesis engine: xpath and url
Changed the xpath to use the official download mirror.

Changed the URL to libgen.fun as this is the official instance. (https://en.wikipedia.org/wiki/Library_Genesis)
2022-04-27 13:11:34 +02:00
Allen
624b58764a
[enh] Add pagination to Brave
Also added ```&spellcheck=1``` because now it is disabled by default, not returning any ```suggestion_xpath```.
2022-04-23 14:06:53 +02:00
Allen
f039873e1a
[mod] Make a default domain for ebay
Which can be changed.
2022-04-17 10:37:12 +02:00
Vojtěch Fošnár
de4af2fefd [enh] add seznam autocomplete 2022-04-14 03:02:05 +02:00
Austin Huang
19fa0095a0
(fix) satisfy the linter, and btw reduce timeout 2022-04-01 09:23:24 -04:00
Austin Huang
934ae4e086
(feat) add jisho.org
Closes #1016
2022-03-31 14:45:39 -04:00
Alexandre Flament
dec04c0ed6
Merge pull request #999 from return42/fix-doc-settings2
[docs] document settings.yml: search.languages
2022-03-26 17:09:07 +01:00
Markus Heiser
a521d4985a [docs] document settings.yml: search.languages
Requested-by: @dalf https://github.com/searxng/searxng/pull/996#discussion_r830858139
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-26 13:32:42 +01:00
Markus Heiser
7653e8bf16
Merge pull request #995 from return42/fix-932
[interim fix] static files can't be delivered by HTTP.
2022-03-25 06:56:11 +01:00
Markus Heiser
32a35b4654 [docs] improve settings.yml documentation (admin/engines/settings)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-21 08:48:06 +01:00
Markus Heiser
fd5fcdedce [interim fix] static files can't be delivered by HTTP.
Since PR 932 [1][2] static files can't be delivered by HTTP server any longer.

This patch makes the hash paramter in the URL of static files:

    /static/themes/simple/css/searxng.min.css?5fde34a74bc438c7b56ec8c6501e131cc9914bd8

optional.  By default the hash parameter is disabled.

HINT:

  Instances that do not deliver static files by their HTTP server and have a
  long expire time [3] should enable this option.

----

This is only a interim solution, on the long run:

    make static.build.commit

creates files including the file name:

    css/searxng-5fde34a74bc438c7b56ec8c6501e131cc9914bd8.min.css

and a mapping.json with this content[4]

[1] https://github.com/searxng/searxng/issues/964
[2] https://github.com/searxng/searxng/pull/932#issuecomment-1067039518
[3] 5583336440
[4] https://github.com/searxng/searxng/pull/932#issuecomment-1067216426

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-20 16:28:14 +01:00
Markus Heiser
e4a2bfba38 [fix] settings.yml: ui.results_on_new_tab
Closes: https://github.com/searxng/searxng/issues/987
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-19 23:06:21 +01:00
Markus Heiser
20f4538e13 [fix] engine: Semantic Scholar (Science) // rework & fix
Closes: https://github.com/searxng/searxng/issues/939
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-05 11:53:41 +01:00
Alexandre Flament
8230603f48
Merge pull request #916 from dalf/pref_infinite_scroll2
Convert the infinite_scroll plugin as a preference (second version)
2022-02-21 12:49:04 +01:00
Alexandre Flament
56e34947a6 [mod] infinite_scroll as preference
* oscar theme: code from searx/plugins/infinite_scroll.py
* simple theme: new implementation

Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
2022-02-20 22:58:51 +01:00
Markus Heiser
5ec9eb8030 [mod] switch default theme from oscar to simple
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-20 18:36:41 +01:00
Markus Heiser
ae8b174e93 [fix] replace peer.tube by peertube.biz
More peertube instances are listed at [1]

[1] https://instances.joinpeertube.org/instances

Closes: https://github.com/searxng/searxng/issues/881
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-13 22:48:09 +01:00
Markus Heiser
f2e16904a5 [mod] disable deezer engine by default
To play content from deezer a account is needed, the majority of the SearXNG
won't have.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-07 22:58:05 +01:00
Markus Heiser
e2ec6b4211 [fix] invidious engine: store random base_url in param
Two different threads ( = two different user queries) can call the request
function in a row and then the response function.  The namespace will be same
since this is the same engine.

To keep exactly the same value ``base_url`` must be stored in params and then
retrieve using ``resp.search_params["base_url"]``.

Suggested-by: @dalf https://github.com/searxng/searxng/pull/862#discussion_r799324861
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-04 15:42:06 +01:00
Markus Heiser
d92b3d96fd [fix] solidtorrents engine: JSON API no longer exists
The API endpoint, we where using does not exist anymore.  This patch is a
rewrite that parses the HTML page.

Related: https://github.com/paulgoio/searxng/issues/17
Closes: https://github.com/searxng/searxng/issues/858

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-04 14:53:37 +01:00
Alexandre Flament
f79b0fce06 [enh] limiter plugin
can replace filtron:
* rate limite the number of request per IP and per (IP, User-Agent)
* block some bots

use Redis
data stored in Redis never contains the IP addresses, only HMAC using the secret_key

Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-02 09:15:59 +01:00
Alexandre Flament
ebd3013a1a [mod] tineye engine: minor changes
* remove "disable: false" in settings.yml
* use the json() method from httpx.Response (faster character encoding detection)
2022-01-30 20:49:22 +01:00
Markus Heiser
a6b879f19c [mod] tineye engine: set engine_type to 'online_url_search'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-30 16:30:52 +01:00
Allen
880555e263 [enh] engine - add Tineye reverse image search
Other optional parameter ..

`&sort=crawl_date`
    can be appended to search_string to sort results by date.

`&domain=example.org`
    can be implemented to search_string to get results from just one domain.

Public instances could get relatively fast timed-out for 3600s.

--

Merged from @allendema's commit [1] and slightly modfied / see [2].

Related-to: [1] 455b2b4460
Related-to: [2] https://github.com/searx/searx/pull/3040
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-28 09:06:44 +01:00
Markus Heiser
ccffab3a43
Merge pull request #813 from MarcAbonce/change-wolfram-category
Move WolframAlpha away from Science category
2022-01-26 14:10:19 +01:00
Markus Heiser
10c9b9775d [mod] improve setup of invidious engine
- My experience is, that a timeout of 5 sec is not need, I got fast response
  less than a second.

- https://invidious.tube/ redirects to http://ww25.invidious.tube/
  - in SearXNG defaults the http protocol is unsafe and raise an error
  - https://ww25.invidious.tube has SSL_ERROR_UNSAFE_NEGOTIATION

Related-to: https://github.com/searxng/searxng/issues/821
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-25 19:02:11 +01:00
Marc Abonce Seguin
1311745f24 move WolframAlpha away from Science category 2022-01-23 23:24:02 -06:00
Markus Heiser
977e9a4330
Merge pull request #686 from return42/lib_redis
Add redis DB and connector
2022-01-11 19:55:14 +01:00
Léon Tiekötter
37baf46ece [fix] Rename ccengine engine to openverse
The CC engine was merged with WordPress and renamed to Openverse

Source: https://wordpress.org/news/2021/05/welcome-to-openverse/
2022-01-07 13:06:05 +01:00
Markus Heiser
ced656606f
Merge pull request #709 from return42/drop-etools
[fix] drop etools engine module
2022-01-07 11:18:47 +01:00
Markus Heiser
5dd3442f83 [fix] drop etools engine module
The implementation of the etools engine is poor.  No date-range support, no
language support and it is broken by a CAPTCHA.

etools is a metasearch engine, the major search engines it supports (google,
bing, wikipedia, Yahoo) are already available in SeaarXNG.

While etools does support several engines we currently don't support directly,
support for them should be added directly to SearXNG if there is demand.

In practice: in SearXNG the worse etools results will be mixed with good results
from other engines we have (as long as there is no captcha).

At best case, what we win with etools is in e.g. results from de.ask.com in a
query from a german request .. in all other cases worse results are bubble up in
SearXNG's result list.

[1] https://github.com/searxng/searxng/issues/696#issuecomment-1005855499

Closes: https://github.com/searxng/searxng/issues/696
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-07 10:41:09 +01:00
Léon Tiekötter
3ab826de22 Drop microsoft academic engine
Microsoft academic was discontinued on 2021-12-31.

Source: https://www.microsoft.com/en-us/research/project/academic/articles/microsoft-academic-to-expand-horizons-with-community-driven-approach/
2022-01-07 01:35:13 +01:00
Alexandre Flament
aedd6279b3
Merge pull request #634 from not-my-profile/powered-by
Introduce `categories_as_tabs` & group engines in tabs
2022-01-06 09:22:02 +01:00
Markus Heiser
0ebad8220f
Merge pull request #688 from dalf/settings_enable_stats
[enh] settings.yml: implement general.enable_metrics
2022-01-05 18:53:49 +00:00
Alexandre Flament
2134703b4b [enh] settings.yml: implement general.enable_metrics
* allow not to record metrics (response time, etc...)
* this commit doesn't change the UI. If the metrics are disabled
  /stats and /stats/errors will return empty response.
  in /preferences, the columns response time and reliability will be empty.
2022-01-05 19:03:04 +01:00
Martin Fischer
a4c2cfb837 [enh] change categories_as_tabs from a list to a dict
The tab icon names are currently hard coded in the templates.
This commit lets us introduce an icon property in the future, e.g:

categories_as_tabs:
  general:
    icon: search-outline
2022-01-05 11:03:44 +01:00
Martin Fischer
3f76c9c96c [enh] enable English dictionaries by default
These dictionaries are no longer part of the general category,
so they're no longer queried by default -> we can enable them
by default without degrading general query performance.
2022-01-05 11:03:44 +01:00
Martin Fischer
5d74bf3820 [enh] move dictionaries, Erowid & IMDb out of general category
The general category is the category that is searched by default.
From a privacy standpoint it doesn't make sense to send all general
queries to specialized search engines that cannot deal with those
queries anyway.
2022-01-05 11:03:44 +01:00
Martin Fischer
b02f762687 [enh] add more categories 2022-01-05 11:00:11 +01:00
Markus Heiser
d7b4306444 [fix] links in comments - https://docs.searxng.org/
fixed links from https://searxng.github.io/searxng/ to https://docs.searxng.org/
2022-01-04 16:04:49 +01:00
Markus Heiser
a6cfab93fa [enh] add redis connector searx/shared/redisdb.py
Add a redis connector, the default DB connector is a socket at::

    unix:///usr/local/searxng-redis/run/redis.sock?db=0

To set up a redis instance simply use::

    $ ./manage redis.build
    $ sudo -H ./manage redis.install

A hint for developers:

To get access rights to this instance, your developer account needs to be added
to the *searxng-redis* group::

    $ sudo -H ./manage redis.addgrp "${USER}"
    # don't forget to logout & login to get member of group

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 20:56:39 +01:00
Martin Fischer
8e9ad1ccc2 [enh] introduce categories_as_tabs
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.

This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs  can still be searched with !bangs.
2022-01-03 07:01:49 +01:00
Alexandre Flament
76cbfbbdda reference docs.searxng.org 2022-01-02 21:18:29 +01:00
Martin Fischer
19339067c6 [del] remove geektimes (redirects to habrahabr) 2021-12-22 13:20:56 +01:00
Martin Fischer
fb037e7a83 [del] remove whaleslide (website is down)
Closes #639.
2021-12-22 13:19:27 +01:00
Martin Fischer
e28c6bda35 [doc] introduce about.language and sort engines by it 2021-12-21 09:58:51 +01:00
Martin Fischer
1fb1cca519 [doc] fix about.website for hoogle 2021-12-20 12:12:09 +01:00
Martin Fischer
06beffdf31 [doc] add about.website for wiby 2021-12-20 07:59:51 +01:00
Alexandre Flament
6bb6f7ffc1
Merge pull request #602 from return42/fix-600
[fix] move engines gpodder and AZlyrics into category music
2021-12-16 18:18:16 +01:00
Markus Heiser
07f2a9ee3c [fix] engine: MediathekViewWeb (mvw) disable by default
The MediathekViewWeb delivers only content for the german speaking area.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-15 19:03:35 +01:00
Markus Heiser
709482088e [fix] move engines gpodder and AZlyrics into category music
Closes: https://github.com/searxng/searxng/issues/600
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-15 09:43:56 +01:00
Markus Heiser
8cc7c880ae
Merge pull request #587 from dalf/fix-gigablast
[fix] gigablast engine
2021-12-12 15:58:13 +01:00
Alexandre Flament
1a6207574e [fix] gigablast engine
fetch extra params after 3000 seconds
2021-12-06 22:55:15 +01:00
Alexandre Flament
fbc2a6ab4b [del] remove yggtorrent
yggtorrent is behind cloudflare now
close #580
2021-12-06 21:59:51 +01:00
Alexandre Flament
42c1a938f0
Merge pull request #485 from dalf/pref_query_in_title
[mod] new preference: query_in_title
2021-11-24 13:10:27 +01:00
Alexandre Flament
d7eb373371 [fix] npm engine: don't ignore the first result
"from" parameter start at 0
2021-11-20 08:22:48 +01:00
Alexandre Flament
bc0cf9e83e
Merge pull request #521 from dalf/simple-theme-style-pref
Simple theme: allow the user to choose auto, light, dark style
2021-11-19 17:14:35 +01:00
Alexandre Flament
ec5a82fccd [enh] simple theme: add "simple-style" preferences 2021-11-19 16:45:50 +01:00
Markus Heiser
86090e13d3 [mod] enable reddit - to have at least one in category social-media
Since digg no longer works, we do nat have a active engine in the social-media
category.  Enable reddit by default to have at least one engine back in this
category.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-19 15:06:42 +01:00
Markus Heiser
4c82ac7670 [drop] engine digg - https://digg.com/api is no longer available
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-19 15:00:22 +01:00
Markus Heiser
4c2df9358a [fix] enigne library genesis - remove 'enable_http: true'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-19 13:25:08 +01:00
mrpaulblack
af79cce7cf [engines] move library of genesis from general to files category 2021-11-17 00:57:13 +01:00
Markus Heiser
6debda2404 [mod] settings.yml - polish up comments about plugins
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 15:40:00 +01:00
Alexandre Flament
80fb77476f [mod] new preference: query_in_title
* disable by default
* settings.yml: ui.query_in_title
* in /preferences: privacy tab

when enabled, the result page's title contains the user query.

previously:
* oscar theme: the query was always included
* simple theme: the query was included with the GET method
2021-11-08 21:40:53 +01:00
Alexandre Flament
64b29ad838 [mod] microsoft academic: increase timeout to 6 seconds
also avoid a crash when there is no result

close #433
2021-10-26 12:26:43 +02:00
Alexandre Flament
60db344cc0 [enh] reduce the available language list
close #406

settings['search']['languages'] contains the languages codes and 'all'
2021-10-26 10:27:05 +02:00
Paolo Basso
64df011e2f [mod] engines - add zlibrary engine 2021-10-11 14:58:44 +00:00
Alexandre Flament
3fad483b7a SearXNG: partial update: reference /etc/searxng/settings.yml 2021-10-11 12:28:55 +00:00
Markus Heiser
9fb77065bd [fix] engine torznab - marginal issues reported from linters
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-07 15:27:55 +00:00
Paolo Basso
d803df8d89 [mod] engines - add torznab WebAPI 2021-10-07 15:27:55 +00:00
Markus Heiser
19e41c137e [mod] set 'engine.supported_languages' from the origin python module
The key of the dictionary 'searx.data.ENGINES_LANGUAGES' is the *engine name*
configured in settings.xml.  When multiple engines are configured to use the
same origin engine (e.g. `engine: google`)::

    - name: google
      engine: google
      use_mobile_ui: false
      ...

    - name: google italian
      engine: google
      use_mobile_ui: false
      language: it
      ...

    - name: google mobile ui
      engine: google
      shortcut: gomui
      use_mobile_ui: true

There exists no entry for ENGINES_LANGUAGES[engine.name] (e.g. `name: google
mobile ui` or `name: google italian`).  This issue can be solved by recreate the
ENGINES_LANGUAGES::

    make data.languages

But this is nothing an SearXNG admin would like to do when just configuring
additional engines, since this just doubles entries in ENGINES_LANGUAGES and
BTW: `make data.languages` has various external requirements which might be not
installed or not available, on a production host.

With this patch, if engine.name fails, ENGINES_LANGUAGES[engine.engine] is used
to get the engine.supported_languages (e.g. `google` for the engine named
`google mobile`).

For an engine, when there is `language: ...` in the YAML settings, the engine
supports only one language, in this case engine.supported_languages should
contains this value defined in settings.yml (e.g. `it` for the engine named
`google italian`).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Closes: https://github.com/searxng/searxng/issues/384
2021-10-07 08:45:02 +02:00
Paul Alcock
823d44ed0a [mod] engines - add IMDB / Internet Movie Database
Merged from @Guilvareux's commit [1] and slightly modfied / see [2].

[1] https://github.com/searx/searx/pull/2980/commits/f2f90071
[2] https://github.com/searx/searx/pull/2980
2021-10-03 11:44:25 +02:00
Markus Heiser
5efe77bdf5 [mod] engines - add superuser.com (Stack Exchange API)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-28 19:20:13 +02:00
Markus Heiser
29eb06ab16 [mod] engines - add askubuntu.com (Stack Exchange API)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-28 19:17:26 +02:00
Markus Heiser
b62851559b [mod] replace old stackoverflow engine by Stack Exchange API v2.3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-28 19:12:37 +02:00
Alexandre Flament
8961131497 [fix] fix the about section of some engines 2021-09-24 20:20:30 +02:00
Markus Heiser
de5a8ee7d7 [fix] settings - displayed name is SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-13 14:36:24 +02:00
Alexandre Flament
0f43b39eac [enh] add hostname_replace plugin
* backport of https://github.com/searx/searx/pull/2724
* allow to remove result if the replacement is the boolean value false
2021-09-11 13:23:06 +02:00
Alexandre Flament
39bbd15b39
Merge pull request #287 from return42/azlyrics
[enh] add engine AZLyrics.com (XPath)
2021-09-04 13:36:09 +02:00
Markus Heiser
730073b850 [enh] add suggestions to brave engine
Suggestions should be added too.

    suggestion_xpath: //div[@class="text-gray h6"]/a

You can try it with:

    !brave recurzuoin

Suggested-by: @allendema in https://github.com/searx/searx/issues/2857#issuecomment-904837023
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-04 11:38:59 +02:00
Allen
0648757e7f [enh] add engine AZLyrics.com (XPath)
Upstream example query:
  https://search.azlyrics.com/search.php?q=The+verbal+acupunture&w=lyrics&p=1

Cherry picked from [1] and slightly modified to pass yamllint test.

[1] https://github.com/allendema/searx/commit/8b6d5a0e5

Reviewed-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-04 10:58:34 +02:00
Kevin Köllmann
2b3687780c [enh] Add Packagist search engine (JSON)
Packagist's API: https://packagist.org/apidoc
2021-08-31 17:59:34 +02:00
Markus Heiser
15bf9fc850 [fix] settings.yml - replace http:// URLs by https://
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-07 16:43:19 +02:00
Alexandre Flament
f30d01ffab [mod] settings.yml: remove locales
There are detected from the searx/translations directory
2021-08-03 15:44:45 +02:00
Alexandre Flament
38ee88b390
Merge pull request #229 from dalf/version-from-git
version based on git
2021-07-30 18:57:00 +02:00
Alexandre Flament
4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
Alexandre Flament
ed0db4d61a
Merge pull request #226 from return42/hardening
[mod] hardening SearXNG instances by default (formats)
2021-07-27 11:22:18 +02:00
Alexandre Flament
24fcd7475a Revert "[brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42"
This reverts commit e946c3462c.
2021-07-27 08:40:53 +02:00
Markus Heiser
beff764d74
Merge pull request #216 from return42/fix-searx.sh
normalize environment of installation tasks (shell) with YAML settings
2021-07-27 05:32:57 +00:00
Markus Heiser
7b40c736e6 [mod] hardening SearXNG instances by default (formats)
Deny formats has been implemented in 6ed4616d.

To harden SearXNG instances by default, other formats than HTML should be
denied.  Most of JSON, RSS and CSV requests are bots [1]::

    Bots are the only users of this feature on a public instance, and they abuse
    it too much that the engines rate limit pretty quickly the IP address of the
    instance.

[1] https://github.com/searxng/searxng/issues/95

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 12:59:09 +02:00
Noémi Ványi
5e648e77ab [docs] document NoSLQ engines MongoDB and redis
Slightly modified merge of commit [3b192e638]

[3b192e638] https://github.com/searx/searx/commit/3b192e638

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 11:39:11 +02:00
Markus Heiser
e946c3462c [brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42
Do not merge this patch in master branch of SearXNG!  This branch exists only
for testing the feature branch fix-searx.sh @return42.

This patch changes the buildenv to::

    GIT_URL='https://github.com/return42/searxng'
    GIT_BRANCH='fix-searx.sh'
    SEARX_PORT='7777'
    SEARX_BIND_ADDRESS='127.0.0.12'

To test installation procedure, clone feature branch (fix-searx.sh)::

    $ cd ~/Downloads
    $ git clone --branch fix-searx.sh https://github.com/return42/searxng searxng
    $ cd searxng

    $ ./utils/searx.sh install all
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
b6a55e223c [mod] reduce enviroment variables in shell scripts to what is needed
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts.  This patch reduce the variables in ./utils/brand.env to what is
needed.  The following ('brand:' settings) can be removed from this file:

- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL

Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:

- GIT_URL            <--> brand.git_url
- GIT_BRANCH         <--> brand.git_branch
- SEARX_URL          <--> server.base_url  (aka PUBLIC_URL)
- SEARX_PORT         <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Alexandre Flament
f523fd3ea7
Merge pull request #211 from MarcAbonce/onions_v3_fix_searxng
Update onion engines to v3
2021-07-16 17:25:37 +02:00
Marc Abonce Seguin
1b05ea6a6b update onion engines to v3
remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
2021-07-16 01:36:34 -07:00
Markus Heiser
0a9cd08bf1 [enh] add mongodb offline engine
Cherry-Pick: https://github.com/searx/searx/commit/198aad43
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15 21:35:33 +02:00
Markus Heiser
420e3caba3 [mod] enable Qwant engines (web, news, images, video) by default
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA.  Let
us enable Qwant engines by default.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Markus Heiser
ef6e1bd6b9 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Markus Heiser
2767046065 [fix] drop engine dogpile
The engine was added in commit a4b07460 but now it shows new issues [1].

In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]

  Powered by technology, Dogpile returns all the best results from leading
  search engines including Google and Yahoo!

Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough.   It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.

With this patch the dogpile engine is removed

[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-13 14:42:09 +02:00
Markus Heiser
513c73a309 [drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/

Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-11 13:24:33 +02:00
Allen
3b2e0a93b7 [enh] add gpodder.net (JSON)
Engine just for Podcasts.  An API which returns Podcasts and their Info like:
website, author etc.

Upstream query example: https://gpodder.net/search.json?q=linux
2021-07-03 22:30:20 +02:00
Markus Heiser
d5eedf1b61 [yamllint] fix some warning issues reported by yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 21:55:36 +02:00
Allen
9648d71ad8 [enh] add woxikon.de synonyme (Xpath)
Added synonyme.woxikon.de using the xpath engine.  Adds a site which returns
word synonyms although just in German.

Depending on the query not all synonyms are shown because of not the best xpath
selection.  But should do the job just fine.

Upstream example query: https://synonyme.woxikon.de/synonyme/test.php
2021-07-03 21:51:26 +02:00