Commit graph

2886 commits

Author SHA1 Message Date
Markus Heiser
c6a5cc019a [brand] searxng is a fork from searx
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 11:42:42 +02:00
Markus Heiser
6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
spongebob33
7528e38c8a add core.ac.uk engine 2021-04-24 08:55:45 +02:00
Alexandre Flament
525bfe7830
Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament
7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament
09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Markus Heiser
c54bf42cb9 [mod] metrics: add to pylint and justify indentations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 06:58:49 +02:00
Alexandre Flament
65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00
Alexandre Flament
3755cf2eba More branding 2021-04-22 21:10:22 +02:00
Alexandre Flament
04b34c70b8 More branding 2021-04-22 20:14:16 +02:00
Alexandre Flament
9d84025e4f searxng brand 2021-04-22 19:57:02 +02:00
Alexandre Flament
d01741c9a2
Merge pull request #15 from return42/add-springer
Add a search engine for Springer Nature
2021-04-22 13:23:31 +02:00
Markus Heiser
213041adc7 [enh] themes - add result.metadata to result sub header
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-22 12:37:18 +02:00
Pierre Chevalier
a80bf1ba97 [enh] Add Springer Nature engine
Springer Nature is a global publisher dedicated to providing service to research
community [1] with official API [2].

To test this PR, first get your API key following this page:

   https://dev.springernature.com/signup

In searx/engines/springer.py at line 24, add this API key.  I left my own key,
commented out in the line aboce.  Feel free to use it, if needed.

[1] https://www.springernature.com/
[2] https://dev.springernature.com/
2021-04-22 12:35:25 +02:00
habsinn
41a2e3785e [enh] add engine using API from "The Art Institute of Chicago" 2021-04-22 12:25:43 +02:00
Alexandre Flament
981cdb7acf
Merge pull request #10 from return42/fix-youtube
[fix] youtube - send CONSENT Cookie to not be redirected
2021-04-22 12:21:31 +02:00
Markus Heiser
9a05ad4c73 [enh] add DOI resolver from sci-hub / replace default DOI
The new sci-hub URLs are comming from @aurora-vasiliev [1].

[1] https://github.com/searx/searx/pull/2706

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 12:13:01 +02:00
Markus Heiser
e9a6ab4015 [fix] youtube - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, youtube requests are redirected to confirm and get a CONSENT
Cookie from https://consent.youtube.com

This patch adds a CONSENT Cookie to the youtube request to avoid redirection.

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Reported-by: https://github.com/searx/searx/issues/2774
2021-04-22 12:09:09 +02:00
Markus Heiser
247c46c6b0 [drop] plugin HTTPS rewrite - rule base is to old (update is to big)
The usefulness of the _HTTPS rewrite_ plugin is questionable:

- the 36 rule files have not been updated since 2015 [1]
- actual there are 23760 rule files in the https-everywhere repo [2]

For the first, we can remove this plugin.  For a complete new implementation, it
might be good to know that there is a project "https-everywhere : Privacy for
Pythons" [3]

related:  https://github.com/return42/searx-next/issues/8

[1] https://github.com/return42/searx-next/tree/d187a1d/searx/plugins/https_rules
[2] https://github.com/EFForg/https-everywhere/tree/master/src/chrome/content/rules
[3] https://github.com/jayvdb/https-everywhere-py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 11:47:16 +02:00
Alexandre Flament
c6d5605d27
Merge pull request #7 from searxng/metrics
Metrics
2021-04-22 08:34:17 +02:00
Alexandre Flament
baff1cbbab fix issues from review 2021-04-21 18:49:13 +02:00
Alexandre Flament
b7848e3422 [fix] searxng fix: sjp engine 2021-04-21 16:31:29 +02:00
Alexandre Flament
7cfd8d900a [mod] oscar: /preferences , engines tab: report engine times
* display the median time instead of the average.
* add a "Reliability" column (sum up the metrics and the checker results).
* the "selected language", "SafeSearch", "Time range" values are displayed as "broken" when the checker tests fail.
2021-04-21 16:24:46 +02:00
Alexandre Flament
c27fef1cde [mod] metrics: add secondary parameter
Some error won't stop the engine:
* additional HTTP redirects for example
* some invalid results

secondary=True allows to flag these errors as not important.
2021-04-21 16:24:46 +02:00
Alexandre Flament
7acd7ffc02 [enh] rewrite and enhance metrics 2021-04-21 16:24:46 +02:00
Alexandre Flament
aae7830d14 [mod] refactoring: processors
Report to the user suspended engines.

searx.search.processor.abstract:
* manages suspend time (per network).
* reports suspended time to the ResultContainer (method extend_container_if_suspended)
* adds the results to the ResultContainer (method extend_container)
* handles exceptions (method handle_exception)
2021-04-21 16:24:46 +02:00
Alexandre Flament
ae5954f2da Merge remote-tracking branch 'return42/fix-preference-save' 2021-04-21 16:24:22 +02:00
Alexandre Flament
624d3ec6db Merge remote-tracking branch 'return42/fix-url-bar-suggestion' 2021-04-21 16:23:52 +02:00
Alexandre Flament
92db0227b1 Merge remote-tracking branch 'dalf/oscar-images' 2021-04-19 09:36:54 +02:00
Alexandre Flament
48720e20a8 Merge remote-tracking branch 'searx/master' 2021-04-19 09:35:12 +02:00
Noémi Ványi
8362257b9a
Merge pull request #2736 from plague-doctor/sjp
Add new engine: SJP - Słownik języka polskiego
2021-04-16 17:30:14 +02:00
Noémi Ványi
e56323d3c8
Merge pull request #2759 from ypid/fix/typo
Fix grammar mistake in debug log output
2021-04-16 17:26:45 +02:00
Plague Doctor
d275d7a35e Code refactoring. 2021-04-16 12:23:27 +10:00
Markus Heiser
f637bfc635 [mod] oscar's "default" template should make use of result.thumbnail
Some engine do have set result.img_src, other return a result.thumbnail.  If
result.img_src is unset and a result.thumbnail is given, show it to the UI.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-15 08:52:11 +02:00
Markus Heiser
062d589f86 [fix] xpath expressions to grap all items from bandcamp's response
I also found some items missing a thumbnail and I used text_extract for content
and title, to remove unneeded whitespaces.

BTW: added bandcamp's favicon

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-15 08:52:11 +02:00
Kyle Anthony Williams
4d3c399ee9 [feat] add bandcamp engine 2021-04-15 08:52:11 +02:00
Alexandre Flament
d14994dc73 [httpx] replace searx.poolrequests by searx.network
settings.yml:

* outgoing.networks:
   * can contains network definition
   * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
     keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
   * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
   * local_addresses can be "192.168.0.1/24" (it supports IPv6)
   * support_ipv4 & support_ipv6: both True by default
     see https://github.com/searx/searx/pull/1034
* each engine can define a "network" section:
   * either a full network description
   * either reference an existing network

* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
2021-04-12 17:25:56 +02:00
Markus Heiser
c4793afadc [fix] https-scheme missing in preferences-page
This patch is an addition to PR #2656 which removed all usage of `base_url` from
the templates, except one was forgotten in the cookie URL of the preferences.

closes: 2740

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12 16:02:16 +02:00
Markus Heiser
2bf297b19f [fix] redirect when saving preferences
Erroneously commit 87e4c4762 droped the 302 redirect.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12 15:29:08 +02:00
Robin Schneider
dfc66ff0f0
Fix grammar mistake in debug log output 2021-04-11 22:12:53 +02:00
Alexandre Flament
eaa694fb7d [enh] replace requests by httpx 2021-04-10 15:38:33 +02:00
Plague Doctor
599ff39ddf Fix conflicts 2021-04-09 06:54:03 +10:00
Noémi Ványi
cc359345a8
Merge pull request #2735 from plague-doctor/wordnik
Add new engine: Wordnik.com
2021-04-08 19:48:13 +02:00
Plague Doctor
6631f11305 Add new engine: SJP 2021-04-08 10:21:54 +10:00
Plague Doctor
7035bed4ee Add new engine: Wordnik.com 2021-04-08 09:58:00 +10:00
Alexandre Flament
c09ff4faf2 [fix] fix PR 2656
SCRIPT_NAME remove trailing slash to avoid infinite redirect
2021-04-07 13:05:55 +02:00
Noémi Ványi
07f5edce3d Add Meilisearch engine
Website: https://www.meilisearch.com/
2021-04-06 21:57:05 +02:00
Noémi Ványi
a477a3a687
Merge pull request #2728 from return42/fix-loader
[fix] settings_loader.py - use update_dict only for mapping types
2021-04-05 21:32:41 +02:00
3nprob
2ca0aa4f29 Remove hubsbpot tracking URL params
More information here:
https://knowledge.hubspot.com/settings/how-do-i-create-a-tracking-url
https://knowledge.hubspot.com/ads/ad-tracking-in-hubspot
https://knowledge.hubspot.com/ctas/calls-to-action-frequently-asked-questions
https://meta.stackexchange.com/questions/263392/what-are-these-very-long-parameters-in-so-careers-feedback-always-welcome-url
2021-04-06 02:11:09 +09:00
Markus Heiser
9c10b15096 [fix] settings_loader.py - use update_dict only for mapping types
I can't set `default_doi_resolver` in `settings.yml` if I'm using
`use_default_settings`.  Searx seems to try to interpret all settings at root
level in `settings.yml` as dict, which is correct except for
`default_doi_resolver` which is at root level and a string::

    File "/usr/lib/python3.9/site-packages/searx/settings_loader.py", line 125, in load_settings
        update_settings(default_settings, user_settings)
    File "/usr/lib/python3.9/site-packages/searx/settings_loader.py", line 61, in update_settings
        update_dict(default_settings[k], v)
    File "/usr/lib/python3.9/site-packages/searx/settings_loader.py", line 48, in update_dict
        for k, v in user_dict.items():
    AttributeError: 'str' object has no attribute 'items'

Signed-off-by: Markus Heiser <markus@darmarit.de>
Suggested-by:  @0xhtml https://github.com/searx/searx/issues/2722#issuecomment-813391659
2021-04-05 16:33:48 +02:00
Alexandre Flament
7089526723
Merge pull request #2656 from return42/fix-url_for
[fix] url_for(..., _external=True) in templates
2021-04-05 14:50:39 +02:00
Markus Heiser
87e4c47621 [fix] url_for(..., _external=True) in templates
The `url_for` function in the template context is not the one from Flask, it is
the one from `webapp`.  The `webapp.url_for_theme` is different from its
namesake of Flask and has it quirks, when called with argument `_external=True`.

The `webapp.url_for_theme` can't handle absolute URLs since it pokes a leading
'/', here is the snippet of the old code::

    url = url_for(endpoint, **values)
    if settings['server']['base_url']:
        if url.startswith('/'):
            url = url[1:]
        url = urljoin(settings['server']['base_url'], url)

Next drawback of (Flask's) `_external=True` is, that it will not return the HTTP
scheme when searx (the Flask app) listens on http and is proxied by a https
server.

To get the right scheme `HTTP_X_SCHEME` is needed by Flask (werkzeug).  Since
this is not provided in every environment (e.g. behind Apache mod_wsgi or the
HTTP header is not fully set for some other reasons) it is recommended to
get *script_name*, *server* and *scheme* from the configured `base_url`.  If
`base_url` is specified, then these values from are given preference over any
Flask's generics.

BTW this patch normalize to use `url_for` in the `opensearch.xml` and drop the
need of `host` and `urljoin` in template's context.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-05 14:34:45 +02:00
Alexandre Flament
7a0fbdecc4 [enh] oscar: image thumbnail layout
Adjust thumbnail sizes to fill the container width
2021-04-05 11:29:17 +02:00
Noémi Ványi
ba90c5a2e0
Merge pull request #2718 from return42/fix-publishedDate
[fix] publishedDate: don't try to get date from empty string or None
2021-04-04 23:00:22 +02:00
Markus Heiser
ebfd0eb2b7 [fix] default_doi_resolver in preferences
Instead of a hard-coded `oadoi.org` default, use the default value from
`settings.yml`.

Fix an issue in the themes: The replacement 'current_doi_resolver' contains the
doi_resolver_url, not the name of the DOI resolver.  Compare return value of::

    searx.plugins.oa_doi_rewrite.get_doi_resolver(...)

Fix a typo in `get_doi_resolver(..)`:  suggested by @kvch:

  *L32 should set doi_resolver not doi_resolvers*

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-04 13:36:33 +02:00
Markus Heiser
c12826c6d5 [fix] publishedDate: don't try to get date from empty string or None
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-04 13:16:38 +02:00
Noémi Ványi
7612e5d9fd
Merge pull request #2712 from searx/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2021-04-03 23:15:43 +02:00
Noémi Ványi
985e8b28fe
Merge pull request #2710 from searx/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-04-03 23:02:44 +02:00
Noémi Ványi
feb2d81874
Merge pull request #2709 from searx/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-04-03 23:02:26 +02:00
Noémi Ványi
9a962c5369
Merge pull request #2711 from searx/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-04-03 23:01:43 +02:00
Noémi Ványi
5ca34ac571
Merge pull request #2713 from searx/update_data_update_languages.py
Update searx.data - update_languages.py
2021-04-03 23:01:00 +02:00
Markus Heiser
169438137f [fix] url bar autocomplete (opensearch suggestions)
Since #2593 is merged the OpenSearch-Format is buggy.  The loop in [1] will
change raw_text_query object and this will change also the value of
`raw_text_query.query` on every `raw_text_query.changeQuery(result)`.

This patch fixes this issue by storing the initial query value in `sug_prefix`.

[1] ac0fdc3b96/searx/webapp.py (L804-L806)

OpenSearch-Format::

    [ "<query>",
      [ "<term 1>", "<term 2>", ... "<term n>" ],
      [ "<content 1>", "<content 2>", ..., "<content n>" ],
      [ "<url 1>", "<url 2>", ..., "<url n>" ]
    ]

- https://www.google.com/support/enterprise/static/gsa/docs/admin/current/gsa_doc_set/xml_reference/query_suggestion.html#1080002
- https://developer.mozilla.org/en-US/docs/Archive/Add-ons/Supporting_search_suggestions_in_search_plugins#implementing_search_suggestion_support_on_the_server

Legacy-Format::

    [ "<term 1>", "<term 2>", ..., "<term n>" ]

- https://www.google.com/support/enterprise/static/gsa/docs/admin/current/gsa_doc_set/xml_reference/query_suggestion.html#1081079

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-03 18:18:50 +02:00
dalf
c0668d248e Update searx.data - update_languages.py 2021-04-01 07:00:46 +00:00
dalf
ad74c42aa1 Update searx.data - update_firefox_version.py 2021-04-01 07:00:39 +00:00
dalf
23dc7ef6d1 Update searx.data - update_wikidata_units.py 2021-04-01 07:00:37 +00:00
dalf
eb5cd7a543 Update searx.data - update_ahmia_blacklist.py 2021-04-01 07:00:33 +00:00
dalf
f804f54ca3 Update searx.data - update_currencies.py 2021-04-01 07:00:29 +00:00
Marc Abonce Seguin
419b907a0b fix dark "expand" button from infobox 2021-03-28 21:54:37 -07:00
Adam Tauber
ae122ea943 [enh] release v1.0.0 2021-03-27 20:30:08 +01:00
Adam Tauber
15f2385a7b [enh] update translations from transifex 2021-03-27 19:10:54 +01:00
Alexandre Flament
bb316d2698
Merge pull request #2682 from dalf/fix-checker
[fix] checker: various bug fixes
2021-03-27 17:44:29 +01:00
Alexandre Flament
725a69616b
Merge pull request #2681 from dalf/fix-wikipedia-title
[fix] wikipedia: remove HTML from the title
2021-03-27 17:43:36 +01:00
Noémi Ványi
9bb312c505 Remove duplicated key from dict in Semantic Scholar 2021-03-27 16:58:32 +01:00
Noémi Ványi
f596f5767b fix Semantic Scholar engine 2021-03-27 16:54:01 +01:00
Adam Tauber
28286cf3f2 [fix] update seznam engine to be compatible with the new website 2021-03-27 15:29:04 +01:00
Alexandre Flament
0b45afd4d7 [fix] checker: various bug fixes
* initialize engine_data (youtube engine)
* don't crash if an engine don't set result['url']
2021-03-25 09:37:37 +01:00
Alexandre Flament
fcfcf662ff [fix] wikipedia: remove HTML from the title
fr.wikipedia.org (and it seems not other wikipedia websites),
adds HTML to api_result['displayTitle'].
(Search for '!wp :fr Braid' for example)

The commit uses api_result['title']
2021-03-25 08:31:39 +01:00
Adam Tauber
06b754ad67 [mod] increase lobste.rs engine timeout to avoid timeouts most of the time 2021-03-25 01:22:36 +01:00
Adam Tauber
0ba71c3644 [fix] make ina engine compatible with the new response json 2021-03-25 01:20:41 +01:00
Adam Tauber
6255b33c9d [fix] rewrite hoogle to use html/xpath instead of json
the json response has been changed and it contains html chunks which is
not compatible with our json engine, so we have to switch to html/xpath
parsing
2021-03-25 01:13:24 +01:00
Adam Tauber
45f0e1a859 [fix] update geektimes.ru url - it redirects to habr.com 2021-03-25 01:02:19 +01:00
Adam Tauber
50ba2b9e87 [fix] update google play movies xpath 2021-03-25 00:55:53 +01:00
Adam Tauber
88657fe9c2 [fix] update google play apps xpath 2021-03-25 00:55:43 +01:00
Adam Tauber
5f450fda74 [enh] add year filter to duckduckgo 2021-03-25 00:25:36 +01:00
Adam Tauber
fd737dc9d8 [fix] remove debug code 2021-03-24 23:54:39 +01:00
Alexandre Flament
d648001688 [mod] preferences: a tooltip is shown when the mouse is over the engine names 2021-03-22 08:22:59 +01:00
Alexandre Flament
6bd01bf81f [mod] oscar: fix the sourcemap URL in *.min.css
Close https://github.com/searx/searx/issues/2670

Note: clean-css contains a bug:
* a multiline comment or URL adds "$stdin" to the sourcemap (see src/less/logicodev/search.less)
* in this case when the user opens the devtools, the browser fails to load this "https://.../$stdin" URL
* it is not a problem and the error appears only when the user actively tries to debug the CSS.
* seems related to https://github.com/jakubpawlowicz/clean-css/issues/593
2021-03-21 18:03:40 +01:00
Alexandre Flament
a48ec0b4bd
Merge pull request #2671 from searx/update-soundcloud
[mod] soundcloud: faster initialization
2021-03-21 15:10:39 +01:00
Alexandre Flament
30c950a2c7
Merge pull request #2660 from dalf/upd-translations
[mod] replace /translations.js with an embedded JSON
2021-03-21 12:39:26 +01:00
Alexandre Flament
38c210d746
[mod] soundcloud: faster initialization
The get_cliend_id() function:
* fetches https://soundcloud.com
* then fetches each referenced javascript URL to get the client id.

This commit fetches the javascript URLs in the reverse order: the client id is in the last javascript URL.
2021-03-21 09:29:53 +01:00
James Higginbotham
ce6eb81a71
Update settings.yml to enable HTTP for yacy
Added a line to the yacy entry to enable HTTP if the local yacy instance isn't using HTTPS. Otherwise, an error will be thrown in the logs: "No connection adapters were found for 'http://localhost:8090/yacysearch.json...'". This is likely related to ticket #2641 that forces HTTPS by default.
2021-03-19 15:06:25 -06:00
Alexandre Flament
2b0dd96bd3 [mod] oscar: remove space
* reduce by 15% the uncompressed output (on average)
* dos2unix searx/templates/oscar/result_templates/files.html
2021-03-17 09:22:05 +01:00
Dr. Rolf Jansen
7a9dc63d74
Merge branch 'master' into conditional-sigusr1 2021-03-16 08:45:57 -03:00
Alexandre Flament
6553c79029 [mod] replace /translations.js by embedded JSON
In webapp.py, there is a new function "get_translations" lists available translations

Close #2064
2021-03-16 11:22:21 +01:00
Alexandre Flament
32cd0d31b3 [mod] upgrade pygments
add searx_extra/update/update_pygments.py to update the css style of the oscar and simple themes.
2021-03-16 09:07:08 +01:00
Dr. Rolf Jansen
2a6dbeb6a5
Merge branch 'master' into conditional-sigusr1 2021-03-15 19:31:44 -03:00
Adam Tauber
4c631ac6d0 [fix] remove debug code 2021-03-15 21:47:27 +01:00
Dr. Rolf Jansen
4a27dabcf7
Merge branch 'master' into conditional-sigusr1 2021-03-15 17:03:36 -03:00
Noémi Ványi
8158d8654a fix Microsoft Academic engine 2021-03-15 20:21:28 +01:00
Adam Tauber
f97b4ff7b6 [fix] update youtube_noapi paging 2021-03-15 17:22:31 +01:00
Adam Tauber
dd34ac396c
Merge pull request #2652 from kvch/solr-engine
Add Apache Solr engine
2021-03-15 15:39:39 +01:00
Alexandre Flament
1664258061
Merge pull request #2655 from return42/fix-imports
[fix] remove unused import from yahoo-news engine
2021-03-15 08:38:34 +01:00
Alexandre Flament
5b176b3496
Merge pull request #2659 from MarcAbonce/onions-http-fix
Fix HTTP error in onion engines
2021-03-15 08:33:38 +01:00
Marc Abonce Seguin
f4a0a4d756 fix HTTP error in onion engines
regression from https://github.com/searx/searx/pull/2641
most onion websites only serve HTTP, so it must be enabled
2021-03-14 20:23:07 -07:00
Rolf
80025c3244 Windows does not support SIGUSR1, so don't use it unconditionally. 2021-03-14 19:04:36 -03:00
Markus Heiser
6e1f1085ef [fix] remove unused import from yahoo-news engine
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-03-14 15:13:57 +01:00
Markus Heiser
3703ebb22a [drop] Acgsou engine - www.acgsou.com no longer exists
- https://www.acgsou.com/ acgsou.com is redirected to 36dm.club
- @rinpatch do not plan on maintaining the engine [1]

[1] https://github.com/searx/searx/pull/1283#issuecomment-798783585

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-03-14 11:49:18 +01:00
Noémi Ványi
ff527e2681 Add Solr engine 2021-03-13 21:18:09 +01:00
Alexandre Flament
9292571304
Merge pull request #2346 from dalf/upgrade-oscar
[mod] oscar: upgrade dependencies
2021-03-13 09:29:13 +01:00
Alexandre Flament
92dd5e245e
Merge pull request #2626 from mikeri/solidtorrents
Add Solid Torrents engine
2021-03-12 19:45:22 +01:00
Alexandre Flament
a1a492baed
Merge pull request #2641 from dalf/disable_http_by_default
[mod] by default allow only HTTPS, not HTTP
2021-03-12 19:21:46 +01:00
Alexandre Flament
cb04d42806 [mod] oscar: update README.rst 2021-03-11 09:33:04 +01:00
Alexandre Flament
86912e2272 [mod] oscar: get bootstrap and typeahead from NPM 2021-03-11 09:33:01 +01:00
Alexandre Flament
44407353ef [mod] oscar: get leaflet and jquery from NPM
easy to upgrade (package.json)
2021-03-11 09:32:22 +01:00
Alexandre Flament
c7133efb12 [mod] oscar: move compiled files to the src directory 2021-03-10 19:28:51 +01:00
Alexandre Flament
eda3b513ac [mod] oscar: remove polyfills for Internet Explorer 2021-03-10 19:01:16 +01:00
Alexandre Flament
1268910274 [mod] oscar: remove unused images 2021-03-10 19:01:16 +01:00
Alexandre Flament
bdb41bea7b [mod] theme: remove require-2.1.15.min.js
See https://github.com/requirejs/requirejs/issues/1816

requirejs loads one file: leaflet.

This commit:
* removes requirejs
* load leaflet using <script src...> HTML tag in searx/templates/oscar/base.html
2021-03-10 19:01:15 +01:00
Alexandre Flament
2f3d5ec2af [mod] oscar: upgrade npm dependencies 2021-03-10 19:01:14 +01:00
Markus Heiser
96422e5c9f [fix] APKMirror engine - update xpath selectors and fix img_src
BTW: make the code slightly more readable

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-09 08:34:57 +01:00
Markus Heiser
d2faea423a [fix] rewrite Yahoo-News engine
Many things have been changed since last review of this engine.  This patch fix
xpath selectors, implements suggestion and is a complete review / rewrite of the
engine.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-03-08 11:43:34 +01:00
Alexandre Flament
99e0651cea [mod] by default allow only HTTPS, not HTTP
Related to https://github.com/searx/searx/pull/2373
2021-03-08 11:35:08 +01:00
Michael Ilsaas
5549d58de3 Add Solid Torrents engine 2021-03-07 18:14:30 +01:00
Adam Tauber
44f4a9d49a [enh] add ability to send engine data to subsequent requests 2021-03-06 12:12:35 +01:00
Alexandre Flament
87f4cc4a9e
Merge pull request #2631 from searx/update_data_update_languages.py
Update searx.data - update_languages.py
2021-03-06 10:03:00 +01:00
Markus Heiser
4845183128 [mod] don't dump traceback of SearxEngineResponseException on init
When initing engines a "SearxEngineResponseException" is logged very verbose,
including full traceback information:

    ERROR:searx.engines:yggtorrent engine: Fail to initialize
    Traceback (most recent call last):
      File "share/searx/searx/engines/__init__.py", line 293, in engine_init
        init_fn(get_engine_from_settings(engine_name))
      File "share/searx/searx/engines/yggtorrent.py", line 42, in init
        resp = http_get(url, allow_redirects=False)
      File "share/searx/searx/poolrequests.py", line 197, in get
        return request('get', url, **kwargs)
      File "share/searx/searx/poolrequests.py", line 190, in request
        raise_for_httperror(response)
      File "share/searx/searx/raise_for_httperror.py", line 60, in raise_for_httperror
        raise_for_captcha(resp)
      File "share/searx/searx/raise_for_httperror.py", line 43, in raise_for_captcha
        raise_for_cloudflare_captcha(resp)
      File "share/searx/searx/raise_for_httperror.py", line 30, in raise_for_cloudflare_captcha
        raise SearxEngineCaptchaException(message='Cloudflare CAPTCHA', suspended_time=3600 * 24 * 15)
    searx.exceptions.SearxEngineCaptchaException: Cloudflare CAPTCHA, suspended_time=1296000

For SearxEngineResponseException this is not needed.  Those types of exceptions
can be a normal use case.  E.g. for CAPTCHA errors like shown in the example
above. It should be enough to log a warning for such issues:

    WARNING:searx.engines:yggtorrent engine: Fail to initialize // Cloudflare CAPTCHA, suspended_time=1296000

closes: #2612

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05 17:26:22 +01:00
Alexandre Flament
0165e14a7f
Merge pull request #2632 from searx/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-03-05 11:59:44 +01:00
Alexandre Flament
152f6fc1da
Merge pull request #2630 from searx/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-03-05 11:59:20 +01:00
dalf
1e8b846954 Update searx.data - update_currencies.py 2021-03-05 10:56:57 +00:00
dalf
2f8a708481 Update searx.data - update_wikidata_units.py 2021-03-05 10:56:49 +00:00
dalf
d9dc3376d0 Update searx.data - update_languages.py 2021-03-05 10:56:46 +00:00
dalf
2857473553 Update searx.data - update_ahmia_blacklist.py 2021-03-05 10:56:33 +00:00
Alexandre Flament
aac37f288f
Merge pull request #2593 from dalf/update-autocomplete
Update autocomplete
2021-03-04 10:51:09 +01:00
Alexandre Flament
63f17d2e4c [enh] autocomplete refactoring, autocomplete on external bangs 2021-03-01 19:12:32 +01:00
Markus Heiser
d48e2e7b0b [enh] google scholar - python implementation of the engine
The old xpath configuration for google scholar did not work and is replaced by a
python implementation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-01 15:16:37 +01:00
Alexandre Flament
4fa1290c11 [fix] answers: don't crash when the query is an empty string 2021-03-01 10:52:39 +01:00
Alexandre Flament
e2fb500892
Merge pull request #2608 from return42/unittest2
[py2to3] use unittest from py3, remove unittest2 from py2
2021-03-01 10:05:38 +01:00
Alexandre Flament
0c663e25fc
Merge pull request #2604 from searx/update_data_firefox_version
Update searx.data - firefox_version
2021-03-01 10:03:39 +01:00
Alexandre Flament
f77983e174
Merge pull request #2602 from MarcAbonce/fix-bing-fetch-languages
Fix fetch_languages for Bing
2021-03-01 09:06:37 +01:00
GazoilKerozen
5f6ac3afa2
Add Freesound engine (#2596)
Add freesound engine with player.

Co-authored-by: Gazoil <maildeguzel@gmail.com>
2021-03-01 08:52:36 +01:00
Markus Heiser
3bae35940a [py2to3] use unittest from py3, remove unittest2 from py2
- unittest2 is a backport of the new features added to the unittest testing
  framework in Python 2.7

- unittest2 was only needed in py2 and can be dropped now

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-28 11:37:06 +01:00
Alexandre Flament
b05f4d0664
Merge pull request #2605 from searx/update_data_currencies
Update searx.data - currencies
2021-02-26 12:56:32 +01:00
Alexandre Flament
aec5188b51
Merge pull request #2606 from searx/update_data_wikidata_units
Update searx.data - wikidata_units
2021-02-26 12:55:51 +01:00
dalf
893b6e4901 Update searx.data - ahmia_blacklist 2021-02-26 08:31:15 +00:00
dalf
7b9005df31 Update searx.data - wikidata_units 2021-02-26 08:31:01 +00:00
dalf
4c8ae5b7ed Update searx.data - firefox_version 2021-02-26 08:30:45 +00:00
dalf
d2778b5efe Update searx.data - currencies 2021-02-26 08:30:45 +00:00
Marc Abonce Seguin
d6681fd33b remove articles number from engines_languages.json 2021-02-25 23:54:21 -07:00
Marc Abonce Seguin
9b6ffed061 fix fetch_languages for bing
Bing has a list of regions that it supports and some of these regions
may have more than one possible language.

In some cases, like Switzerland, these languages are always shown as
options, so there is no issue. But in other cases, like Andorra, Bing
will only show one language at the time, either the region's default or
the request's language if the latter is supported by that region.

For example, if the HTTP request is in French, Andorra will appear as
fr-AD but if the same page is requested in any other language Andorra
will appear as ca-AD.

This is specially a problem when Bing assumes that the request is in
English because it overrides enough language codes to make several major
languages like Arabic dissappear from the languages.py file.

To avoid that issue, I set the Accept-Language header to a language
that's only supported in one region to hopefully avoid these overrides.
2021-02-25 23:51:49 -07:00