Commit graph

2786 commits

Author SHA1 Message Date
Markus Heiser
c6a5cc019a [brand] searxng is a fork from searx
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 11:42:42 +02:00
Markus Heiser
6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
spongebob33
7528e38c8a add core.ac.uk engine 2021-04-24 08:55:45 +02:00
Alexandre Flament
525bfe7830
Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament
7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament
09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Markus Heiser
c54bf42cb9 [mod] metrics: add to pylint and justify indentations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 06:58:49 +02:00
Alexandre Flament
65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00
Alexandre Flament
3755cf2eba More branding 2021-04-22 21:10:22 +02:00
Alexandre Flament
04b34c70b8 More branding 2021-04-22 20:14:16 +02:00
Alexandre Flament
9d84025e4f searxng brand 2021-04-22 19:57:02 +02:00
Alexandre Flament
d01741c9a2
Merge pull request #15 from return42/add-springer
Add a search engine for Springer Nature
2021-04-22 13:23:31 +02:00
Markus Heiser
213041adc7 [enh] themes - add result.metadata to result sub header
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-22 12:37:18 +02:00
Pierre Chevalier
a80bf1ba97 [enh] Add Springer Nature engine
Springer Nature is a global publisher dedicated to providing service to research
community [1] with official API [2].

To test this PR, first get your API key following this page:

   https://dev.springernature.com/signup

In searx/engines/springer.py at line 24, add this API key.  I left my own key,
commented out in the line aboce.  Feel free to use it, if needed.

[1] https://www.springernature.com/
[2] https://dev.springernature.com/
2021-04-22 12:35:25 +02:00
habsinn
41a2e3785e [enh] add engine using API from "The Art Institute of Chicago" 2021-04-22 12:25:43 +02:00
Alexandre Flament
981cdb7acf
Merge pull request #10 from return42/fix-youtube
[fix] youtube - send CONSENT Cookie to not be redirected
2021-04-22 12:21:31 +02:00
Markus Heiser
9a05ad4c73 [enh] add DOI resolver from sci-hub / replace default DOI
The new sci-hub URLs are comming from @aurora-vasiliev [1].

[1] https://github.com/searx/searx/pull/2706

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 12:13:01 +02:00
Markus Heiser
e9a6ab4015 [fix] youtube - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, youtube requests are redirected to confirm and get a CONSENT
Cookie from https://consent.youtube.com

This patch adds a CONSENT Cookie to the youtube request to avoid redirection.

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Reported-by: https://github.com/searx/searx/issues/2774
2021-04-22 12:09:09 +02:00
Markus Heiser
247c46c6b0 [drop] plugin HTTPS rewrite - rule base is to old (update is to big)
The usefulness of the _HTTPS rewrite_ plugin is questionable:

- the 36 rule files have not been updated since 2015 [1]
- actual there are 23760 rule files in the https-everywhere repo [2]

For the first, we can remove this plugin.  For a complete new implementation, it
might be good to know that there is a project "https-everywhere : Privacy for
Pythons" [3]

related:  https://github.com/return42/searx-next/issues/8

[1] https://github.com/return42/searx-next/tree/d187a1d/searx/plugins/https_rules
[2] https://github.com/EFForg/https-everywhere/tree/master/src/chrome/content/rules
[3] https://github.com/jayvdb/https-everywhere-py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 11:47:16 +02:00
Alexandre Flament
c6d5605d27
Merge pull request #7 from searxng/metrics
Metrics
2021-04-22 08:34:17 +02:00
Alexandre Flament
baff1cbbab fix issues from review 2021-04-21 18:49:13 +02:00
Alexandre Flament
b7848e3422 [fix] searxng fix: sjp engine 2021-04-21 16:31:29 +02:00
Alexandre Flament
7cfd8d900a [mod] oscar: /preferences , engines tab: report engine times
* display the median time instead of the average.
* add a "Reliability" column (sum up the metrics and the checker results).
* the "selected language", "SafeSearch", "Time range" values are displayed as "broken" when the checker tests fail.
2021-04-21 16:24:46 +02:00
Alexandre Flament
c27fef1cde [mod] metrics: add secondary parameter
Some error won't stop the engine:
* additional HTTP redirects for example
* some invalid results

secondary=True allows to flag these errors as not important.
2021-04-21 16:24:46 +02:00
Alexandre Flament
7acd7ffc02 [enh] rewrite and enhance metrics 2021-04-21 16:24:46 +02:00
Alexandre Flament
aae7830d14 [mod] refactoring: processors
Report to the user suspended engines.

searx.search.processor.abstract:
* manages suspend time (per network).
* reports suspended time to the ResultContainer (method extend_container_if_suspended)
* adds the results to the ResultContainer (method extend_container)
* handles exceptions (method handle_exception)
2021-04-21 16:24:46 +02:00
Alexandre Flament
ae5954f2da Merge remote-tracking branch 'return42/fix-preference-save' 2021-04-21 16:24:22 +02:00
Alexandre Flament
624d3ec6db Merge remote-tracking branch 'return42/fix-url-bar-suggestion' 2021-04-21 16:23:52 +02:00
Alexandre Flament
92db0227b1 Merge remote-tracking branch 'dalf/oscar-images' 2021-04-19 09:36:54 +02:00
Alexandre Flament
48720e20a8 Merge remote-tracking branch 'searx/master' 2021-04-19 09:35:12 +02:00
Noémi Ványi
8362257b9a
Merge pull request #2736 from plague-doctor/sjp
Add new engine: SJP - Słownik języka polskiego
2021-04-16 17:30:14 +02:00
Noémi Ványi
e56323d3c8
Merge pull request #2759 from ypid/fix/typo
Fix grammar mistake in debug log output
2021-04-16 17:26:45 +02:00
Plague Doctor
d275d7a35e Code refactoring. 2021-04-16 12:23:27 +10:00
Markus Heiser
f637bfc635 [mod] oscar's "default" template should make use of result.thumbnail
Some engine do have set result.img_src, other return a result.thumbnail.  If
result.img_src is unset and a result.thumbnail is given, show it to the UI.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-15 08:52:11 +02:00
Markus Heiser
062d589f86 [fix] xpath expressions to grap all items from bandcamp's response
I also found some items missing a thumbnail and I used text_extract for content
and title, to remove unneeded whitespaces.

BTW: added bandcamp's favicon

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-15 08:52:11 +02:00
Kyle Anthony Williams
4d3c399ee9 [feat] add bandcamp engine 2021-04-15 08:52:11 +02:00
Alexandre Flament
d14994dc73 [httpx] replace searx.poolrequests by searx.network
settings.yml:

* outgoing.networks:
   * can contains network definition
   * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
     keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
   * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
   * local_addresses can be "192.168.0.1/24" (it supports IPv6)
   * support_ipv4 & support_ipv6: both True by default
     see https://github.com/searx/searx/pull/1034
* each engine can define a "network" section:
   * either a full network description
   * either reference an existing network

* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
2021-04-12 17:25:56 +02:00
Markus Heiser
c4793afadc [fix] https-scheme missing in preferences-page
This patch is an addition to PR #2656 which removed all usage of `base_url` from
the templates, except one was forgotten in the cookie URL of the preferences.

closes: 2740

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12 16:02:16 +02:00
Markus Heiser
2bf297b19f [fix] redirect when saving preferences
Erroneously commit 87e4c4762 droped the 302 redirect.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12 15:29:08 +02:00
Robin Schneider
dfc66ff0f0
Fix grammar mistake in debug log output 2021-04-11 22:12:53 +02:00
Alexandre Flament
eaa694fb7d [enh] replace requests by httpx 2021-04-10 15:38:33 +02:00
Plague Doctor
599ff39ddf Fix conflicts 2021-04-09 06:54:03 +10:00
Noémi Ványi
cc359345a8
Merge pull request #2735 from plague-doctor/wordnik
Add new engine: Wordnik.com
2021-04-08 19:48:13 +02:00
Plague Doctor
6631f11305 Add new engine: SJP 2021-04-08 10:21:54 +10:00
Plague Doctor
7035bed4ee Add new engine: Wordnik.com 2021-04-08 09:58:00 +10:00
Alexandre Flament
c09ff4faf2 [fix] fix PR 2656
SCRIPT_NAME remove trailing slash to avoid infinite redirect
2021-04-07 13:05:55 +02:00
Noémi Ványi
07f5edce3d Add Meilisearch engine
Website: https://www.meilisearch.com/
2021-04-06 21:57:05 +02:00
Noémi Ványi
a477a3a687
Merge pull request #2728 from return42/fix-loader
[fix] settings_loader.py - use update_dict only for mapping types
2021-04-05 21:32:41 +02:00
3nprob
2ca0aa4f29 Remove hubsbpot tracking URL params
More information here:
https://knowledge.hubspot.com/settings/how-do-i-create-a-tracking-url
https://knowledge.hubspot.com/ads/ad-tracking-in-hubspot
https://knowledge.hubspot.com/ctas/calls-to-action-frequently-asked-questions
https://meta.stackexchange.com/questions/263392/what-are-these-very-long-parameters-in-so-careers-feedback-always-welcome-url
2021-04-06 02:11:09 +09:00