mirrors/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-11-23 11:21:00 +00:00
Code Issues Projects Releases Wiki Activity
searxng/searx/engines
History
Markus Heiser 9328c66e93 [fix] google news - send CONSENT Cookie to not be redirected 
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...

This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.

The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..

Using common google search from different domains::

    google.com:        CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
    google.de:         CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    google.fr:         CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826

When searching about videos (google-videos)::

    google.es:         CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
    google.de:         CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171

Google news has only one domain for all languages::

    news.google.com:   CONSENT=YES+cb.{{date}}-14-p0.de+FX+816

Using google-scholar search from different domains::

    scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    scholar.google.fr: does not use such a cookie / did not ask the user
    scholar.google.es: does not use such a cookie / did not ask the user

Interim summary:

  Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
  More experience is need before we generalize the CONSENT cookies over all
  google engines.

Related:

- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 13:21:20 +02:00
..
1337x.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
__init__.py [fix] sys.exit(1) when there is duplicate engine name 2021-06-01 16:37:20 +02:00
ahmia.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
apkmirror.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
archlinux.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
artic.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
arxiv.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
bandcamp.py [fix] xpath expressions to grap all items from bandcamp's response 2021-04-15 08:52:11 +02:00
base.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
bing.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
bing_images.py Fix paging of Bing Images 2021-02-22 21:19:34 +01:00
bing_news.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
bing_videos.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
btdigg.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
ccengine.py Add Creative Commons search engine 2021-02-04 11:31:35 +01:00
command.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
core.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
currency_convert.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
dailymotion.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
deezer.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
demo_offline.py [docs] add engine "Demo Offline Engine" 2021-06-04 15:04:38 +02:00
demo_online.py [docs] add engine "Demo Online Engine" 2021-06-04 15:05:58 +02:00
deviantart.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
dictzone.py [fix] typo: online_dictionnary --> online_dictionary 2021-06-04 15:05:58 +02:00
digbt.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
digg.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
docker_hub.py [enh] add engine - Docker Hub 2021-05-30 15:18:36 +02:00
doku.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
duckduckgo.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
duckduckgo_definitions.py [pylint] searx/engines/duckduckgo_definitions.py 2021-06-14 09:22:29 +02:00
duckduckgo_images.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
duden.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
dummy-offline.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
dummy.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
ebay.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
elasticsearch.py [enh] replace requests by httpx 2021-04-10 15:38:33 +02:00
etools.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
fdroid.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
flickr.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
flickr_noapi.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
framalibre.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
freesound.py Add Freesound engine (#2596) 2021-03-01 08:52:36 +01:00
frinkiac.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
genius.py [fix] engine genius should not use the video template 2021-05-24 16:31:14 +02:00
gentoo.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
gigablast.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
github.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
google.py [mod] google - get_lang_info add documentataion & comments 2021-06-11 16:06:36 +02:00
google_images.py [fix] log messages from: google- images, news, scholar, videos 2021-06-11 16:31:50 +02:00
google_news.py [fix] google news - send CONSENT Cookie to not be redirected 2021-06-18 13:21:20 +02:00
google_scholar.py [fix] log messages from: google- images, news, scholar, videos 2021-06-11 16:31:50 +02:00
google_videos.py [fix] log messages from: google- images, news, scholar, videos 2021-06-11 16:31:50 +02:00
ina.py [fix] make ina engine compatible with the new response json 2021-03-25 01:20:41 +01:00
invidious.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
json_engine.py [mod] json_engine: add content_html_to_text and title_html_to_text 2021-02-10 16:42:11 +01:00
kickass.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
loc.py Add Library of Congress engine 2021-02-09 12:45:39 +01:00
mediathekviewweb.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
mediawiki.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
meilisearch.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
microsoft_academic.py fix Microsoft Academic engine 2021-03-15 20:21:28 +01:00
mixcloud.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
mysql_server.py [enh] add MySQL engine 2021-05-28 17:36:46 +02:00
not_evil.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
nyaa.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
opensemantic.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
openstreetmap.py [fix] openstreetmap - fix some minor whitespace & indentation issues 2021-06-09 18:08:23 +02:00
pdbe.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
peertube.py [fix] peertube fetch supported languages 2021-06-04 16:17:20 +02:00
photon.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
piratebay.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
postgresql.py [enh] add PostgreSQL engine 2021-05-28 17:34:44 +02:00
pubmed.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
qwant.py fix Qwant's fetch_languages function 2021-05-02 17:46:40 -07:00
recoll.py [feat] recoll: paged json support 2021-02-07 10:05:35 -03:00
reddit.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
redis_server.py [fix] enigine redis - avoid error when the engine is loaded 2021-06-02 09:54:58 +02:00
rumble.py Added rumble.com video search engine. TODO video embedding. 2021-02-20 12:48:56 +00:00
scanr_structures.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
searchcode_code.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
searx_engine.py [brand] searxng -- fix links to issue tracker & WEB-GUI 2021-04-25 14:25:08 +02:00
semantic_scholar.py Remove duplicated key from dict in Semantic Scholar 2021-03-27 16:58:32 +01:00
sepiasearch.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
seznam.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
sjp.py [fix] searxng fix: sjp engine 2021-04-21 16:31:29 +02:00
solidtorrents.py [fix] URL to solidtorrent result page 2021-04-29 10:40:47 +02:00
solr.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
soundcloud.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
spotify.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
springer.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
sqlite.py [enh] add offline engine for sqlite database 2021-05-27 14:27:11 +02:00
stackoverflow.py [enh] replace requests by httpx 2021-04-10 15:38:33 +02:00
startpage.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
tokyotoshokan.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
torrentz.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
translated.py [fix] typo: online_dictionnary --> online_dictionary 2021-06-04 15:05:58 +02:00
unsplash.py [fix] unsplash engine - 'searx:result: invalid title:' 2021-05-25 17:26:58 +02:00
vimeo.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
wikidata.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
wikipedia.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
wolframalpha_api.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
wolframalpha_noapi.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
wordnik.py [httpx] replace searx.poolrequests by searx.network 2021-04-12 17:25:56 +02:00
www1x.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
xpath.py [enh] XPath engine - add time safe-search support 2021-05-23 22:26:18 +02:00
yacy.py [enh] replace requests by httpx 2021-04-10 15:38:33 +02:00
yahoo.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
yahoo_news.py [pylint] tag PYLINT_FILES by comment # lint: pylint 2021-04-26 20:18:20 +02:00
yggtorrent.py [mod] the bittorent search engines are available only in the files category 2021-05-29 16:14:19 +02:00
youtube_api.py [mod] dynamically set language_support variable 2021-02-01 17:10:37 +01:00
youtube_noapi.py [fix] youtube - send CONSENT Cookie to not be redirected 2021-04-22 12:09:09 +02:00