SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled.
Find a file
Markus Heiser 2499899554 [mod] Google: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.

When ever possible the implementations of the Google engines try to make use of
the async REST APIs.  The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - Google (WEB),
  - Google images,
  - Google news,
  - Google scholar and
  - Google videos

  and remove data from obsolete data type "supported_languages".

  A traits.custom type that maps region codes to *supported_domains* is fetched
  from https://www.google.com/supported_domains

searx/autocomplete.py:
  Reversed engineered autocomplete from Google WEB.  Supports Google's languages and
  subdomains.  The old API suggestqueries.google.com/complete has been replaced
  by the async REST API: https://{subdomain}/complete/search?{args}

searx/engines/google.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - always use the async REST API (formally known as 'use_mobile_ui')
  - use *supported_domains* from traits
  - improved the result list by fetching './/div[@data-content-feature]'
    and parsing the type of the various *content features* --> thumbnails are
    added

searx/engines/google_images.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - if exists, freshness_date is added to the result
  - issue 1864: result list has been improved a lot (due to the new cr parameter)

searx/engines/google_news.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
    *supported_domains* is not needed but a ceid list has been added.
  - different region handling compared to Google WEB
  - fixed for various languages & regions (due to the new ceid parameter) /
    avoid CONSENT page
  - Google News do no longer support time range
  - result list has been fixed: XPath of pub_date and pub_origin

searx/engines/google_videos.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - add paging support
  - implement a async request ('asearch': 'arc' & 'async':
    'use_ac:true,_fmt:html')
  - simplified code (thanks to '_fmt:html' request)
  - issue 1359: fixed xpath of video length data

searx/engines/google_scholar.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - request(): include patents & citations
  - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
  - hardening XPath to iterate over results
  - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
  - issue 1769 fixed: new request implementation is no longer incompatible

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
.devcontainer Add GitHub Codespaces configuration 2023-03-21 20:58:13 +00:00
.github [mod] replace engines_languages.json by engines_traits.json 2023-03-24 10:37:42 +01:00
.vscode Add GitHub Codespaces configuration 2023-03-21 20:58:13 +00:00
dockerfiles Let uWSGI workers and threads be adjustable from env 2023-01-11 14:53:36 +01:00
docs [mod] Google: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
examples Fix whitespaces 2016-07-11 18:52:37 +07:00
searx [mod] Google: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
searxng_extra [mod] replace searx.languages by searx.sxng_locales 2023-03-24 10:37:42 +01:00
src/brand [simple] ImageLayout.watch: img_load_error.svg if img load fails 2021-11-29 21:10:13 +01:00
tests Add "Auto-detected" as a language. 2023-02-17 15:17:36 +00:00
utils [mod] Google: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
.coveragerc [mod] use github actions instead of travis 2020-11-17 15:09:06 +01:00
.dir-locals.el [emacs] .dir-locals.el: add Language Server Protocol Support (LSP) 2023-02-11 13:36:53 +01:00
.dockerignore [fix] tidy up ignore lists .gitignore & .dockerignore 2021-06-22 16:55:30 +02:00
.editorconfig [enh] add .editorconfig 2023-03-20 12:29:12 +01:00
.gitattributes [fix] update .gitattributes 2021-06-22 20:34:39 +02:00
.gitignore SearXNG version: fix make docker 2023-01-20 10:46:00 +00:00
.nvmrc Node: update to node 16.15.1 2022-06-25 14:07:39 +02:00
.pylintrc [mod] make python code pylint 2.16.1 compliant 2023-02-10 13:59:21 +01:00
.weblate weblate: migration to https://translate.codeberg.org/ 2023-01-21 15:45:12 +00:00
.yamllint.yml [enh] add test.yamllint - lint yaml files 2021-06-05 17:41:24 +02:00
AUTHORS.rst [doc] add example for enabling an engine disabled by default 2023-02-12 18:33:38 +01:00
babel.cfg [fix] jinja/babel: WithExtension and AutoEscapeExtension are built-in now. 2022-03-25 09:42:12 +01:00
CHANGELOG.rst [fix] typos / reported by @kianmeng in searx PR-3366 2022-09-27 18:32:14 +02:00
CONTRIBUTING.md reference docs.searxng.org 2022-01-02 21:18:29 +01:00
Dockerfile SearXNG version: fix make docker 2023-01-20 10:46:00 +00:00
LICENSE [fix] full AGPLv3+ license according to #382 2015-07-04 18:23:54 +02:00
Makefile [mod] install developer tools by: make node.env.dev 2023-02-11 13:33:43 +01:00
manage [mod] replace engines_languages.json by engines_traits.json 2023-03-24 10:37:42 +01:00
package.json [mod] install developer tools by: make node.env.dev 2023-02-11 13:33:43 +01:00
PULL_REQUEST_TEMPLATE.md Add PR template and contribution guidelines 2020-07-10 17:10:02 +02:00
pyrightconfig-ci.json [mod] add test.pyright to test & ci.test targets 2022-01-23 08:00:39 +01:00
pyrightconfig.json [fix] pyrightconfig.json include only dedicated folders in the test 2022-01-23 08:00:39 +01:00
README.rst Add GitHub Codespaces configuration 2023-03-21 20:58:13 +00:00
requirements-dev.txt Merge pull request #2280 from searxng/dependabot/pip/master/linuxdoc-20230321 2023-03-24 10:22:08 +01:00
requirements.txt Revert "Bump redis from 4.5.1 to 4.5.3" 2023-03-24 10:12:31 +01:00
SECURITY.md [enh] add security policy 2022-01-25 00:56:20 +01:00
setup.py [mod] replace /help by /info pages and include pages in project docs 2022-03-12 11:36:31 +01:00



Privacy-respecting, hackable metasearch engine

Searx.space lists ready-to-use running instances.

A user, admin and developer handbook is available on the homepage.

SearXNG install SearXNG homepage SearXNG wiki AGPL License Issues commits weblate SearXNG logo


Contact

Ask questions or just chat about SearXNG on

IRC

#searxng on libera.chat which is bridged to Matrix.

Matrix

#searxng:matrix.org

Differences to searx

SearXNG is a fork of searx, with notable changes:

User experience

  • Reworked (and still simple) theme:
    • Usable on desktop, tablet and mobile.
    • Light and dark versions (available in the preferences).
    • Right-to-left language support.
    • Screenshots
  • The translations are up to date, you can contribute on Weblate
  • The preferences page has been updated:
    • Browse which engines are reliable or not.
    • Engines are grouped inside each tab.
    • Each engine has a description.
  • Thanks to the anonymous metrics, it is easier to report malfunctioning engines, so they get fixed quicker
  • Administrators can block and/or replace the URLs in the search results

Setup

  • No need for Morty to proxy images, even on a public instance.
  • No need for Filtron to block bots, as there is now a built-in limiter.
  • A well maintained Docker image, now also built for ARM64 and ARM/v7 architectures. (Alternatively there are up to date installation scripts.)

Contributing

  • Readable debug log.
  • Contributing is easier, thanks to the Development Quickstart guide.
  • A lot of code cleanup and bugfixes.
  • Up to date list dependencies.

Translations

Help translate SearXNG at Weblate

Codespaces

You can contribute from your browser using GitHub Codespaces:

  • Fork the repository
  • Click on the <> Code green button
  • Click on the Codespaces tab instead of Local
  • Click on Create codespace on master
  • VSCode is going to start in the browser
  • Wait for git pull && make install to appears and then to disapear
  • You have 120 hours per month (see also your list of existing Codespaces)
  • You can start SearXNG using make run in the terminal or by pressing Ctrl+Shift+B.

Donations

Support the SearXNG project by clicking the donation page: https://docs.searxng.org/donate.html

Thank you :)