Commit graph

333 commits

Author SHA1 Message Date
Markus Heiser
80aaef6c95
Merge pull request #2357 / limiter -> botdetection
The monolithic implementation of the limiter was divided into methods and
implemented in the Python package searx.botdetection.  Detailed documentation on
the methods has been added.

The methods are divided into two groups:

1. Probe HTTP headers

- Method http_accept
- Method http_accept_encoding
- Method http_accept_language
- Method http_connection
- Method http_user_agent

2. Rate limit:

- Method ip_limit
- Method link_token (new)

The (reduced) implementation of the limiter is now in the module
searx.botdetection.limiter.  The first group was transferred unchanged to this
module.  The ip_limit contains the sliding windows implemented by the limiter so
far.

This merge also fixes some long outstandig issue:

- limiter does not evaluate the Accept-Language correct [1]
- limiter needs a IPv6 prefix to block networks instead of IPs [2]

Without additional configuration the limiter works as before (apart from the
bugfixes).  For the commissioning of additional methods (link_toke), a
configuration must be made in an additional configuration file.  Without this
configuration, the limiter runs as before (zero configuration).

The ip_limit Method implements the sliding windows of the vanilla limiter,
additionally the link_token method can be used in this method.  The link_token
method can be used to investigate whether a request is suspicious. To activate
the link_token method in the ip_limit method add the following to your
/etc/searxng/limiter.toml::

    [botdetection.ip_limit]
    link_token = true


[1] https://github.com/searxng/searxng/issues/2455
[2] https://github.com/searxng/searxng/issues/2477
2023-06-03 06:00:15 +02:00
Markus Heiser
2149e88bdd [mod] template preferences: split into elements (no functional change)
HINT: this patch has no functional change / it is the preparation for following
      changes and bugfixes

Over the years, the preferences template became an unmanageable beast.  To make
the source code more readable the monolith is splitted into elements.  The
splitting into elements also has the advantage that a new template can make use
of them.

The reversed checkbox is a quirk that is only used in the prefereces and must be
eliminated in the long term.  For this the macro 'checkbox_onoff_reversed' was
added to the preferences.html template.  The 'checkbox' macro is also a quirk of
the preferences.html we don't want to use in other templates (it is an
input-checkbox in a HTML form that was misused for status display).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-02 19:05:43 +02:00
Markus Heiser
38431d2e14 [fix] correct determination of the IP for the request
For correct determination of the IP to the request the function
botdetection.get_real_ip() is implemented.  This fonction is used in the
ip_limit and link_token method of the botdetection and it is used in the
self_info plugin.

A documentation about the X-Forwarded-For header has been added.

[1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566211059

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 14:38:53 +02:00
Markus Heiser
16f0db4493 [mod] replace utils.match_language by locales.match_locale
This patch replaces the *full of magic* ``utils.match_language`` function by a
``locales.match_locale``.  The ``locales.match_locale`` function is based on the
``locales.build_engine_locales`` introduced in 9ae409a0 [1].

In the past SearXNG did only support a search by a language but not in a region.
This has been changed a long time ago and regions have been added to SearXNG
core but not to the engines.  The ``utils.match_language`` was the function to
handle the different aspects of language/regions in SearXNG core and the
supported *languages* in the engine.  The ``utils.match_language`` did it with
some magic and works good for most use cases but fails in some edge case.

To replace the concurrence of languages and regions in the SearXNG core the
``locales.build_engine_locales`` was introduced in 9ae409a0 [1].  With the last
patches all engines has been migrated to a ``fetch_traits`` and a
language/region concept that is based on ``locales.build_engine_locales``.

To summarize: there is no longer a need for the ``locales.match_language``.

[1] https://github.com/searxng/searxng/pull/1652

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Alexandre Flament
6748e8e2d5 Add "Auto-detected" as a language.
When the user choose "Auto-detected", the choice remains on the following queries.
The detected language is displayed.

For example "Auto-detected (en)":
* the next query language is going to be auto detected
* for the current query, the detected language is English.

This replace the autodetect_search_language plugin.
2023-02-17 15:17:36 +00:00
Alexandre Flament
6d72ef3cbe
Merge pull request #2109 from ahmad-alkadri/fix/highlight-full-word
Standalone words highlighting for query result in non-CJK characters
2023-01-17 23:24:04 +01:00
ahmad-alkadri
99b5272d9a A little fix and modified the testing for content highlight 2023-01-15 16:51:31 +01:00
Léon Tiekötter
0cedb1c6d8 Add search.suspended_times settings
Make suspended_time changeable in settings.yml
Allow different values to be set for different exceptions.

Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-01-15 09:00:32 +00:00
ArtikusHG
1f8f8c1e91 Replace langdetect with fasttext 2022-12-16 21:07:39 +02:00
Alexandre Flament
269326063a Fix: don't crash when engine or name is missing in settings.yml
SearXNG crashes if the engine or name fields are missing.
With this commit, the app displays an error in the log and keeps loading.

Close #1951
2022-12-04 23:43:59 +01:00
Mohamed Elashri
8d5653e60d
Merge branch 'searxng:master' into master 2022-09-30 23:06:54 +00:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
Mohamed Elashri
5832c70680
correct sci-hub links/ add .ru and remove other 3rd party domains. 2022-09-24 11:03:57 -04:00
Alexandre FLAMENT
4adc9920e9 Remove usage of SEARX environment variables 2022-08-28 17:12:57 +00:00
Markus Heiser
3b0f9c07b2 [fix] improve OpenSearch description
Some HTTP-Clients do have issues with the ``opensearch.xml`` from SearXNG
(related [1][2]) while other OpenSearch descriptions[3] (e.g. from qwant) work
flawles.

Inspired by the OpenSearch description from qwant and with informations from the
specification[4] the ``opensearch.xml`` has been *improved*.

- convert `<Url>` methods from lower case to upper case (`POST`|`GET`)
- add `<moz:SearchForm>` and `xmlns:moz="http://www.mozilla.org/2006/browser/search/"`
- add `<Query role="example" searchTerms="SearXNG" />`  [4]

  OpenSearch description documents should include at least one Query element of
  `role="example"` that is expected to return search results. Search clients may
  use this example query to validate that the search engine is working properly.

- modified `<LongName>` to SearXNG
- modified `<Description>` the word 'hackable' scares uninitiated users and was removed
- add the `type="image/png"` to `<Image>`

Test can be done by::

    make run

Visit http://127.0.0.1:8888/ and add the search engine to your WEB-Browser /
test with different WEB-Browser from desktop and Smartphones (are there any iOS
user here, please test on Safari and Chrome).

[1] https://app.element.io/#/room/#searxng:matrix.org/$xN_abdKhNqUlgXRBrb_9F3pqOxnSzGQ1TG0s0G9hQVw
[2] https://github.com/searxng/searxng/issues/431
[3] https://developer.mozilla.org/en-US/docs/Web/OpenSearch
[4] https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#the-query-element

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-11 19:04:36 +02:00
Alexandre Flament
a1e8af0796 bing.py: resolve bing.com/ck/a redirections
add a new function searx.network.multi_requests to send multiple HTTP requests at once
2022-07-08 22:02:21 +02:00
dependabot[bot]
880378f48d Bump splinter from 0.17.0 to 0.18.0
Bumps [splinter](https://github.com/cobrateam/splinter) from 0.17.0 to 0.18.0.
- [Release notes](https://github.com/cobrateam/splinter/releases)
- [Changelog](https://github.com/cobrateam/splinter/blob/master/docs/news.rst)
- [Commits](https://github.com/cobrateam/splinter/compare/0.17.0...0.18.0)

---
updated-dependencies:
- dependency-name: splinter
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-06-18 08:10:33 +02:00
Markus Heiser
ad964562ce [fix] move locale code from webapp.py to locales.py and fix #1303
To improve modularization this patch:

- moves *locale* related implementation from the webapp.py application to the
  locale.py module.

- The initialization of the locales is now done in the application (webapp) and
  is no longer done while importing searx.locales.

In the searx.locales module a new dictionary named `LOCALE_BEST_MATCH` has been
added.  In this dictionary we can map languages without a translation to
languages we have a translation for.

To fix #1303 zh-HK has been mapped to zh-Hant-TW (we do not need additional
translations of traditional Chinese)

Closes: https://github.com/searxng/searxng/issues/1303
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-12 10:52:26 +02:00
Alexandre Flament
9b3efa6d8a theme: remove __common__ 2022-05-07 19:40:48 +02:00
Markus Heiser
e8541b6006 [theme] peel out oscar from SearXNG development
This is the first step of removing oscar theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-30 13:20:27 +02:00
Markus Heiser
3bb62823ec [fix] dailymotion engine: filter by language & country
- fix the issue of fetching more the 7000 *languages*
- improve the request function and filter by language & country
- implement time_range_support & safesearch
- add more fields to the response from dailymotion (allow_embed, length)
- better clean up of HTML tags in the 'content' field.

This is more or less a complete rework based on the '/videos' API from [1].
This patch cleans up the language list in SearXNG that has been polluted by the
ISO-639-3 2 and 3 letter codes from dailymotion languages which have never been
used.

[1] https://developers.dailymotion.com/tools/

Closes: https://github.com/searxng/searxng/issues/1065
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-16 09:27:34 +02:00
Markus Heiser
effcde3d0e [fix] add missing territory (country) name
Related-to: https://github.com/searxng/searxng/pull/1029#issuecomment-1086824911
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-05 16:48:25 +02:00
Markus Heiser
6c747c40c1 [fix] issue when upgrading from werkzeug v2.0.3 to v2.1.0
In v2.1.0 werkzeug [1] fixed an issue [2] to keep relative redirect locations by
default [3].  Since relative locations are returned, we need to fix out test
cases to avoid AssertionErrors like this one::

    ======================================================================
    FAIL: test_index_html_get (tests.unit.test_webapp.ViewsTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/runner/work/searxng/searxng/tests/unit/test_webapp.py", line 105, in test_index_html_get
      self.assertEqual(result.location, 'http://localhost/search?q=test')
    AssertionError: '/search?q=test' != 'http://localhost/search?q=test'
    - /search?q=test
    + http://localhost/search?q=test

[1] https://werkzeug.palletsprojects.com/
[2] https://github.com/pallets/werkzeug/issues/2352 fixed in
[3] https://github.com/pallets/werkzeug/pull/2354

Related-to: https://github.com/searxng/searxng/pull/1039#issuecomment-1085538288
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-01 10:34:21 +02:00
Alexandre Flament
b692035120
Merge pull request #894 from dalf/simple-aria-1
[enh] simple: basic ARIA fixes
2022-03-19 12:00:03 +01:00
Markus Heiser
b1912607ae [mod] replace /help by /info pages and include pages in project docs
This patch implements a bolierplate to share content from info-pages of the
SearXNG instance (URL /info) with the project documentation (path /docs/user).

The info pages are using Markdown (CommonMark), to include them in the project
documentation (reST) the myst-parser [1] is used in the Sphinx-doc build chain.

If base_url is known (defined in settings.yml) links to the instance are also
inserted into the project documentation::

    searxng_extra/docs_prebuild

[1] https://www.sphinx-doc.org/en/master/usage/markdown.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12 11:36:31 +01:00
Alexandre Flament
a250ddadb9 [enh] simple: basic ARIA fixes 2022-03-08 19:37:21 +00:00
Alexandre Flament
1c7d8815fb [mod] URL for the static file contains the sha1
* allow to cache the static file forever
* avoid bugs when the static files are updated but not reloaded
2022-03-05 10:50:48 +00:00
mrpaulblack
21e3c40516 [simple theme] replace Image_layout.js with flexbox CS impl.
* drop image_layout.js from simple theme
* move image_layout.js to oscar theme and delete common js dir (since its empty now)
* align top position of image detail modal with bottom position of search header
* use flexbox to display images; row height can be set via @results-image-row-height in defenitions.less
* display span title underneath each image with a max width of 12rem
* increase margin and padding around image article on desktop and tablet
* make article height smaller on phone layout (height of 6rem) to display more content on current view
* remove content from result, if the title and content matches
* use a group that cotains the flex image article, if images are mixed with other categories
* fix pylint issues in webapp.py
* use the default.html result template in unit tests (thanks @return42)
2022-02-26 22:31:47 +01:00
Markus Heiser
fdeece0aae [fix] changeover of the unit tests to the simple theme
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-20 18:51:12 +01:00
Alexandre Flament
bf987bb608
Merge pull request #848 from not-my-profile/help-route
Introduce `/help` route
2022-02-05 08:52:19 +01:00
Markus Heiser
24e2acf88e
Merge pull request #839 from dalf/docs_searx_utils
[mod] add documentation about searx.utils
2022-02-04 21:28:27 +00:00
Martin Fischer
07936060d1 preferences: Set autocomplete=off for form
Otherwise you can change the value of a select,
refresh the page and the preferences stay changed,
leaving the wrong impression that they were saved.
2022-02-04 21:11:29 +01:00
Martin Fischer
fb9eedbf40 [enh] introduce /help route
Translation will be implemented in the future.
For now the "en" in /help/en/<pagename> is hardcoded.
2022-02-01 06:28:26 +01:00
Martin Fischer
b93711b45d [help] remove link from about.md title
Now that about.html extends page_with_header.html
it already has a link to the start page and removing
the link makes it easier to extract the page title
from the Markdown for the following commit.
2022-02-01 06:28:26 +01:00
Alexandre Flament
0eacc46ee3 [mod] add documentation about searx.utils
This module is a toolbox for the engines.
Is should be documented.

In addition, searx/utils.py is checked by pylint.
2022-01-29 22:49:42 +01:00
Martin Fischer
fdf562bc32 [typing] add results.Timing 2022-01-17 11:42:48 +01:00
Markus Heiser
7cdd31440e [fix] external bangs: don't overwrite Bangs in data trie
Bangs with a `*` suffix (e.g. `!!d*`) overwrite Bangs with the same
prefix (e.g. `!!d`) [1].  This can be avoid when a non printable character is
used to tag a LEAF_KEY.

[1] https://github.com/searxng/searxng/pull/740#issuecomment-1010411888

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-12 19:37:13 +01:00
Martin Fischer
e12525a1fa
Merge pull request #708 from not-my-profile/pref-refactor
Refactor `preferences`
2022-01-07 09:45:23 +01:00
Martin Fischer
180d4d068b [refactor] refactor SwitchableSetting
The previous implementation used two hash sets and a list.
... that's not necessary ... a single hash map suffices.

And it's also less error prone ... because the previous data structure
allowed a setting to be enabled and disabled at the same time.
2022-01-06 18:56:02 +01:00
Martin Fischer
bb06758a7b [refactor] add type hints & remove Setting._post_init
Previously the Setting classes used a horrible _post_init
hack that prevented proper type checking.
2022-01-06 14:21:14 +01:00
Martin Fischer
61935c72ef [fix] remove broken ? search operator
The ? search operator has been broken for some time and
currently only raises the question why it's still there.

## Context ##

The query "Paris !images" searches for "Paris" in the "images" category.

Once upon a time Searx supported "Paris ?images" to search for "Paris"
in the currently enabled categories and the "images" category.

The feature makes sense ... the ? syntax does not.
We will hopefully introduce a +!images syntax in the future.

Fixes #702.
2022-01-06 14:10:58 +01:00
Alexandre Flament
aedd6279b3
Merge pull request #634 from not-my-profile/powered-by
Introduce `categories_as_tabs` & group engines in tabs
2022-01-06 09:22:02 +01:00
Alexandre Flament
9004e84b6e
Merge pull request #596 from dalf/upgrade-httpx
Upgrade httpx
2022-01-05 19:13:48 +01:00
Alexandre Flament
e64c3deab7 [mod] upgrade httpx 0.21.2
httpx 0.21.2 and httpcore 0.14.4 fix multiple issues:
* https://github.com/encode/httpx/releases/tag/0.21.2
* https://github.com/encode/httpcore/releases/tag/0.14.4

so most of the workarounds in searx.network have been removed.
2022-01-05 18:46:00 +01:00
Martin Fischer
a4c2cfb837 [enh] change categories_as_tabs from a list to a dict
The tab icon names are currently hard coded in the templates.
This commit lets us introduce an icon property in the future, e.g:

categories_as_tabs:
  general:
    icon: search-outline
2022-01-05 11:03:44 +01:00
Martin Fischer
8e9ad1ccc2 [enh] introduce categories_as_tabs
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.

This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs  can still be searched with !bangs.
2022-01-03 07:01:49 +01:00
Alexandre Flament
76cbfbbdda reference docs.searxng.org 2022-01-02 21:18:29 +01:00
Alexandre Flament
7d4834ac4d [mod] webutils.py: remove dead code
secret_key can't be bytes (see settings_default.py)
2021-12-28 10:14:38 +01:00
Markus Heiser
d84226bf63 [fix] issues reported by pylint
Fix pylint issues from commit (3d96a983)

    [format.python] initial formatting of the python code

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 10:16:20 +01:00
Markus Heiser
3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:26:22 +01:00