Commit graph

24 commits

Author SHA1 Message Date
Markus Heiser
7c4e4ebd40 [log] warning with URL in case of 'raise_for_httperror'
In order to be able to implement error handling, it is necessary to know which
URL triggered the exception / the URL has not yet been logged.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-11-23 11:33:19 +01:00
Markus Heiser
542f7d0d7b [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.

- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two

  1. ./searx/engines -> lint engines with additional builtins
  2. ./searx ./searxng_extra ./tests -> lint all other python files

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11 14:55:38 +01:00
Markus Heiser
50d5a9ff60 [fix] issues reported by pylint 3.1.0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-09 09:28:13 +01:00
Alexandre Flament
b4e4cfc026 Bump httpx 0.21.2 from to 0.24.1 2023-08-21 22:05:12 +02:00
Evhorizon
1517724615
Update network.py 2022-11-06 20:35:30 +01:00
Alexandre Flament
32e8c2cf09 searx.network: add "verify" option to the networks
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.

SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
2022-10-14 13:59:22 +00:00
Alexandre Flament
a1e8af0796 bing.py: resolve bing.com/ck/a redirections
add a new function searx.network.multi_requests to send multiple HTTP requests at once
2022-07-08 22:02:21 +02:00
Léon Tiekötter
0cbf73a1f4
Allow 'using_tor_proxy' to be set for each engine individually
Check 'using_tor_proxy' for each engine individually instead of checking globally

[fix] searx.network: update _rdns test to the last httpx version

Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-01-27 22:37:02 +01:00
Martin Fischer
def62c3a47 [typing] add type hints for dictionaries 2022-01-17 11:42:48 +01:00
Alexandre Flament
e64c3deab7 [mod] upgrade httpx 0.21.2
httpx 0.21.2 and httpcore 0.14.4 fix multiple issues:
* https://github.com/encode/httpx/releases/tag/0.21.2
* https://github.com/encode/httpcore/releases/tag/0.14.4

so most of the workarounds in searx.network have been removed.
2022-01-05 18:46:00 +01:00
Markus Heiser
3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:26:22 +01:00
Alexandre Flament
f9c6393502 [enh] verify that Tor proxy works every time searx starts
based on @MarcAbonce commit on searx
2021-10-12 21:01:02 +02:00
Alexandre Flament
dc74df3a55
Merge pull request #261 from dalf/upgrade_httpx
[upd] upgrade httpx 0.19.0
2021-09-17 11:48:37 +02:00
Markus Heiser
443bf35e09 [pylint] fix global-variable-not-assigned issues
If there is no write access, there is no need for global.  Remove global
statement if there is no assignment.

global-variable-not-assigned:
  Using global for names but no assignment is done Used when a variable is
  defined through the "global" statement but no assignment to this variable is
  done.

In Pylint 2.11 the global-variable-not-assigned checker now catches global
variables that are never reassigned in a local scope and catches (reassigned)
functions [1][2]

[1] https://pylint.pycqa.org/en/latest/whatsnew/2.11.html
[2] https://github.com/PyCQA/pylint/issues/1375

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17 10:14:27 +02:00
Alexandre Flament
8e73438cbe [upd] upgrade httpx 0.19.0
adjust searx.network module to the new internal API
see https://github.com/encode/httpx/pull/1522
2021-09-17 10:06:22 +02:00
Alexandre Flament
91a6d80e82 [mod] debug mode: log HTTP requests with network name
For example wikipedia requests use the logger name "searx.network.wikipedia"

Log is disable when searx_debug is False
2021-09-11 10:13:14 +02:00
Markus Heiser
2a3b9a2e26 [pylint] searx: drop no longer needed 'missing-function-docstring'
Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914168470
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 13:34:35 +02:00
Alexandre Flament
3b0f70ed0f [mod] /image_proxy: use HTTP/1 instead of HTTP/2
httpx: HTTP/2 is slow when a lot data is downloaded.
https://github.com/dalf/pyhttp-benchmark

also, the usage of HTTP/1 decreases the load average
2021-08-24 14:51:20 +02:00
Alexandre Flament
4b07df62e5 [mod] move all default settings into searx.settings_defaults 2021-06-01 08:10:15 +02:00
Markus Heiser
2128022f72 [coding-style] searx/network/network.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:48:47 +02:00
Markus Heiser
44efa911ba [pylint] searx/network/network.py & add global (NETWORKS)
No functional change!

- fix messages from pylint
- add ``global NETWORKS``

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:46 +02:00
Alexandre Flament
283ae7bfad [fix] searx.network: fix rare cases where LOOP is None
* searx.network.client.LOOP is initialized in a thread
* searx.network.__init__ imports LOOP which may happen
  before the thread has initialized LOOP

This commit adds a new function "searx.network.client.get_loop()"
to fix this issue
2021-04-27 17:47:36 +02:00
Alexandre Flament
aae7830d14 [mod] refactoring: processors
Report to the user suspended engines.

searx.search.processor.abstract:
* manages suspend time (per network).
* reports suspended time to the ResultContainer (method extend_container_if_suspended)
* adds the results to the ResultContainer (method extend_container)
* handles exceptions (method handle_exception)
2021-04-21 16:24:46 +02:00
Alexandre Flament
d14994dc73 [httpx] replace searx.poolrequests by searx.network
settings.yml:

* outgoing.networks:
   * can contains network definition
   * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
     keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
   * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
   * local_addresses can be "192.168.0.1/24" (it supports IPv6)
   * support_ipv4 & support_ipv6: both True by default
     see https://github.com/searx/searx/pull/1034
* each engine can define a "network" section:
   * either a full network description
   * either reference an existing network

* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
2021-04-12 17:25:56 +02:00