Commit graph

2856 commits

Author SHA1 Message Date
Robert O'Connor
dae5bcf253
some other minor tweaks to move stuff very unlikely to change to the top in efforts to delay cache invalidation 2019-08-18 18:26:40 -04:00
Robby O'Connor
906320d271
add --no-cache to package installs
Add --no-cache to apk upgrade and apk add calls.
2019-08-18 18:20:58 -04:00
Alexandre Flament
8e62f7600f
Merge pull request #1647 from dalf/no-inline-script
No inline script for oscar and simple theme
2019-08-06 12:25:25 +02:00
Dalf
6fbba63c83 oscar and simple themes: no inline script (allow A+ Content-Security-Policy) 2019-08-06 12:14:11 +02:00
Dalf
90b24f1060 manage.sh grunt_build: add node_modules/.bin to the path 2019-08-06 11:49:46 +02:00
Frank de Lange
e4e8e6da4c Add image format and source information to display (#1567)
Add image format and source information to display - needs changes to engines to actually display something. 

Displays result.source (website from which the image was taken) and result.img_format (image type and size).

Result is styled with result-format and result-source classes. See PR #1566 for an example of an engine which has the necessary changes.

Strip <span class="highlight">...</span> in the oscar image template.
2019-08-06 09:31:24 +02:00
Léo Bourrel
88261e111c Fix bing engine results count (#1387)
This PR fixes the result count from bing which was throwing an (hidden) error and add a validation to avoid reading more results than avalaible.

For example :
If there is 100 results from some search and we try to get results from 120 to 130, Bing will send back the results from 0 to 10 and no error. If we compare results count with the first parameter of the request we can avoid this "invalid" results.
2019-08-05 16:15:40 +02:00
Alexandre Flament
12f891da84
Merge pull request #1669 from dalf/engine-fixes
Engine fixes
2019-08-05 15:57:33 +02:00
Dalf
1cee2c1796 [fix] bing engine
before this commit, sometimes there are no results
use a generic user-agent instead of one with the OS "Windows NT 6.3; WOW64"
2019-08-05 15:46:40 +02:00
Dalf
9a378ca8c6 [fix] etymonline, voat engines 2019-08-05 15:44:22 +02:00
Dalf
fcc9587ee9 [fix] fdroid engine 2019-08-05 15:44:02 +02:00
Dalf
fbf6b689dd [fix] dictzone engine 2019-08-05 15:43:21 +02:00
Dalf
9ff5001816 [fix] arxiv engine 2019-08-05 15:43:01 +02:00
Alexandre Flament
91a59ff350
[fix] Docker: without -f parameter, settings.yml is not replaced by the new version (#1668)
Before this commit, the existing settings.yml were always replaced.
2019-08-03 13:53:14 +02:00
Alexandre Flament
333e54943d
[fix] fix monkey patch in test_webapp.py (#1667)
at the end of test_webapp.py, the monkey patch of searx.search.Search was not revert which lead to side effects on other tests
close #1663
2019-08-03 13:23:36 +02:00
Alexandre Flament
d24e7948eb
Merge pull request #1665 from dalf/fix-paging
[fix] fix paging for the oscar theme after PR #1640
Close #1664
2019-08-02 20:38:53 +02:00
Dalf
ba80a0ba38 [fix] fix paging for the oscar theme after PR #1640 2019-08-02 20:04:37 +02:00
Alexandre Flament
72029d27de
[enh] Add timeout limit per request (#1640)
The new url parameter "timeout_limit" set timeout limit defined in second.
Example "timeout_limit=1.5" means the timeout limit is 1.5 seconds.

In addition, the query can start with <[number] to set the timeout limit.

For number between 0 and 99, the unit is the second :
Example: "<30 searx" means the timeout limit is 3 seconds

For number above 100, the unit is the millisecond:
Example: "<850 searx" means the timeout is 850 milliseconds.

In addition, there is a new optional setting: outgoing.max_request_timeout.
If not set, the user timeout can't go above searx configuration (as before: the max timeout of selected engine for a query).

If the value is set, the user can set a timeout between 0 and max_request_timeout using
<[number] or timeout_limit query parameter.

Related to #1077
Updated version of PR #1413 from @isj-privacore
2019-08-02 13:50:51 +02:00
Alexandre Flament
2179079a91
[fix] fix flickr_noapi decoding (#1655)
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
2019-08-02 13:37:13 +02:00
cy8aer
4dc792e1e2 [enh] add invidious engine. (#1657)
closes #1372
2019-08-02 13:25:25 +02:00
Alexandre Flament
02bbbf59d6
[fix] Dockerfile: searx version stores in org.label-schema.version (#1659)
instead of org.label-schema.schema-version
2019-08-01 07:51:09 +02:00
Alexandre Flament
1bed39e6cb
Merge pull request #1658 from dalf/video-fixes
Fix dailymotion, google_videos and youtube_noapi engines
2019-08-01 07:44:30 +02:00
Dalf
0c032c8429 [fix] youtube_noapi engine: fix the title 2019-07-31 08:39:40 +02:00
Dalf
8b7ac56669 [fix] google_videos engine: some results don't a thumbnail 2019-07-31 08:38:45 +02:00
Dalf
d44677e226 [fix] dailymotion engine: remove HTML tags from the description 2019-07-31 08:37:51 +02:00
Alexandre Flament
7f56c78876
Merge pull request #1650 from dalf/wikidata
[fix] wikidata engine: faster processing, remove one HTTP redirection.
2019-07-29 08:28:50 +02:00
Dalf
6e0285b2db [fix] wikidata engine: faster processing, remove one HTTP redirection.
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection)
* url_detail: remove the disabletidy=1 deprecated parameter
* Add eval_xpath function: compile once for all xpath.
* Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}')
* Create an etree.HTMLParser() instead of using the global one (see #1575)
2019-07-29 07:39:39 +02:00
Frank de Lange
cbc5e13275 [enh] flickr_noapi: use complete JSON data block, add 'content', 'img_format', 'source', etc. (#1571)
Fetch complete JSON data block, use legend to extract images. 
Unquote urlencoded strings.
Add image description as 'content'. 
Add 'img_format' and 'source' data (needs PR #1567 to enable this data to be displayed). 
Show images which lack ownerid instead of discarding them.
2019-07-28 10:42:00 +02:00
Alexandre Flament
c6ac39bcea
Merge pull request #1654 from MarcAbonce/small_fixes
[fix] Small fixes in Preferences view's text
2019-07-28 10:31:57 +02:00
Marc Abonce Seguin
59b1370824 [fix] small fixes in preferences view's text 2019-07-27 22:53:46 -07:00
Frank de Lange
204a2cbbf0 [fix] bing_videos (#1579)
use JSON where possible, compose 'content' using all available data, use correct 'url' (direct to source instead of redirect through bing)
2019-07-27 17:49:30 +02:00
Alexandre Flament
4b56a4bfaa
Merge pull request #1653 from dalf/bug-fixes
Various bug fixes
2019-07-27 09:11:03 +02:00
Marc Abonce Seguin
e1bb0e33f2 [fix] escape perecent signs in translation file (#1528) 2019-07-27 09:01:02 +02:00
Dalf
70a01e5f02 [fix] don't merge image results if the img_src is different 2019-07-27 08:52:30 +02:00
Dalf
23611897ec [fix] make sure then engine name is lower case
Minor fix: "%s engine initialized" display the right engine name
2019-07-27 08:52:30 +02:00
Dalf
8645ef717e [fix] settings.yml: make sure all engine names are lower case 2019-07-27 08:52:30 +02:00
Dalf
7d8000d965 [fix] "!wp !wp test" send only one request to wikipedia.
Note that "!general !wp test" is not fixed because the category for "!wp" is "none".
2019-07-27 08:52:30 +02:00
Dalf
2fab23ab9a [fix] fix the debug message "starting webserver on ip:port"
was "port:ip"
2019-07-27 08:52:30 +02:00
Frank de Lange
11fc9913e9 [enh] bing_images: use data from embedded JSON to improve results (e.g. real page title) (#1568)
use data from embedded JSON to improve results (e.g. real page title), add image format and source info (see PR #1567), improve paging logic (it now works)
2019-07-27 08:22:02 +02:00
Alexandre Flament
f34b5cedb1
[fix] fixes google play engines (#1651)
update commit 87baa74a86
2019-07-25 09:31:47 +02:00
Jonas Zohren
cac8e7c6b8 [enh] add npm search engine (#1510) 2019-07-25 08:54:26 +02:00
volth
eb182df132 [mod] restore btdigg engine as btdig.com (#1515) 2019-07-25 08:40:48 +02:00
rachmadani haryono
3b1122c5fa [fix] fix duden engine (#1594) 2019-07-25 08:17:45 +02:00
Venca24
87baa74a86 [fix] fixes google play engines and adds thumbnails to their results (#1612)
fix google play apps, google play apps, google play music engines

xpath engine: thumbnail_xpath can define an optional thumbnail
2019-07-25 07:46:41 +02:00
Alexandre Flament
d0dd296424
Merge pull request #1642 from dalf/start-speed
Speed up start time and decrease CPU load
2019-07-23 13:18:29 +02:00
Dalf
da0ce5880f [fix] fix soundcloud engine, speed up searx start time 2019-07-19 08:59:11 +02:00
Dalf
7e201cbf65 [mod] use cache in _match_language function to speed up searx start time significantly 2019-07-19 08:58:08 +02:00
Alexandre Flament
554a21e1d0
[enh] Add Server-Timing header (#1637)
Server Timing specification: https://www.w3.org/TR/server-timing/

In the browser Dev Tools, focus on the main request, there are the responses per engine in the Timing tab.
2019-07-17 10:38:45 +02:00
Alexandre Flament
cfcbc3a5c3
[mod] settings.yml: update comments about the morty key. (#1639)
Fix #1310
Since commit af77ec35d9bd28facdab645a3d57ae340d2b501c Morty accepts base64 encoded key.
2019-07-17 10:29:28 +02:00
rachmadani haryono
ec88fb8a0f [fix] secret_key can be bytes instead of a string (#1602)
Fix #1600
In settings.yml, the secret_key can be written as string or as base64 encoded data using !!binary notation.
2019-07-17 10:09:09 +02:00