Commit graph

83 commits

Author SHA1 Message Date
Alexandre Flament
5a9551925d [fix] fix bugs introduced by commit 0a2fde19d0 from dalf/searchpy4 2016-12-30 18:10:14 +01:00
Alexandre Flament
28d51fd063 [mod] timeout in log are readable (the timeouts are compare to the start_time of the request). 2016-12-30 17:37:46 +01:00
Alexandre Flament
3a8ab5880a [mod] result_container.extend sets result['engine'] = engine_name for each result 2016-12-30 17:18:02 +01:00
Alexandre Flament
0a2fde19d0 [mod] add a search_one_request_safe function wrapper to call search_one_request. All exceptions are catched in this wrapper. 2016-12-30 11:51:49 +01:00
marc
7fba3d1179 Merge branch 'master' into languages 2016-12-16 22:14:36 -06:00
Adam Tauber
cf5c7d050a [fix] engine setting if no categories specified 2016-12-15 11:59:21 +01:00
marc
1b46ecd21a [fix] broken language selector 2016-12-13 21:36:40 -06:00
Adam Tauber
75aa46a6b3 [fix] remove unused variable 2016-12-10 00:15:53 +01:00
Adam Tauber
0205838f25 [enh] handle engine response crashes 2016-12-10 00:15:33 +01:00
Alexandre Flament
e48f07a367 Merge branch 'master' into searchpy2 2016-12-09 23:11:45 +01:00
Adam Tauber
971ed0abd1 [enh] add quick answer functionality with an example answerer 2016-11-19 20:53:51 +01:00
Alexandre Flament
219f047bf3 Merge branch 'master' into searchpy2 2016-11-06 09:44:07 +01:00
Adam Tauber
7d26b3f8f8 [fix] suspend end time check 2016-11-06 02:53:09 +01:00
Alexandre Flament
01e2648e93 Simplify search.py, basically updated PR #518
The timeouts in settings.yml is about the total time (not only the HTTP request but also the prepare the request and parsing the response)
It was more or less the case before since the threaded_requests function ignores the thread after the timeout even the HTTP request is ended.

New / changed stats :
* page_load_time : record the HTTP request time
* page_load_count: the number of HTTP request
* engine_time : the execution total time of an engine
* engine_time_count : the number of "engine_time" measure

The avg response times in the preferences are the engine response time (engine_load_time / engine_load_count)

To sum up :
* Search.search() filters the engines that can't process the request
* Search.search() call search_multiple_requests function
* search_multiple_requests creates one thread per engine, each thread runs the search_one_request function
* search_one_request calls the request function, make the HTTP request, calls the response function, extends the result_container
* search_multiple_requests waits for the the thread to finish (or timeout)
2016-11-05 13:45:20 +01:00
Alexandre Flament
58a6c045c8 [mod] replace references request.request_data (GET or POST parameters) by request.form (based on merge of POST and GET parameters) 2016-11-02 14:52:22 +01:00
dalf
fbb080f358 Change plugin API :
- pre_search(request, search)
- post_search(request, search)
- on_result(request, search, result)

with
- request is the Flask request
- search a searx.Search instance
- result a searx result as usual
2016-11-02 14:22:16 +01:00
dalf
67e11c42b9 Clean up the architecture
Purposes :
- isolate the plugins calls
- distinction between parsing the web request and running the search (Search class). To be able to test code easily, to run searx code outside a web server, to filter the search query parameters with plugins more easily, etc...

Details :
- request.request_data contains request.form or request.args (initialize inside pre_request() function)
- Query class is renamed RawTextQuery
- SearchQuery class defines all search parameters
- get_search_query_from_webapp create a SearchQuery instance (basically the previous Search.__init__ code)
- Search class and SearchWithPlugins class takes a SearchQuery instance as class constructor parameter
- SearchWithPlugins class inherites from Search class, and run plugins
- A dedicated function search_with_plugins executes plugins to have a well define locals() (which is used by the plugins code).
- All plugins code is executed inside the try...except block (webapp.py, index function)
- advanced_search HTTP parameter value stays in webapp.py (it is only part of UI)
- multiple calls to result_container.get_ordered_results() doesn't compute the order multiple time (note : this method was call only once before)
- paging value is stored in the result_container class (compute in the extend method)
- test about engine.suspend_end_time is done during search method call (instead of __init__)
- check that the format parameter value is one of these : html, rss, json, rss (before the html value was assumed but some text formatting wasn't not done)
2016-11-02 14:22:16 +01:00
Adam Tauber
55abf07a4f [fix] thread safe threaded_requests() function 2016-09-06 00:36:33 +02:00
Adam Tauber
ffbee4bb82 [fix] collect garbage after searches to reduce memory footprint 2016-07-31 23:39:58 +02:00
Noemi Vanyi
16a8641649 advanced-search: panel is visible when checked before 2016-07-25 23:30:48 +02:00
Noemi Vanyi
93c0c49e9a add time range search with yahoo 2016-07-25 23:19:46 +02:00
Noemi Vanyi
5867e97966 more follow ups in new preferences 2016-04-09 19:49:59 +02:00
Noemi Vanyi
8c4db08443 rename blocked engines to diabled engines 2016-04-09 19:49:59 +02:00
Noemi Vanyi
fe691a0988 new preferences handling
Preferences class was introduced in order to handle user preferences. Right now
it parses cookies and the form in preferences. Also it can retrieve settings
based on the name of the setting.

ATTENTION
Please note that engine preferences are handled differently from now on. So it
introduces incompatible changes. Every user who has saved preferences should reset and
save his/her settings again.

This change was needed, because everytime a default disabled engine was
added saved user preferences would broke. Now engine setting tracking is
fixed.
2016-04-09 01:08:44 +02:00
Adam Tauber
10947536aa [enh] suspend engines after error
The duration is based on the number of continuous errors, but maximized in one minute
2016-02-20 00:21:56 +01:00
Adam Tauber
786fe49fab Merge pull request #438 from Cqoicebordel/error_page_format
Don't throw away the request if pageno is wrong
2015-12-01 15:50:47 +01:00
Adam Tauber
b6c3cb0bdd [enh][mod] result handling refactor
Several changes has been made:
 - Parallel result merge
 - Scoring algorithm slightly changed (see result_score())
 - Proper Thread locking on global data manipulation
2015-10-03 17:26:07 +02:00
Cqoicebordel
ed1daa32ea Don't throw away the request if pageno is wrong
Fix #424 but maybe only partially
2015-09-18 20:44:04 +02:00
Adam Tauber
37c3ace309 [fix] add missing scheme to duplicated results too ++ revert gigablasts handling 2015-09-11 18:33:06 +02:00
Adam Tauber
f094188780 [fix] display categories of the selected engines 2015-09-01 15:21:17 +02:00
Adam Tauber
8c4d9b79d5 [fix] engine selection from url 2015-09-01 15:14:02 +02:00
Adam Tauber
42ac2c5b56 [fix] check empty engine language attribute - fixes subtitleseeker 2015-08-25 22:02:18 +02:00
Thomas Pointhuber
5bffa9ca33 [fix] rewrite scheme to http if there is no one, FIX #390 2015-08-24 11:18:58 +02:00
Adam Tauber
7c9f931baf [fix] type error 2015-08-02 20:41:44 +02:00
Adam Tauber
43cd8e0c41 [enh] default settings option to safe_search - #396 2015-08-02 20:32:32 +02:00
Cqoicebordel
00668dd0a1 New feature : language per search engine ! 2015-06-03 17:16:12 +02:00
Adam Tauber
b09aaf126e [mod] allow whitespaces in category/engine list 2015-05-19 11:16:57 +02:00
Adam Tauber
b6de23f540 [enh] cleaner category/engine selection from GET/POST param - closes #289 2015-05-12 08:21:39 +02:00
Adam Tauber
d1b12f9016 [mod] search return value 2015-03-08 23:30:31 +01:00
Adam Tauber
bf09daa31d [enh] log redirects 2015-02-13 13:02:37 +01:00
dalf
5029cb4787 [fix] close #197 for all engines 2015-02-13 11:17:00 +01:00
Thomas Pointhuber
e7de9674b1 [enh] add safesearch functionality
2 = strict
1 = moderate
0 = none
2015-02-08 21:53:37 +01:00
Adam Tauber
1de781a143 [enh] category specific engine deactivation - closes #205 2015-02-03 18:37:38 +01:00
Adam Tauber
b19e681cce [enh] default disabled engines - closes #109 2015-01-31 23:11:45 +01:00
dalf
d07cfd9089 [enh] use one single http connection pool : improve response time. close #100 2015-01-21 11:33:16 +01:00
Thomas Pointhuber
5bfaabeb6b [enh] add nojs support into oscar-template 2015-01-15 15:48:50 +01:00
Adam Tauber
b47b8cc2c6 Merge pull request #166 from Cqoicebordel/non-exclusif-bang
Add non exclusif bang
2015-01-12 14:14:40 +01:00
Adam Tauber
23b4a53248 [enh] engine exception traceback 2015-01-09 04:30:55 +01:00
Adam Tauber
299a80a1eb [enh] using the logger 2015-01-09 04:13:05 +01:00
Cqoicebordel
96c4d52eef Add non exclusif bang
Allow to perform a search while adding an engine (or a category) without adding it "officially" to the request.
'?' is used to add an engine without modifying anything else to the request. For example, you can perform a search in the 'general' category, and if you add '?tw' the result from Twitter will be added to the originals results.
2015-01-03 02:31:23 +01:00