Compare commits

...

3 commits

Author SHA1 Message Date
Grant Lanham Jr ab064f01c0
Merge 23bf9f9d95 into 41f415aabf 2024-04-26 09:34:01 +02:00
Grant Lanham 23bf9f9d95 add base http params 2024-04-25 22:58:19 -04:00
Grant Lanham 2b48584bb4 Fix bing page numbering, add sc to parameter, minor refactor
Bing page numbering doesn't increase by 10 each time. The first page returns 10 results, and all pages thereafter return 14 results. This means we need to update the page numbering

Next, the 'sc' parameter, whatever it means, needs to be present in order to not return the same results.

Finally, the code to check the page had some duplicate checks, so I refactored the code in this section which is low-risk.
2024-04-23 21:03:51 -04:00

View file

@ -95,18 +95,25 @@ def request(query, params):
# don't ask why it is only sometimes / its M$ and they have never been
# deterministic ;)
'pq': query,
# TODO: Figure out how below parameters are populated
'sc': '0-0',
"sp": "-1",
"lq": "0",
"qs": "n",
"ghsh": "0",
"ghacc": "0",
"ghpl": "",
}
# To get correct page, arg first and this arg FORM is needed, the value PERE
# is on page 2, on page 3 its PERE1 and on page 4 its PERE2 .. and so forth.
# The 'first' arg should never send on page 1.
if page > 1:
query_params['first'] = _page_offset(page) # see also arg FORM
if page == 2:
query_params['FORM'] = 'PERE'
elif page > 2:
query_params['FORM'] = 'PERE%s' % (page - 2)
if page == 2:
query_params['FORM'] = 'PERE'
else: # page > 2:
query_params['FORM'] = 'PERE%s' % (page - 2)
params['url'] = f'{base_url}?{urlencode(query_params)}'