Brave Search has a free API, however it's limited by 2000 requests per month. That's something I hit daily, twice. Scraping might be an option here, as it seems to work without JS.

Ecosia does not seem to have an API, scraping will be required if we want to use Ecosia. Running curl on Ecosia results in: Enable JavaScript and cookies to continue, so that might be an issue.

Mojeek.com seems to be really easy to scrape, they have a bunch of <li>'s, sadly no specific classnames etc. They do have an API but it's paid-only.

Google also has a paid API: "Custom Search JSON API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day.

If you need more than 10k queries per day and your Programmable Search Engine searches 10 sites or fewer, you may be interested in the Custom Search Site Restricted JSON API, which does not have a daily query limit."

Bing also allows 1000 requests for free.

1,000 transactions free per month for all markets

I think avoiding APIs and using scraping where possible is our best bet. 1000 requests per month for free is very limited and I'm sure most instances will reach it within a day or two. Also scraping would avoid the instance maintainer having to register to the service, which otherwise may be a privacy issue for them. I don't think that instance maintainers would be willing to pay for search results either.

There might be a way to trick ecosia into sending data that it would get via javascript by copying the requests it makes when it does a search, however if this is too complicated it might be worth leaving it out for now and searching for other easily-scrapable services.

I think avoiding APIs and using scraping where possible is our best bet. 1000 requests per month for free is very limited and I'm sure most instances will reach it within a day or two. Also scraping would avoid the instance maintainer having to register to the service, which otherwise may be a privacy issue for them. I don't think that instance maintainers would be willing to pay for search results either.

I know and strongly agree, but issue #95 mentioned APIs, so I decided to do the extra research to show it's pretty impossible to use them whilst keeping LibreY free as in freedom and as in price.

Ecosia does not seem to have an API, scraping will be required if we want to use Ecosia. Running curl on Ecosia results in: Enable JavaScript and cookies to continue, so that might be an issue.

This is a Cloudflare issue. Without Tor on an server IP, spoofing the user agent seems to be sufficient for bypassing JavaScript blocks:

curl -V:

$ curl -V
curl 8.5.0 (x86_64-unknown-openbsd7.4) libcurl/8.5.0 LibreSSL/3.8.2 zlib/1.3 nghttp2/1.57.0 ngtcp2/0.19.1 nghttp3/0.15.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe UnixSockets

Fail:

$ curl -I https://www.ecosia.org/
HTTP/2 403
...

Success (results in legible response):

$ curl -I -A "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0" https://www.ecosia.org/
HTTP/2 200
...

Made it so that the engines are automatically selected when engine is "auto", based on cooldowns.

Squashed a different bug in the last commit: the caching feature was not working properly when doing fallback, since the the url for the fallback instance was being used. Haven't tested with fallback but should work when switching engines.

Implemented setting for selecting which engine to use. Added this so that users still have the preference of which engine they want to use. While this may cause these users to experience more ratelimits by limiting their search results to a particular engine, it ensures that they still have the freedom to use whichever results source they want, for example if they don't get the particular results they are looking for with other engines.

This prefered engine parameter is passed to fallback instances with param engine, meaning that fallback instances that implement this commit should be able to return results from the engine that the user has selected.

I wasn't able to figure out how to pass a param similar to npfr to duckduckgo, brave and yandex, which on google enables the "did you mean?" feature (rather than "showing results for"). Instead "spellcheck" and "noreask" were used to disable spell correction of results, to avoid confusing queries where LibreY is unable to show the corrected text while still showing results for it (ie gentou linux would show results for gentoo linux without showing the "showing results for, did you mean?")

If someone is able to figure out a way to get a feature like google's npfr for the other engines, that is to still show the misspelled query while offering a suggestion, then that would be great.

Implemented Mojeek and Ecosia search scrapers and they seem to be working well. I haven't been able to figure out how to set the language for mojeek results, nor setting them to not autocorrect searches.

Other than these two small features, I think that this PR is ready. By simply varying results from the 4 previous engines, that was enough to prevent my instance from being ratelimited by all of them at the same time.

Selecting a particular results source seems to work pretty well, allowing users to choose an engine if they aren't satisfied with results from another.

If you think these changes as they are, then feel free to merge, though maybe an issue for adding the missing features should be opened.

Looks very great! Might also be nice to add a feature along the lines of "Use all engines except xyz".

markzg from RiseUp emailed me again, @davidovski, see above.