I’m no expert, but dreaming here - is there a FOSS search engine that can be run in a distributed way by a community? I would happily switch to that if there was.
It’s a really interesting question and I imagine scaling a distributed solution like that with commodity hardware and relatively high latency network connections would be problematic in several ways.
There are several orders of magnitude between the population of people who would participate in providing the service and those who would consume the service.
Those populations aren’t local to each other. In other words, your search is likely global across such a network, especially given the size of the indexed data.
To put some rough numbers together for perspective, for search nearing Google’s scale:
A single copy of a 100PB index would require 10,000 network participants each contributing 10TB of reliable and fast storage.
100K searches / sec if evenly distributed and resolvable by a single node would be at least 10 req/sec/node. Realistically it’s much higher than that, depending on how many copies of the index, how requests are routed, and how many nodes participate in a single query. Of that 10TB of storage per node, substantial amounts of it would need to be kept in memory to sustain the likely hundreds of req/sec a node might see on average.
The index needs to be updated. Let’s suppose the index is 1/10th the size of the crawled data and the oldest data is 30 days (which is pretty stale for popular sites). That’s at least 33PB of data to crawl per day or roughly 3,000Gbps minimum sustained data ingestion. For those 10,000 nodes they would need 1Gbps of bandwidth to index fresh data.
These are all rough numbers but this is not something the vast majority of people would have the hardware and connection to support.
You’d also need many copies of this setup around the world for redundancy and lower latency. You’d also want to protect the network against DDoS, abuse and malicious network participants. You’ll need some form of organizational oversight to support removal of certain data.
Probably the best way to support such a distributed system in an open manner would be to have universities and other public organizations run the hardware and support the network (at a non-trivial expense).
So this is starting to sound more like something that needs to explicitly be paid for in some way (as opposed to just crowd sourcing personal hardware), at least if we want to maintain the same level of service.
I’m no expert, but dreaming here - is there a FOSS search engine that can be run in a distributed way by a community? I would happily switch to that if there was.
It’s a really interesting question and I imagine scaling a distributed solution like that with commodity hardware and relatively high latency network connections would be problematic in several ways.
There are several orders of magnitude between the population of people who would participate in providing the service and those who would consume the service.
Those populations aren’t local to each other. In other words, your search is likely global across such a network, especially given the size of the indexed data.
To put some rough numbers together for perspective, for search nearing Google’s scale:
A single copy of a 100PB index would require 10,000 network participants each contributing 10TB of reliable and fast storage.
100K searches / sec if evenly distributed and resolvable by a single node would be at least 10 req/sec/node. Realistically it’s much higher than that, depending on how many copies of the index, how requests are routed, and how many nodes participate in a single query. Of that 10TB of storage per node, substantial amounts of it would need to be kept in memory to sustain the likely hundreds of req/sec a node might see on average.
The index needs to be updated. Let’s suppose the index is 1/10th the size of the crawled data and the oldest data is 30 days (which is pretty stale for popular sites). That’s at least 33PB of data to crawl per day or roughly 3,000Gbps minimum sustained data ingestion. For those 10,000 nodes they would need 1Gbps of bandwidth to index fresh data.
These are all rough numbers but this is not something the vast majority of people would have the hardware and connection to support.
You’d also need many copies of this setup around the world for redundancy and lower latency. You’d also want to protect the network against DDoS, abuse and malicious network participants. You’ll need some form of organizational oversight to support removal of certain data.
Probably the best way to support such a distributed system in an open manner would be to have universities and other public organizations run the hardware and support the network (at a non-trivial expense).
So this is starting to sound more like something that needs to explicitly be paid for in some way (as opposed to just crowd sourcing personal hardware), at least if we want to maintain the same level of service.
It seems like there are others in the thread with good options
Do you mean like Yacy?
YES
SearXNG?
Good one!
FOSS won’t change this matter unless it somehow can filter out all the low quality AI-generated articles better than Google’s filters.
Don’t allow sites in the index by default, use an allowlist.
Is this allowlist supposed to work on article by article basis? Because that’s what has to be done for publishing platforms like medium.
By default I was thinking site. But for sites with huge variances in page quality you’d need it by page/article as you say.