One of the most unforgivable things about reddit is how pathetic the search engine is, considering the amount of free, top notch information is captured there and you need google +reddit to get at it, what can we do to make federated alternatives self searchable ?

  • headie_sage@fanaticus.socialEnglish
    1·
    1 year ago

    I’ve posted about this before as it relates to mod tools.1

    The search part isn’t all that difficult, there are open source search engines that are easy enough for admins to configure a decent search feature. The more difficult issue is aggregating the data from all our instances to a single source where we can make queries with those existing search engine tools.

    I am going to spend some time this weekend working on a proof of concept for a search engine for mod tools. Big picture solution is:

    1. Instance admins regularly dump anonymized (i.e. no PII) post and comment data to a public source (possibly torrent, possibly sftp)
    2. Other instance admins download each others data and feed it into their search db (e.g. Elasticsearch)
    3. Mods & users create tools using this data

    BTW: this isn’t a novel idea:

    • This is what pushshift is for reddit (check out their FAQ/wiki). We’re missing mod tools big time and searching/aggregating is huge part of mod tools.
    • Up until recently, like last week, Stack Exchange provided a regular dump of their data to the Internet Archive for posterity’s sake

    EDIT: Linked my OG post on the subject [1](https://fanaticus.social/post/1955)