It’s a bit of a weird shower thought but basically I was wondering hypothetical if it would be possible to take data from a social media site like Reddit and map the most commonly used words starting at 1 and use a separate application to translate it back and forth.
So if the word “because” was number 100 it would store the value with three characters instead of seven.
There could also be additions for suffixes so “gardening” could be 5000+1 or a word like “hoped” could be 2000-2 because the “e” is already present.
Would this result in any kind of space savings if you were using larger amounts of text like a book series?
That’s literally how compression works
I figured that’s roughly how JPEGs work but I wasn’t sure how exactly text compression would work. Do you have any recommendations for videos on data compression?
Jpeg compression is much more complex than what you described. I recommend this video as a primer: https://youtu.be/0me3guauqOU
This technique is called Huffman coding, and it’s used in a number of compression algorithms.
Huh that was an interesting read. I’m not sure I understood the entirety of it but it sounds like it would be a lot more efficient than what I was thinking
Yes, it’s very efficient and the core of what complession formats like .zip do.
The main difference to your idea is that computers count in binary like 0, 1, 10, 11, 100, 101 and so on and that you don’t want to assign these very low codes words directly. Say you’d have assigned 1 the most common word, then that will would be encoded very short, but you’d sort of take the one away from all the other codes, as you don’t know if 11 is twice the most common word or once the 11th (3rd in decimal) common word.
Huffman essentially computes the most optimal word assignments mathematically.
The main other difference between your suggestion and most compression algorithms is that you wouldn’t use a huge dictionary in real time and loading it and looking into it would be very slow. Most compression algorithms have a rather small dictionary buildin and/or they build one on the fly looking at the data that they want to compress.
its a good idea, you might want to look at huffman codes to start your reading
That’s pretty much what a tokenizer does for Large Language Models like Chat-GPT. You can see how it works here: https://platform.openai.com/tokenizer
Type in the word ‘Antidisestablishmentarianism’ and you can see it becomes 5 tokens instead of 28 characters.
The best compression algorithms are good at this. Look at the size this Reddit download and compare when you uncompress it.
There’s a website, I can’t remember the link, but any text you search can be found to have already been written on one of its pages.
It used an algorithm to generate every possible page of text in the English language, including gobbledygook and random keyboard smashing.
You can then share a link to a full page of text that is far smaller than the page itself, given infinite computational resources, it would be possible to parse any program or any bit of software into its text equivalent and then generate the URL that attaches to this algorithm for that entire page reducing a thousand characters to 16.
It would then be possible to make a page consisting of these 16 digit characters and then compress that page to another page, turning 70,000 words of text or so into 16 digits.
Once you had a page full of those, you could compress them again by finding a page that contains a link to the links of all of the pages.
Given that this is perfect compression you could literally reduce any file to the URL of the page that you need to use (and some information about how many pages deep you have to go and any needed information on the final file structure and format at the end of the reconstruction) to regenerate the page that you need to use to regenerate the page that you need to use to regenerate the page and so on until you have fully reconstructed the file.
And, given that these pages that can be summoned by a URL and do not actually exist until they are generated by the algorithm, it is entirely feasible that the algorithm itself could be stored on your local computer, meaning that with a somewhat complicated but still reasonable system, you could send a file of any size to any person in a text.
The problem is the computational resources needed to utilize this algorithm. It would have to crunch out the numbers to rebuild every single file that is sent its way and even on a fairly fast system that’s going to take some time.
However, if anyone wants to take this idea and run with it I’m pretty sure the world would appreciate your efforts. Since we have the ability to create a page of text using an algorithm then we probably should use that and then once it becomes commonplace I’m sure people will build chiplet accelerators for the process.
There’s a website, I can’t remember the link, but any text you search can be found to have already been written on one of its pages.
It’s The Library of Babel
You can type up to 3200 characters in lower case. With a short sentence though, the “title” and page number may be longer than your original text!
Edit: Fixed url title
This is kind of blowing my mind.
https://libraryofbabel.info/bookmark.cgi?361:9
Thank you for the link
It’s the Library of Babel. Sorry, pedantic, I know, but the Tower of Babel is something different. If people go searching for that they’re going to find a lot of discussion around the bible story and no links to the text generator.
There is simply no way to pre-store any arbitrary long text because the possible combination is just tooo huge. It is like draw a bullseye where you arrow landed. You search any paragraph, it search existing index, if not found then save it and assign it a new index. Of course it may use some algorithms to optimize/decrease the space needed.
From it’s wiki page, https://en.m.wikipedia.org/wiki/The_Library_of_Babel_(website) ‘’’ The website can generate all possible pages of 3200 characters and allows users to choose among about 10^4677 potential pages of books. ‘’’
PS, I think again, indeed this actually does not need any storage, it can be URL to text encoding/decoding, the URL itself can determine/generate the actual text. And same for the opposite direction
it would be possible to parse any program or any bit of software into its text equivalent and then generate the URL that attaches to this algorithm for that entire page reducing a thousand characters to 16.
This can’t work. Let’s use a simpler example, instead of 16 characters for the link let’s say it’s a single digit and let’s say the content of the “page” is 4 digits. One digit has 10 possible values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. 4 digits have 10,000 possible combinations. With only one digit to index into the 10,000 possible combinations, you can point to only 10 of them.
It’s the same thing for pages of text. If you have a 16 character link and the content you’re trying to index with it is more than 16 characters then you can only point to some of the possibilities in the larger set.
