3500 Books??? Really?

[wesleyer]

Originally Posted by vugtitan

i tried it with winrar and i managed to get a 400kb text file down to 100k.

but what is stop me from compressing the compressed file over and over again until is down to the 100 ratio to 1 or even using fractal compression?

in theory you could keep compressing it down until it is just down to just a few bytes.

remember how in the big bank the entire universe was formed from a single ultra dense super compressed single particle ?

Similarly we could do the same with information.

Maybe you need to read a little bit more about Information Theory.

Here goes a treatise on it: aK

You just need to uncompress it.

[vugtitan]

Originally Posted by desertblues

In the last few days I've sent, through Calibre, my whole booklibrary of several years(I like to browse and choose)on my Kindle.; 1128 items. Yes, I know it is probably a bit weird, but I like my books with me, I read a lot and I have trouble throwing anything away.
My PC had some trouble with it and it did cost a lot of time. That is; I didn't use Calibre before, so I had to start from scratch.
Yesterdayevening the battery was full and in the morning it was empty; I guess because of indexing?
I don't think the amount of books has any influence on the Kindle3. I can find and open a book now just as fast as in the beginning, when I only had the ones of Amazon (20 or so).

Why did you PC have trouble with simple things like books?

Could you load another 3500-1128=2372 books to see what happens?

I still have not received my kindle after ordering last week and i am desperate to know if i can load all my books on to it.

[vugtitan]

Originally Posted by wesleyer

Maybe you need to read a little bit more about Information Theory.

Here goes a treatise on it: aK

You just need to uncompress it.

Just had a brilliant idea after reading about the compression algorithm on wiki.

the problem is that the dictionary in say winrar/winzip records the location frequency and recurrence of each letter in a table of the compressed file and uses this table to uncompress it.

The dictionary cannot be compressed anymore as it stands.


for example:-

"Ask not what your country can do for you -- ask what you can do for your country."

The quote has 17 words, made up of 61 letters, 16 spaces, one dash and one period. If each letter, space or punctuation mark takes up one unit of memory, we get a total file size of 79 units. To get the file size down, we need to look for redundancies.

Immediately, we notice that:

* "ask" appears two times
* "what" appears two times
* "your" appears two times
* "country" appears two times
* "can" appears two times
* "do" appears two times
* "for" appears two times
* "you" appears two times



Now my idea is why not have a separate dictionary for the primary dictionary which will specify the primary dictionary's dataset.because the primary dictionary is 10 times smaller then the secondary dictionary which describes how to uncompress the primary dictionary and the primary dictionary will expand the compressed file.Thus the secondary dictionary simply ONLY relates to the primary dictionary and how to create it.

thus you would have 2 separate files on compressing rar1 and rar2 and rar2 only relates to rar1 dictionary's creation and then rar1 will recreate original file.

by this method you could achieve nearly unlimited compression for not just text but video too.just imagine a dvd video file downloading in seconds.

could you people give me a critical analysis of any mistakes in the above above idea?

[Tiersten]

Originally Posted by vugtitan

by this method you could achieve nearly unlimited compression for not just text but video too.just imagine a dvd video file downloading in seconds.

could you people give me a critical analysis of any mistakes in the above above idea?

No. It doesn't work like that. You need to read up more on information theory and compression algorithms. Following your logic, you could keep compressing a compressed file until it becomes 1 bit. Now try and decompress that.

The description of compression algorithms in Wikipedia is also significantly simplified. Only very naive or simple algorithms will do what you said.

I'm not going into the math of information theory or compression algorithms but you get to a point where the file doesn't have any redundancy at all or very little. At this point, the overheads from compression are greater than the gains.

[wesleyer]

Originally Posted by vugtitan

Now my idea is why not have a separate dictionary for the primary dictionary which will specify the primary dictionary's dataset.because the primary dictionary is 10 times smaller then the secondary dictionary which describes how to uncompress the primary dictionary and the primary dictionary will expand the compressed file.Thus the secondary dictionary simply ONLY relates to the primary dictionary and how to create it.

thus you would have 2 separate files on compressing rar1 and rar2 and rar2 only relates to rar1 dictionary's creation and then rar1 will recreate original file.

by this method you could achieve nearly unlimited compression for not just text but video too.just imagine a dvd video file downloading in seconds.

could you people give me a critical analysis of any mistakes in the above above idea?

You realize you would still have to keep all those files, right ?
It does not matter whether you just compress the dictionary for the second pass, in order to retrieve the original file the first compressed file would still be required.

As Tiersten pointed out, if your method worked you could compress anything to one bit. So, I'll give you one bit: 1.
Now tell me: which book this one bit decompresses to ? Is it the Bible ? Is it the newest Stieg Larsson novel ? Is it a treatise on Information Theory ?
Don't you realize you can't get all that information from a single bit ?

Take a look at this: http://www.faqs.org/faqs/compression...section-8.html

[SpeckledJim]

Originally Posted by vugtitan

could you people give me a critical analysis of any mistakes in the above above idea?

I'd say your first mistake is to claim you have a brilliant idea for unlimited compression when you fail to grasp the basics of the problem.

Many people have spent many years coming up with the compression systems we have today. It's just a tad naive to imagine you've stumbled upon the data equivalent of perpetual motion after simply reading a Wikipedia article.

[kacir]

Originally Posted by vugtitan

could you people give me a critical analysis of any mistakes in the above above idea?

This could be a good starting point for study of a really good compression algorithm.
http://en.wikipedia.org/wiki/Bzip

Bzip is open source, so you can study it and try to improve.
The important thing is not to invent things that had been invented before, but stand on the shoulders of giants.

[whitearrow]

I have 458 items on my K3, about 100 of which are samples or short stories (>100k). The rest are "normal" with a few that are very large (cookbook, complete Jane Austen). Except for a few stuck Topaz files, everything indexed very quickly and it works well overall.

[desertblues]

I transfered about 1200 books, normal size, in batches of 300. It went well. Indexing took one batterycycle; I don't know how long because it did it at night.
I had no problem using the Kindle afterwards. But right after my last batch of books, I started to sort them in collections and that was too much. It didn't show the option add/remove items anymore, while I was in a collection and wanted to add several items. I had to slow down; give it a break for an hour or so( and plugged it in to load, although the battery indicated to be full) and then I finished without further problems.
I'm a happy camper now, having done all the work and am reading at last.

[bjones6416]

Originally Posted by desertblues

I transfered about 1200 books, normal size, in batches of 300. It went well. Indexing took one batterycycle; I don't know how long because it did it at night.
I had no problem using the Kindle afterwards. But right after my last batch of books, I started to sort them in collections and that was too much. It didn't show the option add/remove items anymore, while I was in a collection and wanted to add several items. I had to slow down; give it a break for an hour or so( and plugged it in to load, although the battery indicated to be full) and then I finished without further problems.
I'm a happy camper now, having done all the work and am reading at last.

Thanks for the updates on how it is going for you with the K3! I know a lot of people are interested in this right now.

[Nathanael]

Originally Posted by Oldpilot

Okay, $9.99 times 3500 ... $3,496.50!

Egads! Methinks thou needst a new calculator.

--Nathanael

[desertblues]

BJones6416: I think, after loading several books; one has to wait till the indexing is finished. Somewhere on my Kindle, after loading a lot of books, I saw the announcement ' indexing 35 books' ( or something similar). But I couldn't find the button to let it re-appear.
Indexing seems to take a lot of energy and workspace out of the Kindle; any further action while indexing is bound to confuse the Kindle.
Phhh..; I have the Kindle for one week and already am speaking about it like it is a person or pet, like my cat Polly.

[desertblues]

Originally Posted by vugtitan

Why did you PC have trouble with simple things like books?

Could you load another 3500-1128=2372 books to see what happens?

I still have not received my kindle after ordering last week and i am desperate to know if i can load all my books on to it.

Dear Vugtitan. Nice try; but I loaded all the books with my fingers crossed, as I wrecked another ereader(iLiad) a few months ago with my inexperienced experiments. For which I still am sad/mad, also because iRex went broke and it can't be fixed.
I'm now through with loading my books (1534). It works like this: 200 or 300 per batch through Calibre---let the K3 rest a night for indexing and reloading.
When sorting the books in collections: I've experienced that the K3 doesn't crash, but that the option ' add/remove titels' disappears-----then let it rest for an hour while reloading, and then it's fine.
Calibre crashed twice while processing my books; that is, while it searched for metadata in bulk. So I guess it isn't my PC that is the problem.
But, apart from the browser, that keeps freezing on me when I use it outside of my home, I'm satisfied with the K3. Although I'll mail Amazon whether I need to install the new update.

[HarryT]

Originally Posted by desertblues

BJones6416: I think, after loading several books; one has to wait till the indexing is finished. Somewhere on my Kindle, after loading a lot of books, I saw the announcement ' indexing 35 books' ( or something similar). But I couldn't find the button to let it re-appear.

Do a search. The number of books remaining to be indexed is displayed at the end of the search results.

[desertblues]

Thnx, Harry