Forma Large Database Question

[93terp]

I have what seems to be an incredibly large Kobo database (800+MB for approximately 6,500 kepubs). All of my books are sideloaded. If I log out of my Kobo account and log back in, will I have to resend everything back to my Forma? Or will the login/logout just rebuild the database while maintaining my books on the device?

[DevonHess]

Nope. It'll delete your progress though. Make sure you have a backup of everything.

[PeterT]

Actually it will only delete the non sideloaded content and leave the side loaded intact. But.... why not try using Kobo Utilities in calibre to do a compress of the database.

[davidfor]

Originally Posted by 93terp

I have what seems to be an incredibly large Kobo database (800+MB for approximately 6,500 kepubs). All of my books are sideloaded. If I log out of my Kobo account and log back in, will I have to resend everything back to my Forma? Or will the login/logout just rebuild the database while maintaining my books on the device?

That seems a bit big. I have about 8000 books on my Clara HD and the database is a little 200MB. Though that is 80% epub, which has less records in the database. But, I wouldn't expect it to be 4 times as much space. But, it will also depend on the length of the comments/synopsis for the book and if you are making annotations.

You could try compressing the database. That should reduce the size, but, the amount will depend on how fragmented the database is. If you are a calibre user, my Kobo Utilities plugin has this as a function. Or you can use any SQLite database manager to do it. And I would recommend taking a backup of the database before doing this.

If you logout of the Kobo account, the database is removed and a new one created when you login again. Then the sideloaded books will be imported again. You will lose any annotations, reading status and collections for these books. Again, if you are a calibre user, some of these can be restored afterwards.

Could I ask that when you do either of the above, that you post the new database size. I'm curious about how much these steps actually help.

[93terp]

Originally Posted by PeterT

Actually it will only delete the non sideloaded content and leave the side loaded intact. But.... why not try using Kobo Utilities in calibre to do a compress of the database.

I've done that but it usually doesn't result in much of a compression.

[93terp]

Originally Posted by davidfor

That seems a bit big. I have about 8000 books on my Clara HD and the database is a little 200MB. Though that is 80% epub, which has less records in the database. But, I wouldn't expect it to be 4 times as much space. But, it will also depend on the length of the comments/synopsis for the book and if you are making annotations.

You could try compressing the database. That should reduce the size, but, the amount will depend on how fragmented the database is. If you are a calibre user, my Kobo Utilities plugin has this as a function. Or you can use any SQLite database manager to do it. And I would recommend taking a backup of the database before doing this.

If you logout of the Kobo account, the database is removed and a new one created when you login again. Then the sideloaded books will be imported again. You will lose any annotations, reading status and collections for these books. Again, if you are a calibre user, some of these can be restored afterwards.

Could I ask that when you do either of the above, that you post the new database size. I'm curious about how much these steps actually help.

I don't have comments, I don't have annotations, I don't have notes or highlights, just the books. And 100% of them are kepubs (conversion to kepub on the sideload from Calibre).

I've used your database compression tool many times. But at this point it's not resulting in much of a noticeable reduction in the size of the database.

Will absolutely post the database before & after sizes later today. Plan to do the login/logout on my Forma shortly.

[93terp]

Originally Posted by davidfor

That seems a bit big. I have about 8000 books on my Clara HD and the database is a little 200MB. Though that is 80% epub, which has less records in the database. But, I wouldn't expect it to be 4 times as much space. But, it will also depend on the length of the comments/synopsis for the book and if you are making annotations.

I also have the Clara HD, as well as the Libra, and can only fit about 50-75% of my 6,500+ book collection. All my books are epub format, that I convert to kepub during the calibre sideload to the device. Is there something "special" you are doing to have such small file sizes such that you can fit that many books on your Clara HD?

[Semwize]

Originally Posted by 93terp

I also have the Clara HD, as well as the Libra, and can only fit about 50-75% of my 6,500+ book collection. All my books are epub format, that I convert to kepub during the calibre sideload to the device. Is there something "special" you are doing to have such small file sizes such that you can fit that many books on your Clara HD?

Change in Kobo Clara the micro SD card to 128 gigabyte and it will hold 100+ thousand books.

Why do you need so much? If you read one book a day, then 6500 of your books you will read in just 18 years ... I am very pleased with your love of reading, but your Clara will not live that long, alas.

[93terp]

Test on my Aura One LE (32GB) (testing on this one first before doing it on my Forma, which I use as my primary reader) -

# of books on the device = 6,748 (all kepubs)

Compression utilizing Kobo Utilities plugin
Database size pre-compression = 808,964MB
Database size post-compression = 808,131MB

Unfortunately the login/logout didn't go so well, and as a result I had to do a factory reset. After I logged out/logged back in, the "Importing content" stopped at 18% and only 918 books were there. C'est la vie. Reloading now.

[DNSB]

One thing to remember is the the database compression is not compression in the sense that, for example, creating a zip archive compresses files.

To quote from the SQLite documentation:

VACUUM command cleans the main database by copying its contents to a temporary database file and reloading the original database file from the copy. This eliminates free pages, aligns table data to be contiguous, and otherwise cleans up the database file structure.

[raygj]

Originally Posted by Semwize

Change in Kobo Clara the micro SD card to 128 gigabyte and it will hold 100+ thousand books.

Why do you need so much? If you read one book a day, then 6500 of your books you will read in just 18 years ... I am very pleased with your love of reading, but your Clara will not live that long, alas.

I upgraded my ClaraHD to 128gig sd card.
But make sure you purchase a modern SD card such as a "Sandisk Extreme Pro microSDXC, SQXCY 128GB, V30, U3, C10, A2, UHS-I, 170MB/s R, 90MB/s W."...the important thing is the 'A2' spec. this speeds up multiple small files R/W access time, speeds up boot-up time , shortens auto-scanning time when you have just sideloaded several hundred ebooks and also faster searching through ebooks in database when you are looking for a specific book title..

[Semwize]

If you are not actively reading comics and pdf, a 128 gigabyte card is a completely waste of money. Optimal - 32 GB

[93terp]

Originally Posted by davidfor

That seems a bit big. I have about 8000 books on my Clara HD and the database is a little 200MB. Though that is 80% epub, which has less records in the database. But, I wouldn't expect it to be 4 times as much space. But, it will also depend on the length of the comments/synopsis for the book and if you are making annotations.

You could try compressing the database. That should reduce the size, but, the amount will depend on how fragmented the database is. If you are a calibre user, my Kobo Utilities plugin has this as a function. Or you can use any SQLite database manager to do it. And I would recommend taking a backup of the database before doing this.

If you logout of the Kobo account, the database is removed and a new one created when you login again. Then the sideloaded books will be imported again. You will lose any annotations, reading status and collections for these books. Again, if you are a calibre user, some of these can be restored afterwards.

Could I ask that when you do either of the above, that you post the new database size. I'm curious about how much these steps actually help.

After reloading my Aura One LE, there's virtually no change in the database size, both before Kobo Utilities compression (827MB) and post-compression (822MB). Any idea what could be leading to such a large database size?

[koboy]

Originally Posted by 93terp

After reloading my Aura One LE, there's virtually no change in the database size, both before Kobo Utilities compression (827MB) and post-compression (822MB). Any idea what could be leading to such a large database size?

i have 4763 kepub books all of them are on the external Micro SD Card of my original Aura 6" with the latest Firmware 4.25.15875. my database is 977MB. is that large? indeed it is. many of the book are over 100MB which may account for the large size of the database. as long as my device/ereader is working to my satisfaction (which it is) do i give the size of the database a second thought? no.

best wishes koboy

[Ripplinger]

Originally Posted by Semwize

If you are not actively reading comics and pdf, a 128 gigabyte card is a completely waste of money. Optimal - 32 GB

Agree. I've got 2326 books currently on a 32GB card I swapped in my Glo HD and have 25.8GB still free. The actual books take up 2.64GB, the rest are system files, covers, etc. I rarely make any annotations, highlights or notes, and usually delete them once I've finished a book. I would expect system slowdowns if I ever managed to fill the 32GB card, I can't imagine filling 128GB with books.

My database is 133MB for 2326 books. It's mostly epubs with a handful of kepubs, but that shouldn't matter for the database. I've never bothered with compressing the database, never felt a need. While not as compressed as davidfor's, it's still at least 2.5x smaller than yours.

By any chance did you take a look at the size of the database before adding any books back? And during the process of adding them back to see if there was any huge jump within a certain batch of books?