Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 04-22-2013, 05:25 AM   #1
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Problem with high number of books on device

Hi,

<long intro>
in the past I had all my books on my Sony PRS-T1 (around 5000, with 2000 series). Finding a book was a bit slow at times, but all in all worked pretty well.

Now I bought a Kobo Glo, and while I'm very happy with the hardware, the software is beginning to irritate me. I put the 8GB sd card I had in my PRS-T1 in the device (formatted as FAT on a Mac) and used calibre to move all my books over there. The device needed a few hours to parse the information (which is ok in my opinion) and then, when I tried to open the list of shelves, simply stopped... Finding and opening a single book worked at least once. Now, when I reboot, the system halts if I do not remove the sd card... If I then put the sd card back into the device it needs half an hour and then starts the parsing again...

This leads me to the conclusion that the Kobo might not be able to work with the number of series I have, and is only barely able to work with the number of books (aah, how I would like my PRS T1 to glow in the dark).

<rambling>
Now, for a small number of books the Kobo is quite nice... (if you ignore the missing back button, but don't get me started on that).
</rambling>
</long intro>

I have marked the books that I want to read with a custom column, in the case of series the first in the series. Is there a way to not only select all books based on the custom column, but also all books that are in the same series?

Cheers, Joachim
jbaumann is offline   Reply With Quote
Old 04-22-2013, 08:27 AM   #2
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
At the moment, the series information on the Kobo devices is only used in the library lists and during a search. It shouldn't have an affect on the performance anywhere in the device.

But, I think you are actually using the series column as the shelf column. Which means you are creating a shelf for each series. In that case you will have 2000 shelves with your 5000 books spread over them (assuming they are all in a series). In that case, opening the shelf list will take a long time. But, I know someone has reported similar figures before and was able to get to the shelf list. They decided it didn't work for them as there is no way to search for the shelves and the list was just to long.

The restart and insert time can be beaten by not powering off the device. Just let it go to sleep without powering off. This uses virtually no power and the battery will last a long time in this state.

After the turning the device on and inserting the SD card, is the black processing screen displayed? Or is the half hour how long it takes before the device shows the books on the SD card in the library lists?

As to the query in calibre, I don't think it can be done. The search and templates work on metadata for a book and can't refer to another book. You can easily select all books with a particular value in a column. You can also select all books in a series. But, I don't think you can combine them in the way you want to.
davidfor is offline   Reply With Quote
Old 04-22-2013, 10:57 AM   #3
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Hi David,

thank for the answer, and yes, I'm using the series column as shelf information (forgot to mention that in my first posting). Since I additionally use author_sort as shelf, I could most probably could cut the number of shelves in half.

But I fear that even then the waiting time will be intolerable, and thus I thought that by only loading the books on the reader that I previously marked as "to_read", adding to that the accompanying books in the respective series, which is why I came up with the idea of something like a "convex envelope".

But I already got the impression that you only have the metadata of the current book and not of others. Would it be possible to add information to the column data? E.g., add a marker to a series entry?

Regarding startup times: After I ran out of patience I did a hard reset (pinhole) and then waited for another eternity before removing the sd card. And it is half an hour before it starts to show the black screen (the processing itself needs 6-8 hours).

Cheers, Joachim
jbaumann is offline   Reply With Quote
Old 04-22-2013, 04:43 PM   #4
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Hi,

now it becomes strange. The internal memory of the Kobo seems to be 100% full (as shown on the device properties page). I haven't put a single book on the internal storage. Can I clean it somehow? This might have to do with the low speed...

And as I understood it from another thread, some information of the sd card is cached in the internal memory. If it tries to cache 5000 books it might lead to that...

Cheers, Joachim
jbaumann is offline   Reply With Quote
Old 04-22-2013, 09:47 PM   #5
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by jbaumann View Post
Hi,

<long intro>
in the past I had all my books on my Sony PRS-T1 (around 5000, with 2000 series). Finding a book was a bit slow at times, but all in all worked pretty well.

Now I bought a Kobo Glo, and while I'm very happy with the hardware, the software is beginning to irritate me. I put the 8GB sd card I had in my PRS-T1 in the device (formatted as FAT on a Mac) and used calibre to move all my books over there. The device needed a few hours to parse the information (which is ok in my opinion) and then, when I tried to open the list of shelves, simply stopped... Finding and opening a single book worked at least once. Now, when I reboot, the system halts if I do not remove the sd card... If I then put the sd card back into the device it needs half an hour and then starts the parsing again...

This leads me to the conclusion that the Kobo might not be able to work with the number of series I have, and is only barely able to work with the number of books (aah, how I would like my PRS T1 to glow in the dark).

<rambling>
Now, for a small number of books the Kobo is quite nice... (if you ignore the missing back button, but don't get me started on that).
</rambling>
</long intro>

I have marked the books that I want to read with a custom column, in the case of series the first in the series. Is there a way to not only select all books based on the custom column, but also all books that are in the same series?

Cheers, Joachim
IN calibre Alt+Shift+S will get all books in the series, but only for one series at a time.

On my mini the more books you have in a collection, the longer it takes to open the shelves.

I had one collection of 400 books which made the shelf opening time more than 40 seconds. removing thet collection (not removing the books) cut the time to about 22 seconds.

I had 3 largish collections of maybe 60 books, removing these collections cut the time to 11 seconds.

These books were still on reader in series collections with less than 20 books each, so it seemed to me that the number of books in each collection, is the big factor since I still had 400+books and 92 collections instead of 96.

My T1 has more than 2000 books and 790 collections, with several collections of 90+ books and takes less than 3 seconds to open the collection menu. Go figure
speakingtohe is offline   Reply With Quote
Old 04-23-2013, 04:03 AM   #6
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Joachim,

I understand what you want to do. I decided recently that when reading a book that was in a series, I would have the rest of the series on the device as well. Or at least the earlier books. But, I only have about 200 books on it, so manually doing that in calibre isn't that hard.

For what you are doing, I would look at the Reading List plugin. With this you can add books to a list and then easily sync that list to the device.

The processing of the SD card seems to be in two phases. The first is reading the card and checking for changes - books added or deleted. Any deletions seem to be removed from the database at this time. If there are additions, then the process screen comes up for the second phase. How long that takes depends on how many new books there are. The last time I tried 500 books, it took less than 10 minutes, so eight hours for 5000 seems to long. That was with FW2.4.0. Older firmware were slower. I haven't tried with 2.5.0.

Recent firmware do not store anything for the books on the SD card in the main memory except for details in the database. Some of the older firmware did store the cover images in the main memory. You will need to look at it when connected to the PC to see what is taking the space.

And as speakingtohe said, the time to open the shelf list is based on how many shelves and how many books are on them. One large shelf tends to affect the time as much as adding a lot of empty shelves.
davidfor is offline   Reply With Quote
Old 04-23-2013, 02:29 PM   #7
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Hi David,

I already had a look at the Reading List Plugin, but at least right now it also does not seem to allow such a kind of filtering.

I have to disagree regarding storing no data on the main memory. In my experiments the internal memory got filled up to 100% (and I have no idea how to clean that up). After a factory reset, on an empty Kobo Glo, I had 1332 of 1361MB available. After copying 220 books on the reader 1266MB remain, which means somewhere are stored, on average, 300K worth of data for each book. And since the books are, on average, larger than 300K it won't be the whole book. When I looked at the mounted volumes, though, there was next to nothing on the internal memory.

This limits the number of books that could be copied even on an arbitrarily large sd card to something around 4.000 books.

Btw., I'm using 2.5.0.

Cheers, Joachim
jbaumann is offline   Reply With Quote
Old 04-23-2013, 11:32 PM   #8
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by jbaumann View Post
I already had a look at the Reading List Plugin, but at least right now it also does not seem to allow such a kind of filtering.
Sorry, I wasn't thinking of the filtering. I find it a useful way to collect a list together for my device. For what you are doing, you would still need to manually select the books and add them to a list. But after that you can maintain the list on the device easily.
Quote:
I have to disagree regarding storing no data on the main memory. In my experiments the internal memory got filled up to 100% (and I have no idea how to clean that up). After a factory reset, on an empty Kobo Glo, I had 1332 of 1361MB available. After copying 220 books on the reader 1266MB remain, which means somewhere are stored, on average, 300K worth of data for each book. And since the books are, on average, larger than 300K it won't be the whole book. When I looked at the mounted volumes, though, there was next to nothing on the internal memory.

This limits the number of books that could be copied even on an arbitrarily large sd card to something around 4.000 books.
Sorry, I wasn't trying to say space wasn't being used. But my understanding of how the devices work and my experience has been that no data is stored on the main memory for books on the SD card, except for entries in the database. This might have changed with FW2.5.0, but I haven't checked. I also don't use the SD card much. There is a card in my Glo, but there are only 10 books on it while I have about 200 on the main memory.

The question is what is taking the space. The only way to know is to connect the device and look at the drive. I am interested to know what it is so that I can look and see if that is happening here and why it is happening. I have just looked at mine, and there is nothing obvious on the main memory that could have come from the SD card.
davidfor is offline   Reply With Quote
Old 04-24-2013, 12:10 AM   #9
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by davidfor View Post
The question is what is taking the space. The only way to know is to connect the device and look at the drive. I am interested to know what it is so that I can look and see if that is happening here and why it is happening. I have just looked at mine, and there is nothing obvious on the main memory that could have come from the SD card.
Where are the books covers stored?
DoctorOhh is offline   Reply With Quote
Old 04-24-2013, 12:22 AM   #10
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
As of FW2.5.0, they are stored in the same memory that the book is. Before that, no covers were stored for books on the SD card.
davidfor is offline   Reply With Quote
Old 04-24-2013, 02:21 AM   #11
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Hi,

finally had the idea to use the command line to look at the card instead of the Mac Finder...

In the .kobo/ directory there is a sqlite instance with 16M and an image/ dir with nearly 60MB. This contains, for each file on the sd card, three files, prefixed with "file____mnt_sd_" followed by author and title and then the suffix epub - N3_FULL.parsed, epub - N3_LIBRARY_FULL.parsed, and pub - N3_LIBRARY_GRID.parsed, respectively. I don't know whether these files contain the book covers as well, but with a size of 50-300K per file and the name of the directory it is at least probable.

Interestingly, I even have kepub directory with a few entries even though I haven't yet bought a Kobo book (and they aren't shown either).

Cheers, Joachim
jbaumann is offline   Reply With Quote
Old 04-24-2013, 03:26 AM   #12
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by jbaumann View Post
finally had the idea to use the command line to look at the card instead of the Mac Finder...
I didn't realise you were using a Mac or I could have warned you about that. There are some instructions somewhere in the Kobo forum about viewing the ".kobo" directory in Finder.
Quote:
In the .kobo/ directory there is a sqlite instance with 16M and an image/ dir with nearly 60MB. This contains, for each file on the sd card, three files, prefixed with "file____mnt_sd_" followed by author and title and then the suffix epub - N3_FULL.parsed, epub - N3_LIBRARY_FULL.parsed, and pub - N3_LIBRARY_GRID.parsed, respectively. I don't know whether these files contain the book covers as well, but with a size of 50-300K per file and the name of the directory it is at least probable.
Those are the cover images and they are just JPEGs. I have no idea why Kobo thought they needed a special extension.

They shouldn't be there and they are definiately not being created on either my Glo or Touch. FW 2.5.0 creates cover image files for books on the SD card in the directory "koboExtStorage\images" on the SD card. Older firmware put them on the main memory, but the latest that did was either 2.0.x or 2.1.1. But, older versions of calibre could create the images on the main memory when you sent books over with the appropriate options. What version of calibre are you running?

In any case, you should be safe deleting the image files. The device will recreate them if needed and they should go to the correct place.
Quote:
Interestingly, I even have kepub directory with a few entries even though I haven't yet bought a Kobo book (and they aren't shown either).
There usually are about five of these and they aren't in the database correctly to see. From memory, they are classics like Moby Dick or something from Mark Twain.
davidfor is offline   Reply With Quote
Old 04-24-2013, 02:52 PM   #13
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Hm,

I'm using the newest version of calibre and I only used 2.5.0. I just had a look at the KoboTouch-Plugin-configuration and there the option "Upload covers for books" is checked. Could have been me a few weeks ago when I played with the plugin-configuration...

Might this be the cause of the extra book covers in the main memory?

Cheers, Joachim
PS: I'll try it...
jbaumann is offline   Reply With Quote
Old 04-24-2013, 10:15 PM   #14
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by jbaumann View Post
Hm,

I'm using the newest version of calibre and I only used 2.5.0. I just had a look at the KoboTouch-Plugin-configuration and there the option "Upload covers for books" is checked. Could have been me a few weeks ago when I played with the plugin-configuration...

Might this be the cause of the extra book covers in the main memory?

Cheers, Joachim
PS: I'll try it...
The current version of calibre shouldn't copy covers to the device if the book is on the SD card. That was fixed in calibre 0.9.21. But, anything put there before that, will still be there. Of course, there could be a bug.

Could you run calibre in debug mode and send me a log? To do this:

- Open the driver configuration and put something in the debug title. This needs to be part of the name of a book you can send to the device.
- Restart calibre in debug mode - Right-click the preferences button and choose "restart in debug mode"
- When calibre restarts, connect the device and send the book to the SD card
- Close calibre.
- The debug log should be displayed. Post this or send it to me in a PM
- Check if the cover was created on the device in the main memory.

If there is something going wrong, hopefully the log will show it.
davidfor is offline   Reply With Quote
Old 04-25-2013, 01:53 AM   #15
jbaumann
Zealot
jbaumann began at the beginning.
 
Posts: 106
Karma: 10
Join Date: Nov 2011
Device: Multiple Kobos, multiple Kindles
Ok, I put all of my books on the reader again, and now there is no extra memory needed. The Kobo still digests the new books, but to me it seems faster than before when the internal memory was full (which seems logical). Also the long time until it started to process the data is now down to only about half a second.

Regarding the speed: The speed for looking up books is better than on the PRS-T1 now, but for browsing shelves, that still is slow. Around 750 shelves need 3 minutes to show.

Thus it works but with the slow shelf management isn't practical.

Thanks for all your help, Joachim
jbaumann is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Sending Series Name/Number to device Susan in Houston Calibre 1 10-26-2012 07:50 AM
Books indexing problem caused by the device. sirostr Amazon Kindle 1 07-08-2012 04:53 PM
Problem moving books to device cmcgranahan Calibre 6 01-11-2011 07:04 PM
Questio: Totla number of books on device Rootman Calibre 1 11-17-2009 09:20 AM


All times are GMT -4. The time now is 12:54 AM.


MobileRead.com is a privately owned, operated and funded community.