Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Kobo Reader

Notices

Reply
 
Thread Tools Search this Thread
Old 03-10-2013, 02:24 PM   #1
GeoffR
Wizard
GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.
 
GeoffR's Avatar
 
Posts: 1,406
Karma: 2747501
Join Date: Nov 2012
Location: New Zealand
Device: Kobo Glo
Potential problems with long file names

From what I gather the FAT32 filesystem used on the Kobo's user-accessible partition has a limit of 65534 "entries" per directory, and a file with a filename of length N occupies 1+N/13 entries (rounded up). So for example a file with a 50 character filename would occupy 5 entries, and so no more than 13,106 such files could be created in a single directory.

The default file naming scheme in Calibre results in quite long pathnames. For example the book "The Door Through Space" by Marion Zimmer Bradley would be stored as:
"/Bradley, Marion Zimmer/Door Through Space, The - Marion Zimmer Bradley.epub".

The Kobo device stores the cover images for this book in three files named similar to:
"file____mnt_onboard_Bradley,_Marion Zimmer_Door_Through_Space,_The_-_Marion_Zimmer_Bradley_epub - N3_LIBRARY_FULL.parsed"

That is 3 files of about 120 characters each, with the files for all books stored in a single directory /.kobo/images/. If that was a typical book then the directory will only hold images for about 2000 such books.

While space for 2000 books might be plenty for those with a 2GB micro SD card, it is likely to become a problem for those of us who have replaced the stock card with a larger one in the hope of storing more books.

I have an 8GB card in my Glo and hope to have more than 2000 books some day, so I guess I will need to choose a more efficient file naming scheme than the one Calibre uses by default.
GeoffR is offline   Reply With Quote
Old 03-10-2013, 02:59 PM   #2
Mrs_Often
Wizard
Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.Mrs_Often ought to be getting tired of karma fortunes by now.
 
Mrs_Often's Avatar
 
Posts: 1,310
Karma: 1495609
Join Date: Dec 2011
Location: The Netherlands
Device: Kobo Touch, Kobo Glo
Good to know. Thanks for sharing
Mrs_Often is offline   Reply With Quote
 
Enthusiast
Old 03-10-2013, 08:06 PM   #3
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 5,482
Karma: 5446898
Join Date: Jul 2011
Location: Australia
Device: Kobo Touch, Kobo Glo
Yes, it's a known problem. It was discussed a couple of times last year. One of the reports hit a problem with about 3000 books. But, I think that was with a 1.9.x firmware version.

With the last few firmware versions, it isn't a problem for the SD card user. The cover images aren't stored at all. It also isn't quite as bad with the 2.x.x firmware. The 1.9.x firmware generated four cover images when a book was sideloaded. 2.x.x generates them as needed and only generates three (one for the home screen and sleep cover, one for the library lists and one for the details page).

Changing the calibre save template is a simple way to slow down reaching the limit. The default is "{author_sort}/{title} - {authors}", so changing to "{author_sort}/{title}" would reduce the length of the name a lot without losing much info in the name.

An alternative would be to rename the image files and update the database to match. Dropping "file..." prefix would save a bit. That wouldn't be to hard to automate. Or maybe a database trigger on insert?
davidfor is offline   Reply With Quote
Old 03-11-2013, 11:49 AM   #4
GeoffR
Wizard
GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.
 
GeoffR's Avatar
 
Posts: 1,406
Karma: 2747501
Join Date: Nov 2012
Location: New Zealand
Device: Kobo Glo
Quote:
Originally Posted by davidfor View Post
Changing the calibre save template is a simple way to slow down reaching the limit. The default is "{author_sort}/{title} - {authors}", so changing to "{author_sort}/{title}" would reduce the length of the name a lot without losing much info in the name.
I have set the template to {author_sort[0]}/{author_sort:list_item(0,\,)}, {title} to save a bit of space, only the surname of the first author is used. I think this would let me have about 3000-3500 books.

I am thinking about using {author_sort[0]}/{id} so each filename is just the Calibre library ID, which would allow about 5400 books.

Quote:
An alternative would be to rename the image files and update the database to match. Dropping "file..." prefix would save a bit. That wouldn't be to hard to automate. Or maybe a database trigger on insert?
Something along those lines would be nice as an option. Do you think using shorter names would help reduce the database size, or are the fields padded out to fill a fixed size?

Last edited by GeoffR; 03-11-2013 at 11:50 AM. Reason: punctuation
GeoffR is offline   Reply With Quote
Old 03-11-2013, 11:35 PM   #5
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 5,482
Karma: 5446898
Join Date: Jul 2011
Location: Australia
Device: Kobo Touch, Kobo Glo
Quote:
Originally Posted by GeoffR View Post
I have set the template to {author_sort[0]}/{author_sort:list_item(0,\,)}, {title} to save a bit of space, only the surname of the first author is used. I think this would let me have about 3000-3500 books.

I am thinking about using {author_sort[0]}/{id} so each filename is just the Calibre library ID, which would allow about 5400 books.
Both are a good idea and would help. Personally, I don't think I'd like to lose the title from the filename, but, I also don't tend to keep that many books on my devices.
Quote:
Something along those lines would be nice as an option. Do you think using shorter names would help reduce the database size, or are the fields padded out to fill a fixed size?
The columns were the names are stored are not fixed length, so it should mean an overall smaller database. But, I don't know if it will be noticeable.

As to the changes, I'll add it to my list to look at, but it won't be near the top.
davidfor is offline   Reply With Quote
Old 03-22-2013, 06:18 PM   #6
fxp33
Groupie
fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.
 
Posts: 199
Karma: 109220
Join Date: Mar 2013
Location: Bordeaux, France
Device: Kobo Glo, Aura HD... Kobo = pdf without reflow !!
Hi,

Had the same problem, although the metadata were already inside the files (so I didn't have to keep the filename). I was stuck at 10 000 books (converted word/odt/pdf and some gutenberg).

Used Antrenamer to:
- add a number at the end of each file (as a unique 5 digit ID) using the "enumeration" function,
- strip the end of the file name except the first 8 characters (usually the author,'s name) to keep an alphabetical order,
- replace accented characters in one single move (é->e; ï->i; '->_ ...)

This way I could go to 11 700 files at the root of the sdcard. As my filename schema is still longer than the short 8.3 structure, I guess my limit will be around 32000 files instead of the 65000 normal limit of fat32.

Other NOT working solution:
1) tried to format as exfat (fat64) : not recognised.
A pity because this way I could have kept the long filenames.

Hope this helps

François
fxp33 is offline   Reply With Quote
Old 03-22-2013, 07:32 PM   #7
GeoffR
Wizard
GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.GeoffR ought to be getting tired of karma fortunes by now.
 
GeoffR's Avatar
 
Posts: 1,406
Karma: 2747501
Join Date: Nov 2012
Location: New Zealand
Device: Kobo Glo
Quote:
Originally Posted by fxp33 View Post
This way I could go to 11 700 files at the root of the sdcard. As my filename schema is still longer than the short 8.3 structure, I guess my limit will be around 32000 files instead of the 65000 normal limit of fat32.
The limit on the number of .epub files per directory is easily solved by creating subdirectories rather than putting them all in the root directory. I use the first letter of the author's surname as the subdirectory name, so even though some letters are more common than others I should be able to load at least 100,000 books before running into limits.

The problem is that the Kobo stores all the cover images and thumbnails created from the books in a single directory, even if the .epub files are in seperate directories. This means that you have three image files per .epub, all in the same directory, and the filnames are very long, which causes the FAT limit on the number of files in the images directory to be reached much earlier than the limit on the number of books.

Quote:
Other NOT working solution:
1) tried to format as exfat (fat64) : not recognised.
A pity because this way I could have kept the long filenames.
I read somewhere here that a linux ext2/3/4 filesystem would work, apparantly the Kobo can use it, and it doesn't have the limits of FAT filesystems. I haven't tried it yet but might if Kobo haven't come up with a better way to store the images before I run into the FAT directory limits.
GeoffR is offline   Reply With Quote
Old 03-22-2013, 08:05 PM   #8
fxp33
Groupie
fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.
 
Posts: 199
Karma: 109220
Join Date: Mar 2013
Location: Bordeaux, France
Device: Kobo Glo, Aura HD... Kobo = pdf without reflow !!
Hi,

Thanks Geoffr.

My first attempt at using subdirectories didn't succeed. My Kobo seemed not to see the content of subdir. I'll sure give it another try ;-)

François
fxp33 is offline   Reply With Quote
Old 03-23-2013, 02:25 PM   #9
tshering
Wizard
tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.
 
Posts: 1,280
Karma: 332560
Join Date: Jun 2012
Device: kobo touch
I am not planning to put that many books on my Touch. But if I were in this situation, I would try to set the ImageID in the database of several books to the same value in order to make them share the same groupe of images. I would start with those epubs that have cover images showing the first page of text, since almost all of these cover images are useless for me. If there would be the need to save even more file names, I would dispense with other covers images too, since I am not much interested in them anyway.
tshering is offline   Reply With Quote
Old 03-23-2013, 04:53 PM   #10
fxp33
Groupie
fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.fxp33 is far more humble than you.
 
Posts: 199
Karma: 109220
Join Date: Mar 2013
Location: Bordeaux, France
Device: Kobo Glo, Aura HD... Kobo = pdf without reflow !!
Hi ,

Subdirectories work fine... thank you very much Geoffr for teaching me this. I really don't know why it didn't work / I didn't find the books - when I first tried.

... and 102 Balzac books in a folder !! my collection will soon be a lot quicker to load in a explorer window :-)

François
fxp33 is offline   Reply With Quote
Old 03-23-2013, 10:03 PM   #11
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 5,482
Karma: 5446898
Join Date: Jul 2011
Location: Australia
Device: Kobo Touch, Kobo Glo
Quote:
Originally Posted by tshering View Post
I am not planning to put that many books on my Touch. But if I were in this situation, I would try to set the ImageID in the database of several books to the same value in order to make them share the same groupe of images. I would start with those epubs that have cover images showing the first page of text, since almost all of these cover images are useless for me. If there would be the need to save even more file names, I would dispense with other covers images too, since I am not much interested in them anyway.
Two things I haven't tried before but worked with the tests I just did:

- If you don't care about the covers, you can set the ImageId to null. This will use the text cover with just the title and type of book on it. No images are stored for them.

- Ages ago someone, PeterT I think, suggested using subdirectories in the images folder. That needs a change to add a slash somewhere in the ImageId and moving and renaming the image files appropriately. I have never gotten around to testing it until now, but it seems to work.

As with the other solutions, they need manual intervention but automating them wouldn't be to hard.
davidfor is offline   Reply With Quote
Old 03-24-2013, 01:27 PM   #12
tshering
Wizard
tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.
 
Posts: 1,280
Karma: 332560
Join Date: Jun 2012
Device: kobo touch
Quote:
Originally Posted by davidfor View Post
- If you don't care about the covers, you can set the ImageId to null. This will use the text cover with just the title and type of book on it. No images are stored for them.
I understand this as meaning that the covers look like those of kepub.epubs. And this is something I would like. But what I see is that the covers show a miniature of the first page of the book. (Edit: A little later I found that you are saying this yourself in another post:
Quote:
Originally Posted by davidfor View Post
Except for TXT formats books, the Kobo devices use the first page of the book as the cover. If it doesn't that is a sign of a problem. There isn't any option to turn this off. And if you look around the forum, there has been a LOT of discussion on getting the covers right.
Anyway, you are right in saying that no images are stored for them. With respect to processing time, however, I guess it might be better to set the imageId-s of all books (or a selected group of books) to the same value so that the share the same stored image files.
Quote:
Originally Posted by davidfor View Post
As with the other solutions, they need manual intervention but automating them wouldn't be to hard.
I am not sure what solutions you are referring to.

Last edited by tshering; 03-24-2013 at 02:36 PM.
tshering is offline   Reply With Quote
Old 03-24-2013, 06:51 PM   #13
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 5,482
Karma: 5446898
Join Date: Jul 2011
Location: Australia
Device: Kobo Touch, Kobo Glo
Quote:
Originally Posted by tshering View Post
I understand this as meaning that the covers look like those of kepub.epubs. And this is something I would like. But what I see is that the covers show a miniature of the first page of the book. (Edit: A little later I found that you are saying this yourself in another post:
Anyway, you are right in saying that no images are stored for them. With respect to processing time, however, I guess it might be better to set the imageId-s of all books (or a selected group of books) to the same value so that the share the same stored image files.
That is the type of cover I meant. But I was also saying that if the ImageId is set to NULL, no cover image is ever generated and that text only cover it used. Performance wise, I don't know if this or the same image for all books would be the best. If the cover doesn't exist, then the text cover flashes up before being replaced by the generated cover. If the cover image already exists, I think this still happens but isn't as noticeable.

If you are using an SD card, setting the ImageId to null should prevent the generation of any cover images.
Quote:
I am not sure what solutions you are referring to.
I made some suggestions on manipulating the ImageId in an earlier post. At the moment, my favourite solution is using subdirectories. If the directory structure for the books is replicated for the images, then the directory size limits won't be hit. But the path length would have to be watched.
davidfor is offline   Reply With Quote
Old 03-25-2013, 05:25 AM   #14
tshering
Wizard
tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.tshering ought to be getting tired of karma fortunes by now.
 
Posts: 1,280
Karma: 332560
Join Date: Jun 2012
Device: kobo touch
davidfor,
thank you for clarification.
tshering is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is there anything in Calibre that can fix long file names automatically? dmunson Calibre 4 02-05-2012 06:12 PM
File Names to long? JTAL604622 Calibre 5 08-31-2011 03:21 AM
Kindle 2 GSM version potential to cause problems? vivekmreddy Amazon Kindle 9 12-09-2009 12:59 PM
ADE and long chapter names in ToC frabjous Reading and Management 3 08-14-2009 11:16 AM
Files with long names? cmhsieh54 iRex 0 08-05-2009 12:43 PM


All times are GMT -4. The time now is 06:33 PM.


MobileRead.com is a privately owned, operated and funded community.