Potential problems with long file names

[GeoffR]

From what I gather the FAT32 filesystem used on the Kobo's user-accessible partition has a limit of 65534 "entries" per directory, and a file with a filename of length N occupies 1+N/13 entries (rounded up). So for example a file with a 50 character filename would occupy 5 entries, and so no more than 13,106 such files could be created in a single directory.

The default file naming scheme in Calibre results in quite long pathnames. For example the book "The Door Through Space" by Marion Zimmer Bradley would be stored as:
"/Bradley, Marion Zimmer/Door Through Space, The - Marion Zimmer Bradley.epub".

The Kobo device stores the cover images for this book in three files named similar to:
"file____mnt_onboard_Bradley,_Marion Zimmer_Door_Through_Space,_The_-_Marion_Zimmer_Bradley_epub - N3_LIBRARY_FULL.parsed"

That is 3 files of about 120 characters each, with the files for all books stored in a single directory /.kobo/images/. If that was a typical book then the directory will only hold images for about 2000 such books.

While space for 2000 books might be plenty for those with a 2GB micro SD card, it is likely to become a problem for those of us who have replaced the stock card with a larger one in the hope of storing more books.

I have an 8GB card in my Glo and hope to have more than 2000 books some day, so I guess I will need to choose a more efficient file naming scheme than the one Calibre uses by default.

[Mrs_Often]

Good to know. Thanks for sharing

[davidfor]

Yes, it's a known problem. It was discussed a couple of times last year. One of the reports hit a problem with about 3000 books. But, I think that was with a 1.9.x firmware version.

With the last few firmware versions, it isn't a problem for the SD card user. The cover images aren't stored at all. It also isn't quite as bad with the 2.x.x firmware. The 1.9.x firmware generated four cover images when a book was sideloaded. 2.x.x generates them as needed and only generates three (one for the home screen and sleep cover, one for the library lists and one for the details page).

Changing the calibre save template is a simple way to slow down reaching the limit. The default is "{author_sort}/{title} - {authors}", so changing to "{author_sort}/{title}" would reduce the length of the name a lot without losing much info in the name.

An alternative would be to rename the image files and update the database to match. Dropping "file..." prefix would save a bit. That wouldn't be to hard to automate. Or maybe a database trigger on insert?

[GeoffR]

Originally Posted by davidfor

Changing the calibre save template is a simple way to slow down reaching the limit. The default is "{author_sort}/{title} - {authors}", so changing to "{author_sort}/{title}" would reduce the length of the name a lot without losing much info in the name.

I have set the template to {author_sort[0]}/{author_sort:list_item(0,\,)}, {title} to save a bit of space, only the surname of the first author is used. I think this would let me have about 3000-3500 books.

I am thinking about using {author_sort[0]}/{id} so each filename is just the Calibre library ID, which would allow about 5400 books.

An alternative would be to rename the image files and update the database to match. Dropping "file..." prefix would save a bit. That wouldn't be to hard to automate. Or maybe a database trigger on insert?

Something along those lines would be nice as an option. Do you think using shorter names would help reduce the database size, or are the fields padded out to fill a fixed size?

[davidfor]

Originally Posted by GeoffR

I have set the template to {author_sort[0]}/{author_sort:list_item(0,\,)}, {title} to save a bit of space, only the surname of the first author is used. I think this would let me have about 3000-3500 books.

I am thinking about using {author_sort[0]}/{id} so each filename is just the Calibre library ID, which would allow about 5400 books.

Both are a good idea and would help. Personally, I don't think I'd like to lose the title from the filename, but, I also don't tend to keep that many books on my devices.

Something along those lines would be nice as an option. Do you think using shorter names would help reduce the database size, or are the fields padded out to fill a fixed size?

The columns were the names are stored are not fixed length, so it should mean an overall smaller database. But, I don't know if it will be noticeable.

As to the changes, I'll add it to my list to look at, but it won't be near the top.

[fxp33]

Hi,

Had the same problem, although the metadata were already inside the files (so I didn't have to keep the filename). I was stuck at 10 000 books (converted word/odt/pdf and some gutenberg).

Used Antrenamer to:
- add a number at the end of each file (as a unique 5 digit ID) using the "enumeration" function,
- strip the end of the file name except the first 8 characters (usually the author,'s name) to keep an alphabetical order,
- replace accented characters in one single move (é->e; ï->i; '->_ ...)

This way I could go to 11 700 files at the root of the sdcard. As my filename schema is still longer than the short 8.3 structure, I guess my limit will be around 32000 files instead of the 65000 normal limit of fat32.

Other NOT working solution:
1) tried to format as exfat (fat64) : not recognised.
A pity because this way I could have kept the long filenames.

Hope this helps

François

[GeoffR]

Originally Posted by fxp33

This way I could go to 11 700 files at the root of the sdcard. As my filename schema is still longer than the short 8.3 structure, I guess my limit will be around 32000 files instead of the 65000 normal limit of fat32.

The limit on the number of .epub files per directory is easily solved by creating subdirectories rather than putting them all in the root directory. I use the first letter of the author's surname as the subdirectory name, so even though some letters are more common than others I should be able to load at least 100,000 books before running into limits.

The problem is that the Kobo stores all the cover images and thumbnails created from the books in a single directory, even if the .epub files are in seperate directories. This means that you have three image files per .epub, all in the same directory, and the filnames are very long, which causes the FAT limit on the number of files in the images directory to be reached much earlier than the limit on the number of books.

Other NOT working solution:
1) tried to format as exfat (fat64) : not recognised.
A pity because this way I could have kept the long filenames.

I read somewhere here that a linux ext2/3/4 filesystem would work, apparantly the Kobo can use it, and it doesn't have the limits of FAT filesystems. I haven't tried it yet but might if Kobo haven't come up with a better way to store the images before I run into the FAT directory limits.

[fxp33]

Hi,

Thanks Geoffr.

My first attempt at using subdirectories didn't succeed. My Kobo seemed not to see the content of subdir. I'll sure give it another try ;-)

François

[tshering]

I am not planning to put that many books on my Touch. But if I were in this situation, I would try to set the ImageID in the database of several books to the same value in order to make them share the same groupe of images. I would start with those epubs that have cover images showing the first page of text, since almost all of these cover images are useless for me. If there would be the need to save even more file names, I would dispense with other covers images too, since I am not much interested in them anyway.

[fxp33]

Hi ,

Subdirectories work fine... thank you very much Geoffr for teaching me this. I really don't know why it didn't work / I didn't find the books - when I first tried.

... and 102 Balzac books in a folder !! my collection will soon be a lot quicker to load in a explorer window :-)

François

[davidfor]

Originally Posted by tshering

I am not planning to put that many books on my Touch. But if I were in this situation, I would try to set the ImageID in the database of several books to the same value in order to make them share the same groupe of images. I would start with those epubs that have cover images showing the first page of text, since almost all of these cover images are useless for me. If there would be the need to save even more file names, I would dispense with other covers images too, since I am not much interested in them anyway.

Two things I haven't tried before but worked with the tests I just did:

- If you don't care about the covers, you can set the ImageId to null. This will use the text cover with just the title and type of book on it. No images are stored for them.

- Ages ago someone, PeterT I think, suggested using subdirectories in the images folder. That needs a change to add a slash somewhere in the ImageId and moving and renaming the image files appropriately. I have never gotten around to testing it until now, but it seems to work.

As with the other solutions, they need manual intervention but automating them wouldn't be to hard.

[tshering]

Originally Posted by davidfor

- If you don't care about the covers, you can set the ImageId to null. This will use the text cover with just the title and type of book on it. No images are stored for them.

I understand this as meaning that the covers look like those of kepub.epubs. And this is something I would like. But what I see is that the covers show a miniature of the first page of the book. (Edit: A little later I found that you are saying this yourself in another post:

Originally Posted by davidfor

Except for TXT formats books, the Kobo devices use the first page of the book as the cover. If it doesn't that is a sign of a problem. There isn't any option to turn this off. And if you look around the forum, there has been a LOT of discussion on getting the covers right.

Anyway, you are right in saying that no images are stored for them. With respect to processing time, however, I guess it might be better to set the imageId-s of all books (or a selected group of books) to the same value so that the share the same stored image files.

Originally Posted by davidfor

As with the other solutions, they need manual intervention but automating them wouldn't be to hard.

I am not sure what solutions you are referring to.

[davidfor]

Originally Posted by tshering

I understand this as meaning that the covers look like those of kepub.epubs. And this is something I would like. But what I see is that the covers show a miniature of the first page of the book. (Edit: A little later I found that you are saying this yourself in another post:
Anyway, you are right in saying that no images are stored for them. With respect to processing time, however, I guess it might be better to set the imageId-s of all books (or a selected group of books) to the same value so that the share the same stored image files.

That is the type of cover I meant. But I was also saying that if the ImageId is set to NULL, no cover image is ever generated and that text only cover it used. Performance wise, I don't know if this or the same image for all books would be the best. If the cover doesn't exist, then the text cover flashes up before being replaced by the generated cover. If the cover image already exists, I think this still happens but isn't as noticeable.

If you are using an SD card, setting the ImageId to null should prevent the generation of any cover images.

I am not sure what solutions you are referring to.

I made some suggestions on manipulating the ImageId in an earlier post. At the moment, my favourite solution is using subdirectories. If the directory structure for the books is replicated for the images, then the directory size limits won't be hit. But the path length would have to be watched.

[tshering]

davidfor,
thank you for clarification.