Multiple Folder; do they slow things down?

[wallcraft]

I was surprised a few weeks ago to see a report of ~30s wait times when going back to home on the K3 with ~1,500 ebooks. I have ~1,000 ebooks on my K3, all managed by Calibre. They are spread across 11 folders, based on tags. I suspect even more folders would help (a sub-folder for each author for example) , but even with 11 folders I see no delay switching back to home when in "most recent" mode. It only takes a second or two to switch between home's ebook sorting options, but a search can take a while.

I have not been able to find definitive information on FAT32 file operations, but it looks like the Directory table is a linear list (i.e. a table). No modern general purpose filesystem would be designed this way, because average and worse case performance of some single-file operations will be proportional to the number of files in the folder. See, for example, a discussion of how Linux's ext3 improved on ext2 at Directory indexing. I'm not sure O(n) file performance is an issue in practice on the Kindle (even though there are multiple files per ebook), but it could be and adding sub-folders is easy with Calibre and likely can't hurt.

[jocampo]

Originally Posted by wallcraft

I was surprised a few weeks ago to see a report of ~30s wait times when going back to home on the K3 with ~1,500 ebooks. I have ~1,000 ebooks on my K3, all managed by Calibre. They are spread across 11 folders, based on tags. I suspect even more folders would help (a sub-folder for each author for example) , but even with 11 folders I see no delay switching back to home when in "most recent" mode. It only takes a second or two to switch between home's ebook sorting options, but a search can take a while.

I have not been able to find definitive information on FAT32 file operations, but it looks like the Directory table is a linear list (i.e. a table). No modern general purpose filesystem would be designed this way, because average and worse case performance of some single-file operations will be proportional to the number of files in the folder. See, for example, a discussion of how Linux's ext3 improved on ext2 at Directory indexing. I'm not sure O(n) file performance is an issue in practice on the Kindle (even though there are multiple files per ebook), but it could be and adding sub-folders is easy with Calibre and likely can't hurt.

Wallcraft,

Very interesting post!

[terencek]

I think it was my post you originally saw, where accessing the Home screen takes about 30 seconds for my collection of more than 1,700 books on the device. I do indeed use Calibre to manage my books, so that every book is actually in a folder (or to be correct a sub-folder) in the Documents directory.

Initially I thought it might be a problem that each book was also accompanied by a separate (but redundant) cover.jpg file and so I did a global deletion of them all. That saved me a hundred MB or so, but still didn't speed up Home screen access.

My current statistics, running a Properties check on the device from Windows 7, shows 1,711 files (i.e. ebooks), and 2,373 folders.

What I haven't done is run a fragmentation assessment utility on the Kindle -- but this is probably academic anyway given that most expert advice is not to defragment a SSD.

[wallcraft]

Originally Posted by terencek

I think it was my post you originally saw, where accessing the Home screen takes about 30 seconds for my collection of more than 1,700 books on the device. I do indeed use Calibre to manage my books, so that every book is actually in a folder (or to be correct a sub-folder) in the Documents directory.

For 1,000 ebooks (say), what might be near optimal is 10 folders each containing 10 folders each containing 10 ebooks. A strategy that is easy to get to from where you are would be to start with 26 subfolders, A-Z, based on the 1st letter of the author (say). This can be setup as the target in Calibre, but for the existing ebooks (if the filename starts with the author) this would be only be 26 cut and paste operations on the USB mounted device. The cut and paste will move the files (mv in Linux), rather than making new copies (cp in Linux), but the Kindle might still decide to re-index them. You don't have to do them all at once, e.g. try A first and see what happens.

[lestatar]

^ this is an awesome thread. Thank you all for your input.

While no stranger to ebook formats and even calibre, I have only now started using calibre to transfer books over to my first dedicated reader device. I never had a need to dig deeper into calibre beyond conversion tasks.

The tips about enabling/disabling subfolders created on Kindles/devices by calibre is priceless to me.

Being new to dedicated ereaders but a very old hand at tech and folder structures for stuff like music, I am still working out the best workflow process for myself and to get ebooks onto my devices.

My own preference for both music and books is to avoid subfolders, for artists or in this case, authors. It is indeed a shame that [at present] real folder browsing on the K3 is not possible - I do hope that Amazon will make this available soon through simple FW update. The only type of sub-folders I might employ for ebooks would be category based, rather than author based as this is how I think I want to organize my digital library.

Please pardon any noob-ish questions below and apologies if I should post on the general discussion subforum:

- Does anyone who doesn't use sub-folder structures on Kindles know if there are any upper file count limits in the root book folders to be aware of?

- What is the exact/default folder name [when connected to PC] that the Kindle uses to look for books?

- Are there any folder file count limits for Android devices and ebooks, also on the root book folders?

- Meta-tags: Is there any type of standard meta-tag format or a universal meta-tag editor for ebooks? Highly unlikely I would guess, if ebooks are anything like digital music [which sucks].

- Do ebook devices like Kindles and Android apps prefer tag into or can they also work off of filenames?

My 20k+ track music library is exclusively HQ LAME mp3. As such, I have no need to worry about various formats and metatag info. It took a long time, but I have my audio collection exactly how I want it. I would like to achieve the same thing with my own huge, but at present, completely disorganized ebook library.

Physically my music lives in this folder hierarchy:
E:\Music
E:\Music\0-9
E:\Music\A-C
E:\Music\D-F
etc

I think I would like to do this for ebooks:
E:\eBooks
E:\eBooks\Horror
E:\eBooks\SciFi
E:\eBooks\Fantasy
E:\eBooks\Non-Fiction
E:\eBooks\Favorite Authors
etc

I also plan to rename all my ebook files to adhere to this kind of uniform file naming convention:

George RR Martin - 1996 - Song Of Ice And Fire 01 - A Game Of Thrones.mobi [AZW, ePub, PDF, whatever].

This will allow me not to need Author sub-folders and organize my books simply by general category.

Anal perhaps, but that's how I organize my stuff.

Any advice from the gurus around here? And I am totally open to other options for organizing my digital book library. It took forever for me to retroactively fix my music collection and get it exactly how I like it. Now, armed with that experience, I would like to get a leg up and do things right with my ebooks, regardless of device.

cheers, tia and sorry for derailing this thread,
-les

[emalvick]

In terms of what you want to do for organizing your files, you could utilize Calibre to organize the naming. It may not be as simple as what you find in terms of music tagging and naming software, but I believe when you convert a file in Calibre, you can dictate its name. In other words, you might convert a mobi file to a mobi file to get the file name you want. It isn't as elegant as music files can be for just renaming, but it could work (I haven't tried it yet myself).

As for naming scheme. You really should choose what's right for you, but you probably already know that. By relying on programs like Calibre for my library (and other software for my music), I think of the file names as more of a backup to the most important info about the file, i.e. the bare minimum I would need to build a database again should I completely lose it. For a book, it seems like author and title would be a bare minimum. You've got that covered and then some. After that it just depends on what you have in your collection and how you look for files.

As an aside, I have a music library that exceeds 20,000 tracks now. However, I organize tracks differently because I tend to be an album oriented listener and it works best for me if each album is in its own subfolder. It doesn't mean it can't work in another configuration, but it is how I've done things. Books should be the same way.

[CWatkinsNash]

Originally Posted by lestatar

While no stranger to ebook formats and even calibre, I have only now started using calibre to transfer books over to my first dedicated reader device. I never had a need to dig deeper into calibre beyond conversion tasks.

The tips about enabling/disabling subfolders created on Kindles/devices by calibre is priceless to me.

Being new to dedicated ereaders but a very old hand at tech and folder structures for stuff like music, I am still working out the best workflow process for myself and to get ebooks onto my devices.

Welcome to the calibre club! There are a lot of powerful options to consider. I can't answer all your questions (no idea regarding file limits) but I will try to answer a few.

- What is the exact/default folder name [when connected to PC] that the Kindle uses to look for books?

It looks for everything in the Documents folder. Beyond that it doesn't seem to matter as long as it's all inside that folder.

- Meta-tags: Is there any type of standard meta-tag format or a universal meta-tag editor for ebooks? Highly unlikely I would guess, if ebooks are anything like digital music [which sucks].

They are like music files in the sense that there are two types of limitations: the metadata spec is different for different file formats (like MP3 vs WAV), and how much is used of any given metadata spec depends on the software being used to read it. For example, PDF can hold only the most basic metadata in the file itself; with ePub, you can specify several metadata fields but most reader software can't use most of it. Then you have fields that should be standard but aren't.

I don't think you'll be surprised by that answer though, considering we STILL don't have full support for the MP3 composer field.

- Do ebook devices like Kindles and Android apps prefer tag into or can they also work off of filenames?

Kindle uses the title and author metadata. If there is none, it uses the filename as the title, IIRC. Android apps vary. Some apps read limited metadata, others read filenames.

I also plan to rename all my ebook files to adhere to this kind of uniform file naming convention:

A few things to look at:

In calibre Preferences > Send to Device: Allows you to specify a template that dictates how the files will be organized on the device, including folder structure and filename.

Preferences> Save to Disk: Allows you to specify a template when using the Save to Disk feature. This is useful in the event that you'd like to have a copy of your library in your preferred file structure. This way, you have a copy that makes you happy (and also serves as a backup) and calibre gets to do its thing in its own library. Cuz calibre has to do its thing in its own library.

Plugboards: If you want to see series names or such on the Kindle, this is your go-to feature. There's a sticky in the calibre forum which explains how it works and has many useful template examples.

Get in there, dig in, play around. If you don't like the results, empty the Kindle and start over. That's the great thing about calibre's library being off-limits - you can screw up everything outside of the library, but the library will always be there to save your a$$. (Just be sure to occasionally back up the library itself - its god-like power doesn't extend to hard drive failures and other catastrophes.)

[lestatar]

@emalvick, CWN: You guys are terrific. Thanks so much for the invaluable tips. Will def start poking around in the incredible calibre - I am sure I will be able to define a workflow/process that works best for me.

<with help from kind folks like you as well>

cheers!
-les

[CWatkinsNash]

Originally Posted by lestatar

@emalvick, CWN: You guys are terrific. Thanks so much for the invaluable tips. Will def start poking around in the incredible calibre - I am sure I will be able to define a workflow/process that works best for me.

<with help from kind folks like you as well>

cheers!
-les

You're welcome, good luck and have fun. Down in the calibre forums you can usually find me in the Plugin subforum. Have you tried plugins yet? It's like a party in your Kindle. Okay, maybe I'm the only one geeky enough to say that, but if you need help defining your workflow, I'm aaalllll about that. You know where to find me.

[j.p.s]

Originally Posted by jocampo

There is a min amount of fragmentation after a lot of writes but because there are no physical heads like regular hard drives, it is not as bad or noticeable as regular disks. The problem is that defragmenting itself affects flash memory lifespan

Now, that's my understanding, ... it would be nice if you have any article or link about it to take a further look ...

I do not recall where I have seen the futility of attempting to defragment flash memory discussed in detail, but it is easy enough to figure out from the basics. I glanced through the following links before my original post, but the former does not explicitly mention fragmentation and I was afraid the latter might seem to imply that any information in it is specific to USB.
http://en.wikipedia.org/wiki/Flash_memory
and
http://en.wikipedia.org/wiki/USB_flash_drive

Flash drives present an interface that appears to be a sequential mapping of storage locations, but not only are they not sequential, the the device frequently changes the physical block that a particular address points to. In addition, a flash memory block is much larger than a disk sector and probably much lager than a file system cluster. The remapping is used to implement wear leveling and the block size is inherent to the physical construction of the flash memory.

Wear leveling is needed because flash memory cells wear out after a number of writes, a smaller number than they would like you to believe. Whenever a file on flash is changed, the flash device copies the entire block containing the change to a never written block and remaps the address to point to that new block. When the device runs out of never written blocks, it has to erase an entire previously written but no longer used block, then write to that. To change even a single bit in a file, the flash needs to read the entire block that has that bit, change that bit, erase an unused block, write the entire changed block and remap it address. Now think what happens when a software application tries to defragment at the file system cluster level.

Fragmentation has no meaning for a flash drive, the block addressing is completely arbitrary. Trying to defragment a flash drive could easily cause every block to be erased and rewritten several times. That is just for the blocks storing the file contents. Just think about the blocks containing the FAT. Those get moved many times in a defragment attempt on a flash drive.

Trying to defragment flash not only does not make data access any faster, it also causes the drive to wear out much sooner.

Something to think about. Let's say you put 100 books on a new reader, a few at a time, then do not delete or add any books for months as you read the books that are there. Every time you switch to reading a different book or stop reading for a while or make a bookmark or highlight, your reader keeps track of your last page read, marks and notes. Those changes may not change the size of the files, so the file system thinks nothing has moved, yet the flash device has already moved blocks around every time the reader has changed your book state. Presumably, the programmers try to minimize the number of writes to flash, but they have to save your last page read sometime, probably when you or the device exits the book (to open another, or go to the bookshelf, or whatever. So your flash is getting "fragmented" even when the file system thinks it is not. And even when the file system thinks there is lots of fragmentation, there really is no difference in access time. But trying to defragment will increase wear and take a long time.

[lestatar]

Originally Posted by j.p.s

...SNIPPED...

Wear leveling is needed because flash memory cells wear out after a number of writes, a smaller number than they would like you to believe. Whenever a file on flash is changed, the flash device copies the entire block containing the change to a never written block and remaps the address to point to that new block. When the device runs out of never written blocks, it has to erase an entire previously written but no longer used block, then write to that. To change even a single bit in a file, the flash needs to read the entire block that has that bit, change that bit, erase an unused block, write the entire changed block and remap it address. Now think what happens when a software application tries to defragment at the file system cluster level.

Fragmentation has no meaning for a flash drive, the block addressing is completely arbitrary. Trying to defragment a flash drive could easily cause every block to be erased and rewritten several times. That is just for the blocks storing the file contents. Just think about the blocks containing the FAT. Those get moved many times in a defragment attempt on a flash drive.

Trying to defragment flash not only does not make data access any faster, it also causes the drive to wear out much sooner.>>>SNIPPED>>>>

^ This. Exactly right in my experience, both as professional geek manager and as decades long computer end-user.

Defragging flash drives is a waste of time and can indeed lead to premature failure as j.p.s. so eloquently points out.

I see this exact topic coming up on mp3 player forums as well and the answer is the same for that stuff too.

No benefit to defragging flash drives so why do it?