Any Way to Subdivide the Main Directory?

[johnelle]

My library has quite a few books. I am now hitting file system limits on the main [Calibre] directory that are causing various instabilities that my intuition (and support) tells me is going to end in disaster if I continue.

Is there some easy way to subdivide the directory by adding another level such as [Calibre\A Authors] [Calibre\B Authors] etc. ?

I would like to keep everything within a single view if that's possible. Otherwise I am going to have to figure out a way to divide it manually into separate libraries.

[JSWolf]

What OS are you using and how many eBooks do you have?

[ownedbycats]

Unless you have upwards of 65,000 authors in the main directory, I wouldn't worry about it. If you do: What the <expletive deleted> are you even doing??

The limit of maximum files in a directory is 65,534 on FAT32, and on modern filesystems such as NTFS it's in the millions.

[johnelle]

Well folks I was purposely trying to stay away from those questions to avoid controversy and focus on the question at hand. That said there are currently 94829 authors. The OS is linux running on a Synology. Yes the files are sitting on a very very high quality NAS--Synology quality is MUCH better than Microsquish and yes I have been lectured about the evils of NAS and I won't change my mind---my files are MUCH safer on my NAS thank you.

So...back to the question. Is there a way to subdivide the main directory other than starting a new library?

[PeterT]

Originally Posted by johnelle

So...back to the question. Is there a way to subdivide the main directory other than starting a new library?

Simply put, NO

[pazos]

Originally Posted by johnelle

... there are currently 94829 authors. The OS is linux running on a Synology. Is there a way to subdivide the main directory other than starting a new library?

Before doing experiments check that your fs (assuming ext4) has free inodes. Nobody cares about inodes because they're usually cheap, but in your case it might not be.

Ext4 has no limits on files per directory and has a limit of files of 4,294,967,295, but the number of inodes available is usually far less than that.

Asumming your NAS is only holding a calibre library with 94829 authors, and each author has 50 books you're spending 4,836,279 inodes just on directories (For the record I have a 3TB partition almost full of stuff with only 1,394,869 inodes used)

Now you need to add 3 inodes more for each book (for cover.jpg, metadata.opf and the book format of your book) which add a grand total of 14,224,350 inodes more. And that is only with one format per book.

Probably you're far from reaching 100% of inode usage, but typing

df -iT

on your terminal is a fun experiment anyways

If you have plenty of free inodes, then is a matter of usability issues (ls can take time to complete, expanding wildcards won't be possible) but you'll never hit filesystem limitations because there're no filesystem limits other than inodes.

Have fun.

[phossler]

Question to the more experienced members --

Could the OP

1. Copy the entire main folder (..\CALIBRE') to ..\CALIBRE_A-F\ using the O/S

2. Delete all folders in ..\CALIBRE_A-F\ with authors G through Z using the O/S

3. Switch/Create Library and point it at ..\CALIBRE_A-F\ using Calibre

4. Library Maintenance/Check Library and then delete the 'missing' books since that library's metadata.db (copy of original) has the entry but the book is not in the ..\CALIBRE_A-F\ folder

5. Repeat for next chunk of authors



Upside is that if it goes all wahoonie-shaped, the original is still there

Downside is have to make the correct 'sub-library' the active one

[capink]

Originally Posted by johnelle

My library has quite a few books. I am now hitting file system limits on the main [Calibre] directory that are causing various instabilities that my intuition (and support) tells me is going to end in disaster if I continue.

Is there some easy way to subdivide the directory by adding another level such as [Calibre\A Authors] [Calibre\B Authors] etc. ?

I would like to keep everything within a single view if that's possible. Otherwise I am going to have to figure out a way to divide it manually into separate libraries.

I don't know what is the cause of your problems. The best approach is to try to solve them at the filesystem level.

If that fails you want to what you asked for above, you can create a new custom column for authors, use the search and replace dialog to copy all your authors to that custom column, and use the standard authors column for partitioning. e.g for all books that first author letter is a, you put an a in the standard authors list, you can do this manually or use regular expression search and replace.

[pdurrant]

Originally Posted by capink

If that fails you want to what you asked for above, you can create a new custom column for authors, use the search and replace dialog to copy all your authors to that custom column, and use the standard authors column for partitioning. e.g for all books that first author letter is a, you put an a in the standard authors list, you can do this manually or use regular expression search and replace.

That is an ingenious work-around. I'm impressed. And it should reduce the number of directories required by 90,000 or so!

Of course, each directory will then have a LOT more entries in it. Might that cause a separate problem?

[johnelle]

Thanks for the suggestions. Mostly likely to start with the inodes after the current backup completes.

The good news is that Synology has amazing support and they were on the system yesterday for a while and they are coming back after the backup completes and I reboot.

[mbovenka]

Originally Posted by pazos

Ext4 has no limits on files per directory and has a limit of files of 4,294,967,295, but the number of inodes available is usually far less than that.

Even so, I doubt it'll be a problem. My Synology, with a 6TB EXT4 partition with besides two large Calibre libraries lots of other stuff on it (about 4.5TB worth) uses 1.7M of 182M inodes.

[capink]

Originally Posted by pdurrant

That is an ingenious work-around. I'm impressed. And it should reduce the number of directories required by 90,000 or so!

Of course, each directory will then have a LOT more entries in it. Might that cause a separate problem?

One can have control over this by changing the partitioning scheme. e.g. aa, ab, ac, ... etc. The search and replace tool can help in automating the process. Still, it is better that the @johnelle is seeking other solutions first, as this might not be the ideal solution as you lose a lot of functionality like downloading author metadata.

[BetterRed]

Originally Posted by capink

…as you lose a lot of functionality like downloading author metadata.

Eh, author metadata From where can calibre download it and where does it put it?

I ask because I'm in the process of building an 'Authors' library that links to/from my Books, Journals and Media libraries via the new calibre:\\\ URL feature.

BR

[capink]

Originally Posted by BetterRed

Eh, author metadata From where can calibre download it and where does it put it?

I ask because I'm in the process of building an 'Authors' library that links to/from my Books, Journals and Media libraries via the new calibre:\\\ URL feature.

BR

Sorry, I misspoke . I meant the data that goes into the author field when downloading book metadata. If someone re-purposes the author column he will have to uncheck the author field in the download book metadata dialog.

[BetterRed]

Originally Posted by capink

Sorry, I misspoke . I meant the data that goes into the author field when downloading book metadata. If someone re-purposes the author column he will have to uncheck the author field in the download book metadata dialog.

Phew - I thought I may have missed something

BR