Calibre Folder Output

[PetrusV]

I have a total 7244 subfolders under my calibre folder.

I host my library on a shared host and have reached the folder limit.

Is there any way to group authors under alphabetic sequence?

Example: all authors with names that start with A to be under subfolder A, and authors that begin with B to be under subfolder B, etc??

If I can have calibre split the authors via subfolder load it would resolve the problem.

[ilovejedd]

Do you need the calibre file/folder structure itself on the shared host? Or would exporting via "Save to Disk" work as well? I expect it's not so much the author folders themselves that's the issue, rather it's the individual folders for each book.

For my save to disk, I use:
{tags}/{authors}/{series} {series_index} {title}

I use tags for genre (one per book). With a library of ~4,000 books, save to disk with the above template gives me ~900 folders versus ~5,000 folders in the native calibre structure.

[PetrusV]

What I actually do is I sync the entire library with nextcloud, making a exact replica.
I use cops to run the library on my server.
Currently I save author,title
I have over 20,000 books and because I save by author the problem is that I have 7244 authors and that exeeds the folder list. If I could group authors by the letters of the alphabet in subfolders I could resolve the problem.

[PeterT]

You can't change the structure of a calibre library. See Want to change folder structure or file names in the Calibre library?

[itimpi]

Originally Posted by PetrusV

What I actually do is I sync the entire library with nextcloud, making a exact replica.
I use cops to run the library on my server.
Currently I save author,title
I have over 20,000 books and because I save by author the problem is that I have 7244 authors and that exeeds the folder list. If I could group authors by the letters of the alphabet in subfolders I could resolve the problem.

You cannot change the internal structure of the Calibre library (there is a forum sticky about this).

What is suggested is that you use the Save to Folder option and provide to that a template that specifies the folder/filename structure that you want.

[PetrusV]

Yes I have read the sticky, I am not trying to change the internal structure, but rather trying to find a solution inside the structure.

If I make a custom column, can I make it save with the column information, then author and then title?
Example: Incan put a A next to all then authors that start with A, B next to all the authors that start with B.

[itimpi]

Originally Posted by PetrusV

Yes I have read the sticky, I am not trying to change the internal structure, but rather trying to find a solution inside the structure.

If I make a custom column, can I make it save with the column information, then author and then title?
Example: Incan put a A next to all then authors that start with A, B next to all the authors that start with B.

No - you cannot change anything within the existing Calibre library folder structure. All you can do is create a new folder structure that is external to the Calibre library using a feature such as the Save to Folder facility.

[ilovejedd]

Originally Posted by PetrusV

What I actually do is I sync the entire library with nextcloud, making a exact replica.
I use cops to run the library on my server.
Currently I save author,title
I have over 20,000 books and because I save by author the problem is that I have 7244 authors and that exeeds the folder list. If I could group authors by the letters of the alphabet in subfolders I could resolve the problem.

What exactly are the restrictions by the host? Is there a maximum number of all subfolders or is it just a limit per node?

For example:

This is allowed (25,005,000 subfolders total):

\parent
    |-- \child (x 5000)
        |-- \grandchild (x 5000 per child)

but this is not allowed (15,000 subfolders total):

\parent
    |-- \child (x 7500)
        |-- \grandchild (x1 each)

Also, I don't quite understand what you mean by you save author,title. Do you mean you already use the "save to disk" function with Author/Title as template?

Does COPS know (or can be configured) to use Author/Title as path/filename and not the normal Author/Title (id)/Title - Author that calibre uses? Because by my estimate, 20,000 titles with 7,244 authors would mean 27,244 folders with calibre's internal folder structure.

One way to get around the per node restriction is to divide your main calibre library into multiple libraries. For example, by genre or by authors (A-E, F-J, K-O, P-T, U-Z).

That said, if you're familiar with PHP scripting, I'm sure it's possible to modify the ebook url in COPS so you can use a custom folder structure. You then copy metadata.db and your save to disk folder to the host.

For what you want, the save to disk template can be:

{author_sort:'uppercase(shorten($,1,'',0))'||}/{title} - {authors}

or

{authors:'uppercase(shorten($,1,'',0))'||}/{title} - {authors}

[PetrusV]

There cannot be more than 7000 folders inside a folder. But there can be unlimited folders and subfolders as long as the first rule is not violated.

If save to disk was the solution I could have done the following:
{author_sort:.1}/{author}/{title} - {authors}
With the above statement 'Andrew Abernethy - The Beginning Of All Things' would have been saved as A/Andrew Abernethy - The Beginning Of All Things
However, I cannot use save to disk because I need the metadata.db (as I use COPS OPDS).

I DID manage to find a way to resolve the issue although it certainly is not pretty. I add an author for the letter of the Alphabet I wanted.
Example:
For all the authors with surname starting with A, I add first author 'A & '.
Daryl Aaron becomes A & Daryl Aaron
Andrew Abernethy becomes A & Andrew Abernethy
Kathy Abranovich becomes A & Kathy Abranovich
This way all these authors are stored under folder A, but their names remain in the database as well.

Like I said, not pretty, but it does work....

[BetterRed]

@PetrusV - maybe you should find a less restrictive host for the library, or pay the existing provider whatever it takes to increase the subfolder limit.

Partitioning your books over multiple physical libraries will impose irritating limitations on searching; you'll forever wonder why you didn't partition on something else - e.g. genre instead of author, or original publication date instead of publisher.

IMO the only 'legitimate' reason to use multiple physical libraries is the nature of the 'books'. FX I have separate libraries for regular books, periodicals, and audio/video. They have very disparate custom columns, and my (mis)use of standard columns varies, e.g. my periodicals library has journal/magazine name in author and edition identifier (issue, month, folio etc) in title.

BR

[ilovejedd]

Originally Posted by PetrusV

There cannot be more than 7000 folders inside a folder. But there can be unlimited folders and subfolders as long as the first rule is not violated.

If save to disk was the solution I could have done the following:
{author_sort:.1}/{author}/{title} - {authors}
With the above statement 'Andrew Abernethy - The Beginning Of All Things' would have been saved as A/Andrew Abernethy - The Beginning Of All Things
However, I cannot use save to disk because I need the metadata.db (as I use COPS OPDS).

I DID manage to find a way to resolve the issue although it certainly is not pretty. I add an author for the letter of the Alphabet I wanted.
Example:
For all the authors with surname starting with A, I add first author 'A & '.
Daryl Aaron becomes A & Daryl Aaron
Andrew Abernethy becomes A & Andrew Abernethy
Kathy Abranovich becomes A & Kathy Abranovich
This way all these authors are stored under folder A, but their names remain in the database as well.

Like I said, not pretty, but it does work....

Technically possible if you're willing to get your hands dirty. I remember using COPS before (around 2012?) and modifying the scripts so that the url follows the format /dl/epub/somecustomcolumn.epub

Had a quick look at COPS (latest master) and I think this might be what needs changing. Note, I don't have a PHP dev environment and it's been years since I last coded anything so no idea if this actually works.

lib\Book.php

line 116:        $this->path = Base::getDbDirectory() . $line->path;
line 117:        $this->relativePath = $line->path;

Assuming you have:

A/Andrew Abernethy/The Beginning Of All Things (1)/The Beginning Of All Things - Andrew Abernethy
K/Kathy Abranovich/Mupster Pupster (2)/Mupster Pupster - Kathy Abranovich

(basically no change to folder or file names, just group authors into folders by first letter)

lib\Book.php

line 116:        $this->path = Base::getDbDirectory() . strtoupper(substr($line->path,0)) . "/" . $line->path;
line 117:        $this->relativePath = strtoupper(substr($line->path,0)) . "/" . $line->path;

[PetrusV]

@BetterRed
Yea the thing is I have a TON of stuff with this host. If I move I must move another 25 domains as well. Everything is cheap, works extremely well and I have unlimited space. The only snag is my calibre library because I have over 20,000 books. (I actually have 6 different libraries split to their relevance already) And yes, they are all related to the library they are in. So splitting this one is not an option

@ilovejedd
Unfortunately the problem is not COPS, but the CentOS server I am on, it limits folders to 7000 per folder. I cannot just use the 'copy to disk; option either, because COPS OPDS needs the metadata.db from calibre as well...

[ilovejedd]

Originally Posted by PetrusV

@ilovejedd
Unfortunately the problem is not COPS, but the CentOS server I am on, it limits folders to 7000 per folder. I cannot just use the 'copy to disk; option either, because COPS OPDS needs the metadata.db from calibre as well...

That's what I'm saying. You can edit the COPS PHP script (I believe lib/Book.php) so it looks for books in a custom folder structure of your choosing.

Granted, I don't recommend changing the path and filename significantly since the more it diverges from calibre's folder native structure, the more involved the changes needed in the COPS PHP scripts.

With the changes to /lib/Book.php in post #11, I wouldn't even bother with save to disk. Just use a shell script to copy the author folders by first letter.

So your calibre library folder on the CentOS server would look like this:

/A
/B
...
/Z
metadata.db

[ilovejedd]

Whoops, looks like that code should have been:

lib\Book.php

line 116:        $this->path = Base::getDbDirectory() . strtoupper(substr($line->path,0,1)) . "/" . $line->path;
line 117:        $this->relativePath = strtoupper(substr($line->path,0,1)) . "/" . $line->path;

[PetrusV]

@ilovejedd That sounds very interesting... I am definitely going to try that as soon as I get s chance!! Thanks for the great advice!!