Getting Calibre to transfer books to device creating subdirectories by author

[DMcCunney]

I'm probably missing the obvious again, but...

When I first began to use Calibre to manage my eBook library and I added books, Calibre created directories under it's main Calibre Library folder as needed, with the author's name as directory name, and further books by that author went into that sub-directory. When I used Calibre to transfer books to my Android tablet, it retained that structure.

On my Android tablet, eBooks are stored in the /eBooks directory on an external 32GB microSD card. Calibre knows to place books transferred to device there, and FBReader knows that's the directory eBooks live in.

Recently I discovered that new books I add to Calibre aren't being placed in existing sub-directories, and new directories aren't being created. All books are winding up in the top level eBooks directory.

I have made no changes I'm aware of in Calibre's config that might affect this, so I'm confused as to why it's occurring. I looked at the Calibre docs, and unless I am misunderstanding them, the default action Calibre takes when transferring books to device should replicate the directory structure Calibre maintains on the desktop.

I'm trying to debug a recent problem with FBReader Premium on the tablet. When it scans for new books, it never successfully completes - a background library scan process continues endlessly, and chews up enough CPU to make reading the current book problematic.

FBReader's author has been sent a copy of the books.db file where FBReader stores its list, and I await commentary. This is not the first time I've had this sort of problem. I don't know if I have the largest library on device that FBReader is called upon to deal with, but I suspect I'm well up on the shortlist. (I am a pack rat who likes carrying a large library around with me.) FBReader's author says there isn't an arbitrary limit in library size. Not in his code, perhaps, but I suspect third-party code he uses might be choking.

Meanwhile, when thousands of eBooks are winding up in the top level directory when they all should be in sub-directories, my spider sense tingles and I wonder if it's a contributing factor.

What should the format string Calibre uses to determine how to transfer books to device be to make it create directories and store books in them? I want to make sure that's what it's using, and eliminate one variable.

Thanks in advance,
______
Dennis

[FizzyWater]

I'm not at home, so I might not be getting the details right - I get the results you're talking about when I use the "export to a single folder" option. (I'm still using a 2.x version of Calibre).

[DMcCunney]

Originally Posted by FizzyWater

I'm not at home, so I might not be getting the details right - I get the results you're talking about when I use the "export to a single folder" option. (I'm still using a 2.x version of Calibre).

Thank you - this is something I can look at.

I'm running the current 64 bit version under Win10 Pro. I'll have to check that option. It's possible I changed the default as part of other fiddles, and didn't realize the implications.

Meanwhile, I popped the card out of the device, put it in an adapter, and went in and removed the thousands of books that had wound up on top level, then replaced the card and booted the device. FBReader Premium is currently rebuilding it's metadata from only the volumes stored in sub-directories. If it successfully completes, I'll have narrowed down the problem.

Once I have Calibre configured to do what I expect, I can re-add the volumes I deleted from the card. It will take a while, but that's okay as long as it works. (The main reason I have the tablet is eBook viewer, so anything that impedes that is a major problem.)
______
Dennis

[BetterRed]

Originally Posted by DMcCunney

Recently I discovered that new books I add to Calibre aren't being placed in existing sub-directories, and new directories aren't being created. All books are winding up in the top level eBooks directory.

When you send a book to a device or save it to disk the output directory and file names are specified by a template in the relevant preference settings, viz:



It would appear that you've lost some of the template you had - make sure you check the device configuration template, it overrides the one in Send To Device preferences.

Originally Posted by DMcCunney

I looked at the Calibre docs, and unless I am misunderstanding them, the default action Calibre takes when transferring books to device should replicate the directory structure Calibre maintains on the desktop.

You are misunderstanding doco. Here's why:

To cope with the vagaries of the different operating and file systems that calibre supports, the author and book directories, and the format file names within the library are ascii-ized and they may be truncated. These names are computed algorithmically - and the algorithm is subject to change.

For Latin alphabets, diacritics such as acute and umlauts etc are discarded, other 'alphabets' are 'latin-ized', so the author iʁɛn nemiʁɔfski directory name will be iRen nemiRofski and the book 辛識平：習近平向世界展示“中國名片” directory name will be Pi Lu Ma Jia , Gan Hei Gou Dang _We.

It's probably 'safer' to test a template with Save to Disk.

BR

[DMcCunney]

Originally Posted by BetterRed

When you send a book to a device or save it to disk the output directory and file names are specified by a template in the relevant preference settings, viz:

Attachment 157758

It would appear that you've lost some of the template you had - make sure you check the device configuration template, it overrides the one in Send To Device preferences.

I just looked in Calibre, and what you posted is what I see.

You are misunderstanding doco. Here's why:

To cope with the vagaries of the different operating and file systems that calibre supports, the author and book directories, and the format file names within the library are ascii-ized and they may be truncated. These names are computed algorithmically - and the algorithm is subject to change.

That's fine, and I don't care.

I've dealt with a number of different architectures and OSes over the years, and am aware of the issues. Calibre is welcome to algorithmically change things. All I care about is that the library structure on device replicates the one in the main Calibre Library on the desktop, and that sub-directories are created based on author names and books by authors go in those sub-directories.

For Latin alphabets, diacritics such as acute and umlauts etc are discarded, other 'alphabets' are 'latin-ized', so the author iʁɛn nemiʁɔfski directory name will be iRen nemiRofski and the book 辛識平：習近平向世界展示“中國名片” directory name will be Pi Lu Ma Jia , Gan Hei Gou Dang _We.

Fine by me. I have very little stuff not in Latin alphabets, and the stuff that is isn't something I'll view on the tablet. (Among other things, I don't transfer PDFs - they are poor fits to 7" device screens. I get ePub by preference, but can handle Mobi as well because FBReader can display an assortment of formats.) Diacritics like umlauts and acute can go away in directory and title names. I don't need them to know what the book is to select it. They will be displayed in the book itself.

It's probably 'safer' to test a template with Save to Disk.

A good thought, and I likely will, but the considerations you mentioned aren't part of the problem I'm trying to solve.
______
Dennis

[BetterRed]

I can't help with FB Reader specific issues - don't use it, never seen it.

I had the impression that you didn't like having everything ending up \ebooks rather than in author (or author_sort) directories, that's what I was trying to help with.

BR

[DMcCunney]

Originally Posted by BetterRed

I can't help with FB Reader specific issues - don't use it, never seen it.

I began using FBReader in a C language version that ran on Windows and Linux. Like the tablet version, it handled ePub, Mobi, and FB2 among other things "native", so I mostly didn't care what format a book was in. A special plus for me was that it handled books in the format used by the Plucker offline HTML viewer for PalmOS device. I have about 4,000 files I converted from HTML to Plucker to read on my Palm, and it was nice to be able to read them on something besides my PDA. It didn't (and doesn't) handle DRM, but I don't get stuff encumbered with DRM and don't care.

The Android version is a port to Java, and has a similar feature set. (It doesn't handle Plucker files, alas.) And it can handle PDF, DjVu, and CBR/CBZ files via plugins. (The Premium version builds in PDF support.) It's highly configurable and works well, but I've seen it have problems with large libraries. (The author has posted several versions intended to fix those issues, but they still bite me.)

I had the impression that you didn't like having everything ending up \ebooks rather than in author (or author_sort) directories, that's what I was trying to help with.

That is exactly my problem. But your observations, while accurate, aren't part of the issue I'm having. I just want Calibre to replicate the directory structure on the card in the tablet that it used in the main library. ASCIIfication is fine by me, and not a factor in what I'm seeing.

I think FizzyWater is on to something, and it's what I'll be looking at next.
______
Dennis

[davidfor]

@DMcCunney: The question is how are you putting the books on the device? If you are using send-to-device, then the template BR pointed to controls it. But, that is the default template. When you configure the device driver in calibre, it copies the template from the default. Any future changes have to be done in the driver configuration. You need to check this is what you expect it to be. The simplest way to do this is when the device is connected. Right-click on the device button in the toolbar and select the configure option. You will then be able to see what template is currently being used.

If you are using some other mechanism to get the books to the device, then you need to find and fix that configuration. The ways I can think of are save-to-disk and Calibre Companion. There is template in the save-to-disk configuration that can be changed. I don't use Calibre Companion and don't know if this is configurable.

[DMcCunney]

Originally Posted by DMcCunney

That is exactly my problem. But your observations, while accurate, aren't part of the issue I'm having. I just want Calibre to replicate the directory structure on the card in the tablet that it used in the main library. ASCIIfication is fine by me, and not a factor in what I'm seeing.

I think FizzyWater is on to something, and it's what I'll be looking at next.

And FizzyWater was onto something. The string in the default template was correct. But it can be overridden by a specific configuration for the target device, and that was wrong. Editing the device config to use the same string as the default template cured the problem, and eBooks are being properly placed in sub-directories on device.

I vaguely recall diddling the device config which is where the error crept in, but I don't recall why I made the change. In any case, things are working as desired.

And once I removed a large number of books that had would up in the top level directory instead of in the proper sub-directories, FBReader stopped choking trying to scan for new books and update it's metadata. I haven't heard back from FBReader's author on my bug report and submission of the books.db file it creates. I need to send him an update on what I did. It may point him in the right direction.
______
Dennis

[theducks]

The template in Preferences is a DEFAULT for NEW devices. Once a device has been connected, it retains the, then current default unless it (the device driver version) is later modified by the user.
The HAVE BEEN USED device drivers are stored in the Calibre configuration folder. You could delete the 'bad' one, restart Calibre, then connect the device. It will grab the current settings template

[BetterRed]

Originally Posted by DMcCunney

And FizzyWater was onto something. The string in the default template was correct. But it can be overridden by a specific configuration for the target device, and that was wrong.

Which is what I suggested it would be in post #4

Originally Posted by BetterRed

[. . .]
It would appear that you've lost some of the template you had - make sure you check the device configuration template, it overrides the one in Send To Device preferences.
[. . .]

BR