Directory mixup

[cns1000]

This is still regarding my progress setting up CC on my Alcatel A30 SD card.

I have been transferring files to the device via USB. Before starting the USB transfers, I had directed CC to use the \CC directory on my SD card, which it has been doing. It sees only the files in that directory, which is its default.

Calibre sees the device and writes to Card A, which is correct; however, it is writing to a different directory, called Books. I had assumed it would find the CC directory which was set up on the SD card, but it does not. Or I should say, it sees it but does not write to it.

So Calibre sees the books in \Books and in \CC (which are both subdirectories of the root), but it writes only to \Books. CC sees only the books in \CC.

I have tried changing CC's default folder from \CC to \Books but it does not seem to want to let me do that. I can select the directory but then I just get a blank black screen. Or occasionally an hourglass with the message "Couldn't query documents."

I could wipe everything out, reinstall CC and start over but that would be a pain. Alternatively, I could uninstall CC, transfer all the files manually from \CC to \Books, and then reinstall and select the \Books folder. Presumably Calibre would still see the files, since it sees both directories.

Alternatively, I could move all the files manually from \Books to \CC, but would CC then pick them up ? And that still wouldn't resolve the question of how to tell Calibre which directory to write to in future.

Any advice as to what approach is recommended would be appreciated.

thanks
Charles

[theducks]

CC has its own config. Calibre does not look for 3rd party settings ( the original CC developer did work on the Calibre Wireless device side, but that remains generic and not CC specific.)
The Calibre Apple or Android driver puts books where its config says to (within the device allowed rules). See if you can make those line up.

EXCEPT IIRC You must use CC to add or CC's DB does not update. DO NOT MIX the 2 methods

[cns1000]

Thank you for the suggestions.

Progress at the moment is as follows: I was wrong in thinking that CC was failing when I asked it to change the default directory. It was working but just very very slow. It took about half an hour before the progress screen with the little Android moving books around came up.

There are about 3800 books at the moment in the directory it is moving and over 9000 in the directory it is moving to. Based on numbers in the directories, it seems to be processing about ten books per hour, which would mean the process would run for over two weeks.

Question: if I were to stop the process, copy everything manually to the target directory (which is the one Calibre seems to want to use) and then uninstall and reinstall CC and select that directory, would it pick up all those books and rebuild the database ? Or would I just be left with a mess ? Calibe would probably be ok I guess since it sees both directories, but what about CC ?

Thanks in advance for any ideas.

[ottischwenk]

Originally Posted by cns1000

There are about 3800 books at the moment in the directory it is moving and over 9000 in the directory it is moving to. Based on numbers in the directories, it seems to be processing about ten books per hour, which would mean the process would run for over two weeks.

Something is not working correctly: with a transfer of around 14,000 books, CC, regardless of which of my devices, needs a little more than 2 hours, which means around 100 / minute

[chaley]

You might be encountering the FAT32 "number of files in a folder" performance problem. This CC FAQ might provide some help: [Android] I have added a lot of books and now adding books takes forever. What do I do?.

As for manually copying the files: (assuming the current devs haven't change this) yes, you can do it. The steps:

1. Copy the directory structure exactly as it is. However, if you are running into the FAT32 performance problem then this operation might also be very slow.
2. Delete or rename the source directory so CC won't try to move the files in the next step.
3. In CC Settings / Formats and folders, point CC to the new directory.

This works because for each book CC keeps the path from the parent directory to the book in its database. The path to the parent is kept elsewhere.

[cns1000]

That is really helpful; thank you for the replies !

Generally, when I have been copying, I find the fastest approach is to connect via USB and use Windows Explorer to move the files. I have also used Total Commander on the Android which seems to work reasonably well also. If I am moving about 3500 I should not have a problem (I hope). Will let you know if there are further difficulties. I presume I am correct then in saying that Calibre will not have a problem with the change ?

[chaley]

Originally Posted by cns1000

I presume I am correct then in saying that Calibre will not have a problem with the change ?

It shouldn't if you change the calibre device plugin's template to 'point' at the new location. But do note that you can't use CC and calibre (cabled) to update the books. CC won't see the books that calibre put there unless CC is scanning the folder on connect (slow), the books are epub, and the book's embedded metadata exactly matches what is in the calibre library.

[cns1000]

Worked, only partially (I LOVE the Android file system !)

I decided it might not be a great idea to move files on the Android using Wndows Explorer; I have found it works well moving from computer to Android and vice versa, but maybe not so much between Android directories. So I decided on Total Commander. This worked ok; it looked like it would take about two hours to move the 3500 or so files, which is better than two weeks.

But the move stopped for no apparent reason and with no error message with 697 files left to transfer and 12648 in the target directory. Tried restarting, as well as restarting the Android; still no go.

Renamed the source directory though there were still files left in it and restarted CC, and it seems to see the new directory ok (maybe it remembers the folder reset operation I interrupted), but it has not updated itself to see the new files. Which I can see because it has not updated the file counts or tag counts in its database.

Tried moving the remaining files by using Windows Explorer and USB to first copy the source files and then copy back to the target directory. The copy worked ok but copy to the target will not; can see the directory and the 12548 files in it, but on trying to copy to it, I get a "device has stopped communicating" message. Calibre seems ok, though at the moment it is still updating its file list.

So maybe there is some directory size limit there that I have hit. Let me know if there are workarounds (have not had time yet to check the link you sent, but I will do so. Will be away for the next couple of days so I do not have more time to play with this now).

Questions:

what is the practical limit on number of books for CC ?

is there any way of getting it to recognize subdirectories, to avoid this problem with large directories ? Other than, which I suppose is an option, storing different formats in different folders ?

Suggestions would be much appreciated,

Thanks
Charles

[chaley]

Originally Posted by cns1000

But the move stopped for no apparent reason and with no error message with 697 files left to transfer and 12648 in the target directory. Tried restarting, as well as restarting the Android; still no go.

It seems you have hit the FAT32 number of files limit. The max number of files depends on the average length of the file name. For example, if the average length is in the 65 character range then the max number of files will be around what you see. It will also be *painfully slow*.

Renamed the source directory though there were still files left in it and restarted CC, and it seems to see the new directory ok (maybe it remembers the folder reset operation I interrupted), but it has not updated itself to see the new files. Which I can see because it has not updated the file counts or tag counts in its database.

Were all those books in the CC database before you tried the move? CC doesn't look for new files, except for epubs when you use the wireless connection.

So maybe there is some directory size limit there that I have hit. Let me know if there are workarounds (have not had time yet to check the link you sent, but I will do so. Will be away for the next couple of days so I do not have more time to play with this now).

Yes, there is a limit. Yes, you appear to have hit it. And yes, there are workarounds. See the link.

Questions:

what is the practical limit on number of books for CC ?

I have seen as many as 30,000. 10,000 to 15,000 was relatively common. There is no built-in limit, but eventually CC will run out of memory and/or performance will be terrible.

is there any way of getting it to recognize subdirectories, to avoid this problem with large directories ? Other than, which I suppose is an option, storing different formats in different folders ?

See the link I sent earlier.

[cns1000]

Yes, I guess it looks like I have run out of directory space.

As far as the question of which books are recognized: the ones in \CC were transferred wirelessly and were recognized by both Calibre and CC. The ones in \Books were transferred via USB (MTP) and were recognized only by Calibre; CC never saw them and still doesn't, though it now sees that directory. Question: If I deleted them and transferred them again via USB, would CC see them ?

If not, it looks like one way or another it might be best to wipe everything and start from scratch, either by using the templates to reduce directory size, the allocation of different formats to different directories, or something of the sort.

Here is a crazy suggestion; maybe you could tell me if there is possiblity it might work: there is a driver available from Paradox Software which supposedly allows you to connect an external USB drive that is formatted with NTFS and will supposedly work with standard Android software and file managers. I have a Note 9 which supposedly could support a USB-C hub (though they seem to be hard to find for Android) and could conceivably connect an external NTFS drive via one of its USB ports. Would CC recognize that ?

Otherwise I am wondering whether in the long run it might be better just to go with Calibre and install it on some compact Windows-based (or maybe Linux-based) tablet that could support a large number of files. Though that would be a much more expensive solution. Or to use CC on my current tablet only selectively.

Any suggestions would be appreciated.

Thanks
Charles