REQ: input on Database Structure and testing a workaround input

[jecilop]

UPDATE: Completely SOLVED as of post 16.

This question is for those who know about the folder/file structure of book entries and their relationship to database integrity. I'm trying something and would like to know if there is some impact unknown to me.

I've found a way to use Calibre to manage my hardcopy audiobooks and not just as a reference entry (and I love it). I know the various ways of getting files into Calibre's management. I know the author/title/filename structure it creates on all those methods.

My challenge is that these methods take a LONG time given the file size of audiobooks and zip files in this case. Copying such large files takes a lot of time and makes it inefficient. Calibre works by copying and creating a folder structure or copying and renaming in to the standard filename structure. Again, in this case it takes a long time per book.

Is there some unforseen impact of having files in the contain folders that do not follow the Calibre naming structure yet still show Calibre and manage as normal? I did some various tests learning some interesting things. I'll not go into every detail. The end result is that I tried 2 basic methods of "moving" files rather than letting the computer work to copy which is, of course, much less time when on the same hard drive.

1) I went directly to the Calibre book storage folder, moved the file into it, and renamed the file to match the Calibre structure of the existing epub copy.
Name - Author.epub
Name - Author.rar (my moved and renamed file - NOT imported)
2) I did as in #1 but DID NOT RENAME the file. From other testing experience, I didn't think it would read the file....

HOWEVER, in order to get Calibre to read the presence of these files which it normally would not do, I did a database rebuild. THIS WORKED!!!
The moved files were read as entries in the database and formats showed up as well as resulting preference rules activated base on those formats. A library check showed NO problems in the database or further file check.
The only 2 odd things were these:

1) The files that I renamed all created thumbnail files that showed up as DB entires in the Calibre formats even if the files were .rar files. These DB formats are easily deleted in a batch.
2) The file that I moved into an existing entry folder and DID NOT RENAME was the same rar file. It reads like anything else, not DB file was created, and the file name remained as the original. THIS IS THE SURPRISE. I thought Calibre couldn't read a file that doesn't follow the standard naming structure.

THIS IS WHERE I NEED INPUT from those in the know. I have restarted and had no issues in this test library. I open the containing folder and see the file still there with the original name. It reads like anything else. A library check shows no problems.

I want to use this last method of simply moving files, not worrying about renaming, then rebuilding the database. The copy speed is not longer a factor then. I'm aware of ensuring data prefs backups.

Is there some unforseen impact of having files in the contain folders that do not follow the Calibre naming structure yet still show Calibre and manage as normal?

Thanks.

[BetterRed]

@jecilop - an easier way for single books, and with Windows and NTFS at least, is to add and rename the BIG file to the book folder as you're doing, then open the book folder (press 'O' key) and drag 'n' drop the BIG file into the Book Details panel (right sidebar).

I just tested it with a 2.5GB file, it was 'registered' immediately, so Windows and/or calibre must have worked out that there was no need to do any copying.

I use the 'trick' in a different context every day. I know its breaks the 'calibre libraries are black boxes' rule, and if it stopped working I'd have consider my options. But, there's usually another way to skin a cat.

I run Check Library on my libraries weekly - very rare that anything is reported, usually something I inadvertently left behind (eg an unpacked CBZ), and I have never needed to use Restore Database in anger.

IMO 'copying the files and then rebuilding the database' is pushing the envelope, and on large libraries I believe rebuilding the database can take a quite a long time.

If you're on Windows you could drop a shortcut (.LNK File) to the BIG file into the Book Details panel and leave the BIG file where it is. LNK files are less than 4K.

BR

[jecilop]

@BetterRed,

That is a very interesting trick in number one. I tried it and find it easily done. However, while I have a lot to add to my existing Calibre entries, I'm going to go the rebuild route then simplydo the rename after (with your trick). That sounds more complicated than it is. I just played with it and found it faster. Your trick will allow me to do the last #2 but then cause the file to be renamed according to the cannon file structure. :-)

That's a good point on large databases. Thankfully, this isn't that large which is why I feel good about giving it a go this way.

I never thought of the link/shortcut files. That's a great idea too. In my situation, I'll proceed with adding the actual files to the audiobook library folders as I keep it on an external drive. Once I get these existing ones in place, I won't have to take any shortcuts for any future additions. There are just enough to do upfront that makes this auto-find and edit a far easier task. They will be a simple add to my primary library in the future(where all entries exist), then copy to the audiobooks - all of this the "traditional" way.

I really appreciate your ideas.

I am still curious about the effect of "non-standard" naming in the folder structure. I know you can put whatever you want; it won't be lost including multiple files of the same type. However, if you run a check it shows these as extra files.

[BetterRed]

@jecilop - Windows will detect if a LNK file target is moved within a domain and repair the LNK file, read up on the Distributed Link Target Service. Don't be put-off by the reference to 'network domains' etc, it works just the same in a single-user/single-computer context as it does in a corporate context.

Re the books with the 'extra files', I'm not exactly sure what you are saying, some screen shots would help. But if you change the Author or Title or you move the book to another library I suspect you will lose the extra files - I suggest you try doing that before you proceed.

BR

[kovidgoyal]

Do not put extra files in calibre library folders. Sooner or later it will bite you and you will lose those files. You can always add a single lnk file via calibre's add files mechanisms, to do what BR is suggesting.

[fidvo]

I ran into a similar problem when building my media library with large movie files, that would take 30 seconds to a minute per movie to add to Calibre. I found a way to move the files instead of copying them. The trick is to use a placeholder file.

WARNING! (Standard disclaimer about modifying files in your Calibre library. If you don't know what you're doing or can't trust yourself to perform these steps exactly, DON"T TRY THIS!)

First, I created a small text file on my hard drive (not in the library path). Then, for each movie file, I performed the following steps:

1. Copy the name of the real movie file and overwrite the name of the text file. IMPORTANT: Include the file extension.

2. Add the renamed text file to Calibre. This should be almost instantaneous because you're only copying a few bytes. Note that, at least for formats Calibre doesn't recognize, it doesn't care whether the file is valid. It won't nag you about the file not being a valid .mp4 file or whatever.

3. Open the containing folder for this "dummy" movie file.

4. Copy the name of the file and paste to overwrite the file name of the real movie file.

5. Drag and drop the real movie file into the library folder, overwriting the dummy file. This should also be instantaneous because you're moving instead of copying the file. Important Note: If it doesn't ask you if you want to overwrite the existing file, stop what you're doing and figure out what went wrong RIGHT NOW. Move the real movie file back out of the library and repeat steps 4 and 5.

If you're careful, it should work fine. The only drawback of this method is that the file size is reported wrong in Calibre, and I haven't found a way to fix it. If an inaccurate file size bothers you, don't use this method.

[BetterRed]

@fidvo - re point 5 - "This should also be instantaneous because you're moving instead of copying the file."

What if the movie file (ie the real MP4) is on a different drive to the library drive - e.g on a server drive and your library is on a local drive? Would it still be instantaneous?

BR

[PeterT]

Also only instantaneous if the movie file is on the same drive as the calibre library, otherwise its a copy.

[fidvo]

Originally Posted by BetterRed

What if the movie file (ie the real MP4) is on a different drive to the library drive - e.g on a server drive and your library is on a local drive? Would it still be instantaneous?

BR

Originally Posted by PeterT

Also only instantaneous if the movie file is on the same drive as the calibre library, otherwise its a copy.

Fair enough. My method only works if the files are already on the same drive. Since the OP was talking about moving files directly into the Calibre library folder to speed up the process, I assumed he wasn't copying them from a different drive.

[eschwartz]

Well, it looks like calibre doesn't optimize the copyfileobj buffer size for large files, which would help a little at least.

Although copying GBs at a time is not going to be pleasant, so I understand why you would rather move them.


...


calibre "supports" files that aren't named "correctly" -- the database holds a reference to the final pathname, since sometimes calibre shortens the filename to work around issues with various os/fs combos.
So a database restore should pick up each uniquely-suffixed file, and add them properly. As you've seen.

As Kovid says you are relying on undocumented trickery which might not be the best way of dealing with calibre. Also, metadata changes in calibre will recalculate the filenames and move them -- so in case anyone thinks this is a way to get around calibre's library structure, think again!
But it *should* be all right. Given that there is already a limited flexibility in the structure, it should serve to add the book at least. I cannot promise what parts of calibre might depend on a heuristic view of the filepath though. Restore/integrity-checking might complain, for instance -- and certainly will if you try pushing it any further!
And it really does strike me as a pretty slow way of dealing with the issue, as a database restore is not exactly fast itself.

[jecilop]

Originally Posted by BetterRed

@jecilop - Windows will detect if a LNK file target is moved within a domain and repair the LNK file, read up on the Distributed Link Target Service. Don't be put-off by the reference to 'network domains' etc, it works just the same in a single-user/single-computer context as it does in a corporate context.

Re the books with the 'extra files', I'm not exactly sure what you are saying, some screen shots would help. But if you change the Author or Title or you move the book to another library I suspect you will lose the extra files - I suggest you try doing that before you proceed.

BR

@BetterRed,
I've been working with my files and was happy with results at first. I understand the dangers of moving something and losing things that Calibre cannot manage if I did that. I had a very unique way of using a particular library so I could avoid loss issues. HOWEVER, the more I was reorganizing, the more I found the shortcut links (LNK) to be a preferable method. HOWEVER.... IF I move the target location, Wdws 7 no longer offers the option to find the new target. I don't have to do this much, but if I reclassify books such that their audibook storage location must change, I'd prefer to move the target and not have to worry about updating the link. I read up on Distributed Link Target Service. This sounds just like the way Windows use to work with shortcuts.

However, I've notice that Wdws 7 doesn't offer the option to find the new target anymore - only to delete the shortcut. As I read further and more posts, it seems they did away with this after XP. So I'm a little lost now I ran tests again, but nothing is automatic nor does it give the option to find the new target. I have to manually edit the link or do a new one - either of which is tedious.
Do you know if I'm missing the obvious here? I used to move targets and have no problems, but it has been a long while such that I may not have noticed the change when Vista hit. (Vista..grrrrr...)

I've found some simple software that is supposed to be able to scan all links and tell you which are broken and lead to a batch fix, but I have yet to actually try it as I don't have a lot YET.

I have the service for DLTS running (trkwrks) so I didn't tweak Wdws 7 and accidentally disable it.

I have 2 entry categories that I prefer be in separate storage folders rather than 1. While 1 would solve the broken link issue while always on the same computer, it will cause other issues. Also, even if I did use only 1 storage folder, if I use the portable storage and end up with another drive letter, that will kill all of the shortcut links since it is NOT automatically redirecting.

Any more thoughts you are willing to share? I went back and prepped most things for LNK files use, but now I fear I've dug a hole.

Here is the simple breakdown of what I would use:

Portable F: Folder A Subfolders X and Y
Target files may get moved between the two at some point. I would want the link stored in the Calibre library to find the target if it gets moved from X to y.
Also, should the Portable F: register as G: on another computer which is likely, I'm hoping there is a smart redirect for that too.

From your post, it sounds like this should be the case. I know such redirection worked at least in the first part once upon a time. These days, Wdws doesn't even give the option to Find Target when I links target is not longer exactly as stored - just to cancel to delete the shortcut.



(((( Please note that the location information for my overall library management is a necessity; I'm not complicating it; I am using Calibre's features most efficiently so I'm not fishing for "why aren't you using the virtual library or improving tag searches" suggestions as some who read this may try to suggest and miss the point of my inquiry and offer ranting replies. I seem to get those on innocuous inquiries from time to time))))))

[eschwartz]

Re: changing drive letters, I would suggest manually switching the drive letter to something like P: (PortableApps, that's what I did) and doing this on every computer you use.

The computer will remember which drive letter you used for that drive.
No need to fight with reassigned drive letters unless you use a LOT of drives at one time
You'll always know, whatever you have in there, that the drive letters will be correct. Once you properly set up that computer.

[BetterRed]

@jecilop - is the drive formatted as NTFS, and how do you move the files i.e. is 3rd party s/w involved - FastCopy, TerraCopy etc, if latter try moving with vanilla Windows.

IIRC the reason it doesn't ask "do you want windows to find the file" is that finding the file is a lot faster since Win7 (maybe Vista) whereas on XP it would whirr away for a while.

Ah-ha - see first sentence; last paragraph in here ==>> Distributed Link Tracking and Object Identifiers

I'm not a 100% sure, but I don't think DLT has ever worked if .LNK target folder/file was on a portable device.

BR

[jecilop]

Originally Posted by BetterRed

@jecilop - is the drive formatted as NTFS, and how do you move the files i.e. is 3rd party s/w involved - FastCopy, TerraCopy etc, if latter try moving with vanilla Windows.

IIRC the reason it doesn't ask "do you want windows to find the file" is that finding the file is a lot faster since Win7 (maybe Vista) whereas on XP it would whirr away for a while.

Ah-ha - see first sentence; last paragraph in here ==>> Distributed Link Tracking and Object Identifiers

I'm not a 100% sure, but I don't think DLT has ever worked if .LNK target folder/file was on a portable device.

BR

NTFS,,yes,
Moved via drag/drop cut/paste in Wdws Explorer.

"Links to files on removable media are not maintained"...got it...so now it's down to storing in a Calibre library on a portable or links that might move although not regularly..Right now, I'm thinking I'll just put links in my library and go from there.

I REALLY appreciate your input!

[BetterRed]

Originally Posted by jecilop

"Links to files on removable media are not maintained"...got it...so now it's down to storing in a Calibre library on a portable or links that might move although not regularly..Right now, I'm thinking I'll just put links in my library and go from there.

I REALLY appreciate your input!

@jecilop - glad we got the bottom of it, and thanks for thanks. I was a tad concerned that your prior comment about "...missing the point of your inquiry and offering a ranting reply... might have been directed at me

I think I worked out why DLT is 'faster' since - 'I suspect Vista'. In earlier versions of Windows (e.g. 2000/XP) a file system object (file or folder) stored its Object Id in an alternate data stream (ADS) - aka a separate file fork - within context of the file itself. The DLT Service relies on Object ID's to work.

Back when XP ruled the roost, there was a lot of brouhaha that malware could squirrel itself away in an Object ID ADS - although I'm not sure any ever did. With the advent of Vista, MS relocated the Object IDs away from file/folder ADS's into a central table. I think they did they same thing with Security ID's. Not only did this quieten down the brouhaha about ADS's and malware, it also made DLT faster - at the expense of a single point of failure, which MS addressed through replicating the MFT and other tricks.

BR