Reimport same library in Calibre (without Save to Disk)?

rebl · 05-26-2015, 06:21 AM

Hello,
Is it possible to re-import the library folder "as is" in Calibre - without using "save to disk"?
I have some files with wrong extension that are causing troubles when converting, but they are already in the library and I can't rename.
I tried to save to disk all library, but some of the files are not exported (some mobi, some docs...).
So I'm thinking what if I move the library folder in a different location (in order to keep the files), delete the library from Calibre, rename the wrong extensions and then re-import from the old library folder the files in a new library?
Would this be possible? any cons?
Thank you.

L.E.: I assume the above might be quite a stupid question, or that I should have tried at least to RTFM first. The problem is I don't know where to search for the info I need.
I don't really know how does Calibre works with the OPF files and if they are up-to-day in every moment or do i need to "commit changes" to save the information to them.
[All right I found this information - they are updated after every change in the library]
I don't know if the OPF file stores also the exact file name and extension - I opened a few, for the doc file the extension was not metioned in the OPF file. So if I rename the doc to rtf, Calibre might import the file and the opf in the same folder even if I've changed the extension?
Also, what is the db file used for?
If all the info needed is in the opf files, will calibre read them if they are just named "metadata.opf" in the same folder as the book? If yes, then I could reimport the library without saving to disk first. But what if calibre would read an opf file only if it has the same name as the book file?

L.E.2:
I did a small test and imported in a new library a subfolder of the existing library containing a single book with 3 formats and an metadata.opf file.
Even if my settings ask for <authors> - <title> and the books were stored as Calibre usually does (title - author.ext) the books were imported correctly, with title and author in the correct places. So it seems that Calibre reads the metadata.opf file if adding "one book per directory". Then why do I see the recommendation to never do this, never copy the library, but instead use the Save to disk? Isn't it simpler to just copy the library and import in the new installation/PC/library? For me save to disk fails outputs a few error messages and there are a few missing books.

L.E.3:
Why do I ask this question here and don't test it myself? Because there are about 20.000 books in the library and I'm afraid I'm going to break something; also every save to disk or re-import all would take more than an hour.... I was hoping to find what's best to do, before doing it. Also, If I want to rename the files inside the existing library and that's not the best thing to do... I can't undo it.
Thanks for any input !!!

rebl · 05-28-2015, 09:56 AM

Sorry to bump this. I am trying to apply BetterRed's method described here:
https://www.mobileread.com/forums/sho...50&postcount=2
So I would try to export all my doc files, delete the ones in the library and reimport the exported ones after doing some operations on them (rename some of them from DOC to RTF).
My library shows in tag browser that I have 15660 books with DOC format. The same can be seen if searching for formats:"=DOC". Some of them have other formats, too.
I selected these books and chose save single format, DOC and the process finished without errors.
However, when searching in the import folder it only shows 15624 doc files.

Does anyone have any ideea why the exported number of DOC files is smaller?
Thank you.

mbovenka · 05-28-2015, 10:42 AM

Quote:

Originally Posted by rebl

I selected these books and chose save single format, DOC and the process finished without errors.
However, when searching in the import folder it only shows 15624 doc files.

Does anyone have any ideea why the exported number of DOC files is smaller?

Duplicate filenames? Perhaps your Save to Disk format string ended up with the same filename for different books.

theducks · 05-28-2015, 11:32 AM

Been there, got the scars

Duplicate filenames can happen in Windows when the same library folder has been operated on by Both Linux (Case Sensitive) and Windows (case insensitive)

rebl · 05-29-2015, 02:24 AM

Thank you all.

Sorry, I meant "However, when searching in the EXPORT folder it only shows 15624 doc files."

Well... duplicate files? this is something new. I am only using Windows 8.1 (not linux). I see it mentioned somewhere else too, also regarding Calibre:
http://ask.metafilter.com/222066/Dup...he-same-folder
I think calibre (even if it's cross patform) should rename the duplicate files it would encounter when saving to disk - and it actually does: I have duplicates (same title and author, different ID) and when saving to disk they appear as file name (1).doc. So I shouldn't have duplicate files -unless maybe when the letter case is different - that would mean I could have FileName.doc and filename.doc in the same folder ...

I am counting the files in the EXPORT folder using Void Tools' "Everything Search" -maybe it is unable to count duplicate files as different files, I'm gonna check it out.

Another solution might be to save to disk using the {id} in the folder name, so even if two files would have the same name they should be in different folders. I will try that.

L.E.: Unforutnatelly I can exclude the duplicate files hypothesis. There are 31274 files in the export folder, including the opf files; there are 15624 doc files and 15627 opf files and 26 covers. The 31274 file count is reported both by windows (folder properties) and "everything search".
I can only assume Calibre is not exporting all my 15660 doc files - but the why isn't there any error message?

BetterRed · 05-29-2015, 04:28 AM

@rebl - calibre reports on files it knows about in its database, ordinarily it does NOT walk the folder tree. Run Library Maintenance-Check library phases 1 and 2 - resolve any errors in phase 2.

BTW in Windows you cannot have 2 files... filename.doc and FileName.doc... in the same folder. You can in Linux and OSX (maybe not all versions).

BR

rebl · 05-29-2015, 06:07 AM

Quote:

Originally Posted by BetterRed

@rebl - calibre reports on files it knows about in its database, ordinarily it does NOT walk the folder tree. Run Library Maintenance-Check library phases 1 and 2 - resolve any errors in phase 2.

BTW in Windows you cannot have 2 files... filename.doc and FileName.doc... in the same folder. You can in Linux and OSX (maybe not all versions).

BR

Thanks BetterRead.
My library is checked and has been checked periodically (doing it right now one more time). It does the vacuum thing, then checking for matching information and files. No errors found. After step 2, no errors, "the files match the information in the database"
In tag browser or format search I see 15660 doc files.
I've done a new save to disk, using {id} in the folder path and the results are surprisingly different than before.
Now I have
- 15660 opf files, 15660 doc files and 26 jpg files.
Windows reports 31346 files in folder properties.
I've deleted the previous export folder with the missing files (which only had 15624 doc files) so I can't go batch the check it in more depth. I also find a small problem in the way I was counting doc files - three of the files had ".doc" in the file names itslef so if not paying attention and searching for .doc I also had three .doc.opf files (but that doesn't explain the missing 36 or possibly 39 doc files).

So my conclusion to this is that it is safer to "uniquify" the save to disk folder naming scheme. If I did this I got the expected result.
When I did not, something wrong happened in the save to this process. Maybe calibre since it also supports linux, is able to produce filename.ext and FileName.ext pairs? I don't know, I can't test it anymore and I've lost a lot of time already to this. I've also seen lots of filename (1).ext and (2) in the same exported book folder... so in theory calibre does produce unique file names but... who knows what happened....

BetterRed · 05-29-2015, 06:36 AM

The use of filename(n).ext names to avoid duplicates is a commonly used Windows convention.

On Windows there is no way that Calibre can create files named filename.doc and FileName.doc in the same location, Windows just won't allow it - period.

Like any application, calibre is subservient to the operating system in use. Calibre also has copious amounts of code to work with and around the vagaries of the different file systems it supports.

BR

rebl · 05-29-2015, 09:08 AM

I won't argue with you since by no means I know the internals of calibre.
But I know sometimes, some (improperly coded) apps can circumvent an OS's limitations and just write improperly named files (for example apps that can produce files ending with dot in the extension or folders ending with dot in the name - such files or folders can be produced by.... I'm not sure... by WinRar -maybe when extracting an archive made under linux or "flexible renamer" when batch renaming files... I don't remember exactly but I know I managed to got such files in my system)... I am not implying calibre would be improperly coded, but bugs happen (also I am not implying the issue I've experience is a bug since I haven't excluded all the possible causes- even me not counting the files properly... but that's little improbable).
Also some cross-plaform apps can sometimes create such problems - for example this case randomly found on web when cygwin created a "readme." file which is not easily deleted in windows (must use a vbs script or the \\?\* trick to delete it).
http://superuser.com/questions/49495...t-in-windows-7
I might try (when I have more time) to reproduce the issue of the missing DOC files by re-saving without the {id} in the folder name and see what files are really missing. For the moment since I've already deleted the initial first export folder (I was little tight on hdd space) I can't really say what was wrong. For me, inserting the {id} in the save path instantly solved the problem so I can only assume that somehow calibre either skipped some files because of name conflict or ... created duplicates

I still wonder what happened in this other person's case, too:
http://ask.metafilter.com/222066/Dup...he-same-folder

L.E.: I can't anything right now (calibre is adding books) but if that is indeed a bug, should be easily reproduced this way:
with an empty database, two books should be added; same author, same format, but different ID's; one should be titled "This is a title" and the other one "This iS a title"; then a save to disk should be performed, with the default path template. If the bug is real, two files should be saved, in author sort\title\ with case sensitive names... I'm burning with curiosity to test this, but it might just be that the issue won't be reprudced this easily and need the whole 15660 files instead lol

eschwartz · 06-11-2015, 08:53 PM

FWIW, I have used the add-one-book-per-directory option on an actual calibre library on several occasions. It works just fine.

Generally, we recommend people forget about the library folder simply because most users are prone to doing things they didn't mean to or shouldn't do -- like editing the files, moving folders, etc. -- and expecting calibre to pick up on that.

The metadata.opf records the metadata from the book record, and when importing with the one-book-per-directory method, any and all formats in that folder will be added with the OPF metadata.

Side note: filenames ending in a dot are valid filenames. Windows doesn't like that because it likes to conceal complexity from the user, and apparently that results in some bugs.

Just like filenames that begin with a dot, which IIRC Windows willtry to stop you from creating, but unix software ported to Windows like Apache webserver require. e.g. the .htaccess file.

The NTFS filesystem allows certain things that Windows doesn't (and IIRC that includes case-sensitivity in filenames. Create it on linux, and it will show and cause errors on Windows). But there are no Windows APIs to do that AFAIK.

05-26-2015, 06:21 AM	#1
rebl r.eads e.njoys b.ooks lol Posts: 76 Karma: 580748 Join Date: Mar 2010 Location: It's time to get this Book a Rest Device: Kindle 4 NT	Reimport same library in Calibre (without Save to Disk)? Hello, Is it possible to re-import the library folder "as is" in Calibre - without using "save to disk"? I have some files with wrong extension that are causing troubles when converting, but they are already in the library and I can't rename. I tried to save to disk all library, but some of the files are not exported (some mobi, some docs...). So I'm thinking what if I move the library folder in a different location (in order to keep the files), delete the library from Calibre, rename the wrong extensions and then re-import from the old library folder the files in a new library? Would this be possible? any cons? Thank you. L.E.: I assume the above might be quite a stupid question, or that I should have tried at least to RTFM first. The problem is I don't know where to search for the info I need. I don't really know how does Calibre works with the OPF files and if they are up-to-day in every moment or do i need to "commit changes" to save the information to them. [All right I found this information - they are updated after every change in the library] I don't know if the OPF file stores also the exact file name and extension - I opened a few, for the doc file the extension was not metioned in the OPF file. So if I rename the doc to rtf, Calibre might import the file and the opf in the same folder even if I've changed the extension? Also, what is the db file used for? If all the info needed is in the opf files, will calibre read them if they are just named "metadata.opf" in the same folder as the book? If yes, then I could reimport the library without saving to disk first. But what if calibre would read an opf file only if it has the same name as the book file? L.E.2: I did a small test and imported in a new library a subfolder of the existing library containing a single book with 3 formats and an metadata.opf file. Even if my settings ask for <authors> - <title> and the books were stored as Calibre usually does (title - author.ext) the books were imported correctly, with title and author in the correct places. So it seems that Calibre reads the metadata.opf file if adding "one book per directory". Then why do I see the recommendation to never do this, never copy the library, but instead use the Save to disk? Isn't it simpler to just copy the library and import in the new installation/PC/library? For me save to disk fails outputs a few error messages and there are a few missing books. L.E.3: Why do I ask this question here and don't test it myself? Because there are about 20.000 books in the library and I'm afraid I'm going to break something; also every save to disk or re-import all would take more than an hour.... I was hoping to find what's best to do, before doing it. Also, If I want to rename the files inside the existing library and that's not the best thing to do... I can't undo it. Thanks for any input !!! Last edited by rebl; 05-26-2015 at 08:50 AM.

05-29-2015, 02:24 AM	#5
rebl r.eads e.njoys b.ooks lol Posts: 76 Karma: 580748 Join Date: Mar 2010 Location: It's time to get this Book a Rest Device: Kindle 4 NT	Thank you all. Sorry, I meant "However, when searching in the EXPORT folder it only shows 15624 doc files." Well... duplicate files? this is something new. I am only using Windows 8.1 (not linux). I see it mentioned somewhere else too, also regarding Calibre: http://ask.metafilter.com/222066/Dup...he-same-folder I think calibre (even if it's cross patform) should rename the duplicate files it would encounter when saving to disk - and it actually does: I have duplicates (same title and author, different ID) and when saving to disk they appear as file name (1).doc. So I shouldn't have duplicate files -unless maybe when the letter case is different - that would mean I could have FileName.doc and filename.doc in the same folder ... I am counting the files in the EXPORT folder using Void Tools' "Everything Search" -maybe it is unable to count duplicate files as different files, I'm gonna check it out. Another solution might be to save to disk using the {id} in the folder name, so even if two files would have the same name they should be in different folders. I will try that. L.E.: Unforutnatelly I can exclude the duplicate files hypothesis. There are 31274 files in the export folder, including the opf files; there are 15624 doc files and 15627 opf files and 26 covers. The 31274 file count is reported both by windows (folder properties) and "everything search". I can only assume Calibre is not exporting all my 15660 doc files - but the why isn't there any error message? Last edited by rebl; 05-29-2015 at 03:34 AM.

05-29-2015, 09:08 AM	#9
rebl r.eads e.njoys b.ooks lol Posts: 76 Karma: 580748 Join Date: Mar 2010 Location: It's time to get this Book a Rest Device: Kindle 4 NT	I won't argue with you since by no means I know the internals of calibre. But I know sometimes, some (improperly coded) apps can circumvent an OS's limitations and just write improperly named files (for example apps that can produce files ending with dot in the extension or folders ending with dot in the name - such files or folders can be produced by.... I'm not sure... by WinRar -maybe when extracting an archive made under linux or "flexible renamer" when batch renaming files... I don't remember exactly but I know I managed to got such files in my system)... I am not implying calibre would be improperly coded, but bugs happen (also I am not implying the issue I've experience is a bug since I haven't excluded all the possible causes- even me not counting the files properly... but that's little improbable). Also some cross-plaform apps can sometimes create such problems - for example this case randomly found on web when cygwin created a "readme." file which is not easily deleted in windows (must use a vbs script or the \\?\* trick to delete it). http://superuser.com/questions/49495...t-in-windows-7 I might try (when I have more time) to reproduce the issue of the missing DOC files by re-saving without the {id} in the folder name and see what files are really missing. For the moment since I've already deleted the initial first export folder (I was little tight on hdd space) I can't really say what was wrong. For me, inserting the {id} in the save path instantly solved the problem so I can only assume that somehow calibre either skipped some files because of name conflict or ... created duplicates I still wonder what happened in this other person's case, too: http://ask.metafilter.com/222066/Dup...he-same-folder L.E.: I can't anything right now (calibre is adding books) but if that is indeed a bug, should be easily reproduced this way: with an empty database, two books should be added; same author, same format, but different ID's; one should be titled "This is a title" and the other one "This iS a title"; then a save to disk should be performed, with the default path template. If the bug is real, two files should be saved, in author sort\title\ with case sensitive names... I'm burning with curiosity to test this, but it might just be that the issue won't be reprudced this easily and need the whole 15660 files instead lol Last edited by rebl; 05-29-2015 at 09:22 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Confused.. Difference between a Calibre Library & Save to Disk	ipsofacto	Library Management	5	01-05-2012 09:46 AM
Can't Save to Disk (calibre 0.8.21, iPad 1)	brynnflynn	Library Management	0	10-13-2011 11:18 AM
Calibre will not save to disk	mrbillb	Calibre	0	09-10-2010 06:56 PM
Save to Disk: How to use Calibre with Windows Mobile devices?	Starson17	Calibre	0	12-27-2009 09:56 PM
Can't SAVE TO DISK Calibre 0.6.24	Rootman	Calibre	5	12-06-2009 05:01 AM

05-28-2015, 09:56 AM	#2
rebl r.eads e.njoys b.ooks lol Posts: 76 Karma: 580748 Join Date: Mar 2010 Location: It's time to get this Book a Rest Device: Kindle 4 NT	Sorry to bump this. I am trying to apply BetterRed's method described here: https://www.mobileread.com/forums/sho...50&postcount=2 So I would try to export all my doc files, delete the ones in the library and reimport the exported ones after doing some operations on them (rename some of them from DOC to RTF). My library shows in tag browser that I have 15660 books with DOC format. The same can be seen if searching for formats:"=DOC". Some of them have other formats, too. I selected these books and chose save single format, DOC and the process finished without errors. However, when searching in the import folder it only shows 15624 doc files. Does anyone have any ideea why the exported number of DOC files is smaller? Thank you.

05-28-2015, 11:32 AM	#4
theducks Well trained by Cats Posts: 31,047 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Been there, got the scars Duplicate filenames can happen in Windows when the same library folder has been operated on by Both Linux (Case Sensitive) and Windows (case insensitive)

05-29-2015, 04:28 AM	#6
BetterRed null operator (he/him) Posts: 21,722 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	@rebl - calibre reports on files it knows about in its database, ordinarily it does NOT walk the folder tree. Run Library Maintenance-Check library phases 1 and 2 - resolve any errors in phase 2. BTW in Windows you cannot have 2 files... filename.doc and FileName.doc... in the same folder. You can in Linux and OSX (maybe not all versions). BR

05-29-2015, 06:36 AM	#8
BetterRed null operator (he/him) Posts: 21,722 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	The use of filename(n).ext names to avoid duplicates is a commonly used Windows convention. On Windows there is no way that Calibre can create files named filename.doc and FileName.doc in the same location, Windows just won't allow it - period. Like any application, calibre is subservient to the operating system in use. Calibre also has copious amounts of code to work with and around the vagaries of the different file systems it supports. BR

06-11-2015, 08:53 PM	#10
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	FWIW, I have used the add-one-book-per-directory option on an actual calibre library on several occasions. It works just fine. Generally, we recommend people forget about the library folder simply because most users are prone to doing things they didn't mean to or shouldn't do -- like editing the files, moving folders, etc. -- and expecting calibre to pick up on that. The metadata.opf records the metadata from the book record, and when importing with the one-book-per-directory method, any and all formats in that folder will be added with the OPF metadata. Side note: filenames ending in a dot are valid filenames. Windows doesn't like that because it likes to conceal complexity from the user, and apparently that results in some bugs. Just like filenames that begin with a dot, which IIRC Windows willtry to stop you from creating, but unix software ported to Windows like Apache webserver require. e.g. the .htaccess file. The NTFS filesystem allows certain things that Windows doesn't (and IIRC that includes case-sensitivity in filenames. Create it on linux, and it will show and cause errors on Windows). But there are no Windows APIs to do that AFAIK.

Advert

Advert