Calibre2opds wiped out my calibre collection

[reup]

ok, you all probably will said it is my fault, but I was trying out Calibre2opds and the result is that I lost all my calibre books, all of it

I had the following tree :

/home/reup/Ebook/

in this I had all my ebooks, under divers directories, and a calibre directory in which there was my calibre database and books already added to calibre

I try to create a catalog of some books, using Calibre2opds, and put it into the same dir, /home/reup/Ebook

result, the ebook directory now contains only the catalog created by Calibre2opds, all other directories have been wiped out, all my calibre books, all my documentation every fucking single thing.

what would have append if I had choose /home/reup as directory ? all my homedir would have disappeared ?

any idea how I can get back my books ??

a crying for help, very pissed off reup

[itimpi]

I do not see what you can do to recover the books except see if they are in a trash can, or recover them from your backups.

I am not sure quite how you managed that as there are some checks made that should prevent it happening unless you have done something that manages to circumvent the checks? It is not a consolation, but I would still like to understand exactly what you did so that the checks against such accidents can be improved. Also, what version of Calibre2Opds were you using - some of the checks were only introduced in the 2.4 series. However I suspect you are on the latest so that it sounds as if the existing checks should be improved if the combination of settings you used that triggered the issue can be identifed.

Normally you would set Calibre2Opds to the folder containing the Calibre library, and create the catalog as a sub-folder of that (the catalog HAS to be a sub-folder of the Calibre library folder to operate correctly). Calibre2Opds will have checked that it has a metadata.db file in the library path you specify before it would even start attempting to generate a catalog. It should also give you a warning that the existing contents of the catalog folder would be purged if it did not find an existing calibre2opds catalog at that location.

[reup]

itimpi,

I thank you for your answer, and I want to make sure that you do not take my bad mood as directed towards you

I just find incredible that someone put out-there a program that start by emptying a directory

you are right, I am using the last version :

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>calibre2opds 2.4</comment>
</properties>

and no, there was no alert, nothing

Normally you would set Calibre2Opds to the folder containing the Calibre library, and create the catalog as a sub-folder of that

I found nowhere in the doc any info stating that I cannot create the catalog in my homedir if I want to. it seems to me that if it start by emptying the dir, that would be a big red alert somewhere. so what, my homedir can be deleted by this program ?

the fact is that the catalog production worked, I end up with a perfect catalog of the 70 books I included into it, a nice directory per writer, so even if there was no metadata.db in the Ebook directory, it did find the Ebooks/Calibre/metadata.db

reup

[reup]

actually, I put everything concerning ebooks in /home/reup/Ebooks, even the program Calibre2opds and guess what, it even deleted itself

reup

[reup]

from the documentation on https://answers.launchpad.net/calibre2opds/+faq/878

Example on a Unix computer:
---------------------------------------
Say you've got Calibre up and running, and that its database folder is ~/calibre/database.
Let's assume that you installed calibre2opds into /usr/local/calibre2opds.

Then, when you type the following command, the catalogs will be generated into ~/calibre/data/_catalog with the default options :

/usr/local/calibre2opds/bin/calibre2opds ~/calibre/data

If you want to change the default behavior, you'll have to use options ; issue the /usr/local/calibre2opds/bin/calibre2opds --help command to see which options are available.

as you can see, you do not have to set the catalog in the same folder than the database

I used the gui, that gives you the possibility to point the result to another folder than the one where calibre is located

reup

[DoctorOhh]

Originally Posted by reup

actually, I put everything concerning ebooks in /home/reup/Ebooks, even the program Calibre2opds and guess what, it even deleted itself

reup

If all of the books are still in their directories you can restore the metadata.db file via the restore database feature, see attached.

[From the manual]All the metadata is stored in a single file called metadata.db, in the top level folder. If this file gets corrupted (or deleted), you may see an empty list of books in calibre. In this case you can ask calibre to restore your books by clicking the arrow next to the calibre icon on the toolbar and selecting Library Maintenance->Restore Library.

Bold words above, my words added for clarity.

[reup]

Originally Posted by dwanthny

If all of the books are still in their directories you can restore the metadata.db file via the restore database feature, see attached.

Bold words above, my words added for clarity.

unfortunately the only books left are the ones copied by the catalog creation

I do not hope to get back my files, I had a lot of files in the Ebooks directories that are not part of calibre, some documentation created by myself, some ebooks downloaded but not yet entered into calibre.

I am only trying to prevent someone else having the same issue

if I had created the catalog into my /home/reup, I would have lost everything, not only ebooks but all my files. how crazy is that !

I have the snapshots of the settings from calibre2opds and also the default_profile.xml. I will try to post it and you can see that nothing I did stoped any warning from happening, but they did not !

cat default.profile.xml 
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>calibre2opds 2.4</comment>
<entry key="BrowseByCoverWithoutSplit">true</entry>
<entry key="GoodreadReviewIsbnUrl">http://www.goodreads.com/review/isbn/{0}</entry>
<entry key="PublishDateAsYear">true</entry>
<entry key="IncludeBooksWithNoFile">false</entry>
<entry key="GenerateHtml">true</entry>
<entry key="GenerateTags">true</entry>
<entry key="ReprocessEpubMetadata">false</entry>
<entry key="IncludeOnlyOneFile">false</entry>
<entry key="CatalogTitle">FNA</entry>
<entry key="WikipediaLanguage">en</entry>
<entry key="GenerateCrossLinks">true</entry>
<entry key="GenerateOpdsDownloads">true</entry>
<entry key="BookLanguageTag">Lang:</entry>
<entry key="LibrarythingAuthorUrl">http://www.librarything.com/author/{0}</entry>
<entry key="WikipediaUrl">http://{0}.wikipedia.org/wiki/{1}</entry>
<entry key="GenerateOpds">true</entry>
<entry key="MinBooksToMakeDeepLevel">50</entry>
<entry key="LibrarythingIsbnUrl">http://www.librarything.com/isbn/{0}</entry>
<entry key="TagsToGenerate">FNA</entry>
<entry key="MaxBeforePaginate">25</entry>
<entry key="CoverHeight">550</entry>
<entry key="CryptFilenames">false</entry>
<entry key="IncludeAboutLink">false</entry>
<entry key="SuppressRatingsInTitles">true</entry>
<entry key="AmazonAuthorUrl">http://www.amazon.com/gp/search/ref=sr_adv_b/?search-alias=stripbooks&amp;unfiltered=1&amp;sort=relevanceexprank&amp;field-author={0}</entry>
<entry key="CopyToDatabaseFolder">false</entry>
<entry key="ZipTrookCatalog">false</entry>
<entry key="GenerateDownloads">true</entry>
<entry key="MinimizeChangedFiles">true</entry>
<entry key="VERSIONCHIP">02030101</entry>
<entry key="MaxBeforeSplit">75</entry>
<entry key="LibrarythingTitleUrl">http://www.librarything.com/title/{0} {1}</entry>
<entry key="GenerateExternalLinks">false</entry>
<entry key="BooksInRecentAdditions">99999</entry>
<entry key="CompatibilityTrick">OPDS</entry>
<entry key="ExternalIcons">false</entry>
<entry key="SaveBandwidth">true</entry>
<entry key="AmazonIsbnUrl">http://www.amazon.com/gp/search/ref=sr_adv_b/?search-alias=stripbooks&amp;unfiltered=1&amp;sort=relevanceexprank&amp;field-isbn={0}</entry>
<entry key="SplitInAuthorBooks">false</entry>
<entry key="TagsToExclude"/>
<entry key="CoverResize">true</entry>
<entry key="TagsToMakeDeep"/>
<entry key="GoodreadIsbnUrl">http://www.goodreads.com/book/isbn/{0}</entry>
<entry key="AmazonTitleUrl">http://www.amazon.com/gp/search/ref=sr_adv_b/?search-alias=stripbooks&amp;unfiltered=1&amp;sort=relevanceexprank&amp;field-title={0}&amp;field-author={1}</entry>
<entry key="MaxSummaryLength">30</entry>
<entry key="DatabaseFolder">/home/reup/Ebooks/Calibre/ALL Ebooks</entry>
<entry key="TargetFolder">/home/reup/Ebooks</entry>
<entry key="OrderAllBooksBySeries">true</entry>
<entry key="GenerateAllbooks">true</entry>
<entry key="SplitByAuthorInitialGoToBooks">false</entry>
<entry key="BrowseByCover">true</entry>
<entry key="SplitInSeriesBooks">false</entry>
<entry key="ShowSeriesInAuthorCatalog">true</entry>
<entry key="CatalogFolderName">FNA_Catalog</entry>
<entry key="SplitTagsOn"/>
<entry key="GoodreadTitleUrl">http://www.goodreads.com/book/title/{0}</entry>
<entry key="IsfdbAuthorUrl">http://www.isfdb.org/cgi-bin/ea.cgi?{0}</entry>
<entry key="MaxMobileResolution">960</entry>
<entry key="Language">en</entry>
<entry key="GenerateHtmlDownloads">true</entry>
<entry key="GenerateRatings">false</entry>
<entry key="ThumbnailGenerate">true</entry>
<entry key="IncludedFormatsList">EPUB, PDF, RTF, TXT, PRC, PDB, MOBI, LRF, LRX, FB2</entry>
<entry key="GoodreadAuthorUrl">http://www.goodreads.com/book/author/{0}</entry>
<entry key="ThumbnailHeight">144</entry>
</properties>

[itimpi]

Originally Posted by reup

from the documentation on https://answers.launchpad.net/calibre2opds/+faq/878

Example on a Unix computer:
---------------------------------------
Say you've got Calibre up and running, and that its database folder is ~/calibre/database.
Let's assume that you installed calibre2opds into /usr/local/calibre2opds.

Then, when you type the following command, the catalogs will be generated into ~/calibre/data/_catalog with the default options :

/usr/local/calibre2opds/bin/calibre2opds ~/calibre/data

If you want to change the default behavior, you'll have to use options ; issue the /usr/local/calibre2opds/bin/calibre2opds --help command to see which options are available.

as you can see, you do not have to set the catalog in the same folder than the database

That answer is obsolete (it applied to calibre2opds v1) as you can no longer provide parameters via the command line. You have to set all parameters via the GUI (although you can run with previously stored parameters via the command line).

I used the gui, that gives you the possibility to point the result to another folder than the one where calibre is located

I am still not sure what you did? When running in normal mode you set the Calibre library path (Database folder) and the catalog folder. The destination folder option is disabled in normal mode. Unless you set something else such as Publish or Nook modes there is no option to specify a different target folder, and they are only expected to be used when copying the library to another location. I still want to work out what this extra step was that you used to specify a folder that was not the Calibre library folder as it sounds like there is a path that does not include the checks that are on with default settings.

[reup]

Unless you set something else such as Publish or Nook modes there is no option to specify a different target folder

if you can tell me how to send you a jpg, I have a screenshot of the gui, you can see clearly that you can select a folder where you want on your hard drive

I have on the first tab of the gui :
database folder /home/reup/Ebooks/Calibre/ALL Ebooks
destination folder /home/reup/Ebooks

reup

[reup]

here I put them. the 4 tabs of the gui :
http://blog.yelomart.fr/images/calibre/calibre1.png
http://blog.yelomart.fr/images/calibre/calibre2.png
http://blog.yelomart.fr/images/calibre/calibre3.png
http://blog.yelomart.fr/images/calibre/calibre4.png


reup

[itimpi]

Reup: Thanks for that - but I am afraid that I get an error 404 trying to view any of those. If you want to email them to me at calibre2opds at itimpi dot freeserve dot co dot uk I can check them that way. Alternatively (and probably more useful) there should be a calibre configuration profile in the .calibre2opds folder under your home folder that will show all the settings used at the time. It will probably be called default.xml if you have not set up an explicitly named profile. An alternative is the calibre2opds.log file from the log sub-folder of that location as that also lists all configuration parameters active when the last generation run was done.

Probably all I need to know is which of the three big 'mode' buttons at the top is selected. You should, I believe, only be able to set the Destination folder if the second (Publish) or third (nook) modes are selected. If that is not the case I definitely want to know as it indicates some unexpected behavior.

As a result of your problem I have pulled up the code that controls the copying of catalog and library files (which thankfully I did not write ) and made certain that in all cases where there is not an existing calibre2opds catalog at the target location then a warning dialog is popped up giving the path sand aying that all the existing contents of that location will be lost. The run is aborted if the user does not OK that warning. The warning is not being pioped up if there is already a catalog there so as it will be an exception hopefully users will reading it before giving the OK to calibre2opds to proceed.

The reason that files get deleted at all is that if you are overwriting an existing catalog and file names have been changed, then if you did not delete files no longer wanted you would gradually accumulate a residue of orphaned catalog files. However I am wondering if that might occasonially be a pain it is worth taking by default? If so I could add a checkbox to the GUI for "never delete files" defaulting to on to allow a user to control such behavior. However, I am not sure that it is worth it now that the checks for for making sure what is being silently overwritten is already a calibre2opds catalog have been improved for the next calibre2opds release.

[reup]

itimpi,

I have send you the 4 snapshot by mail, thank you

in /home/reup/.calibre2opds there is only 3 files and one log dir

.calibre2opds.secureFileManager.xml
profile.xml
default.profile.xml

the first two are the same,

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>calibre2opds 2.4</comment>
</properties>

the last one, default.profile.xml is already posted higher in this thread.

I do not repeat it here to not overfill this thread

so far, I know very little software that can erase so blindly a directory, so potentially dangerous, the famous rm -rf, and some fdisk or mkfs. all are more system commands and usually used only by root. imagine my surprise when just by creating a catalog of current files, I loose 70 000 (yes 70 000) files without any warning. I compile a lot of docs for my work, I lost more than 9gb in few second ans still no warning. I dont think any software should have this"freedom" on my homedir

again, I do not hope to retrieve the files. yes I had a backup, old one, but all the files were there, only all the work done to catalog them all in calibre is lost.

what is the point to use alias rm=rm -i if a simple catalog software can wipe it all out

I am very grateful to calibre dev team, and I believe there is no connection between calibre and calibre2opds. this was my first try of calibre2opds, looking for an easy way to share the doc with my colleagues (we are creating a catalog of SF ebooks, some empty, some with files)

reup

[itimpi]

Originally Posted by reup

itimpi,

I have send you the 4 snapshot by mail, thank you

Thanks - I will await their arrival.

in /home/reup/.calibre2opds there is only 3 files and one log dir

.calibre2opds.secureFileManager.xml
profile.xml
default.profile.xml

the last one, default.profile.xml is already posted higher in this thread.

Sorry - I had missed that.

Looking at that file I think there is a bug in that when calibre2opds is first run on a system the "Mode" being used is not being set to default in the saved settings. That is why you got the option to set the Destination folder which should not be allowed in default mode. The mode is getting stored as soon as you click on one of the mode buttons, but not before.

so far, I know very little software that can erase so blindly a directory, so potentially dangerous, the famous rm -rf, and some fdisk or mkfs. all are more system commands and usually used only by root. imagine my surprise when just by creating a catalog of current files, I loose 70 000 (yes 70 000) files without any warning. I compile a lot of docs for my work, I lost more than 9gb in few second ans still no warning. I dont think any software should have this"freedom" on my homedir

The problem is that any software can have a bugs and you were unfortunate to fall foul of one. Calibre2opds is not MEANT to be able to remove files that are not part of its catalogs, but one of the checks as to whether a particular folder was a catalog folder went wrong (or never got run because of the particular combination of settings) combined with you having an unfortunate folder path setting that meant it was pointing at a folder where it could do some damage. This scenario had never come up in any testing as far as I know.

[reup]

itimpi,

at least something good will come out of it, one bug found and corrected :-)

reup

[itimpi]

Originally Posted by reup

itimpi,

at least something good will come out of it, one bug found and corrected :-)

reup

Agrred.

I have also made the checks on whether the folder containing files to be deleted is a catalog folder more robust and made sure that all output paths include the checks regardles (I think) of the combination of settings the user has set so the chances of an accident are smaller even if something has been overlooked.

Unfortunately I cannot see a way to completely remove the possibility if after warning the use that what they are about to do( and giving them the details) looks dangerous they decide to continue anyway.