Create New Library from Existing Library at Command Line

[chaley]

Originally Posted by BetterRed

I'll dive into the GUI python code with the view to doing a stand-alone rewrite of the Copy Structure from Library function to take from_library and to_library off the command line, shouldn't be too hard.

One way to do this is to "create" the library, copy the existing metadata.db to that library (but not the books themselves), then use the command line to delete all the books in the db.

[BetterRed]

Originally Posted by chaley

One way to do this is to "create" the library

do you mean create the folder for the 'new' library?

Originally Posted by chaley

copy the existing metadata.db to that library (but not the books themselves), then use the command line to delete all the books in the db.

I did try that the other day with calibredb remove 0-???? (cant remember) but when I accessed the 'empty clone' via the GUI I got an error that I interpreted as meaning the database doesn't belong to this library - didn't keep the error message.

I will do it again, when I get back to my main calibre system 2 mora.

If the from_library contains a metadata_db_prefs_backup.json should I also copy it to the to_library along with the empty metadata.db - I didn't do that on the test library I used.

BR

[chaley]

Originally Posted by BetterRed

do you mean create the folder for the 'new' library?

yes.

I did try that the other day with calibredb remove 0-???? (cant remember) but when I accessed the 'empty clone' via the GUI I got an error that I interpreted as meaning the database doesn't belong to this library - didn't keep the error message.

I will do it again, when I get back to my main calibre system 2 mora.

I do this frequently when debugging -- create a folder & drop in a metadata.db. I have never had a failure of any kind. I am interested in hearing what is going wrong for you.

If the from_library contains a metadata_db_prefs_backup.json should I also copy it to the to_library along with the empty metadata.db - I didn't do that on the test library I used.

BR

no. That file is used only by db recovery.

[BetterRed]

Originally Posted by chaley

yes.I do this frequently when debugging -- create a folder & drop in a metadata.db.

IIRC I used calibredb remove 1-100000, but the library only had ~120 books, and book-id high wouldn't have been more than 150 - might that have created a problem?

However I was in a hurry, so it's more likely it was my fat thumb stuck in the pie, will try again later today.

BR

[chaley]

Originally Posted by BetterRed

IIRC I used calibredb remove 1-100000, but the library only had ~120 books, and book-id high wouldn't have been more than 150 - might that have created a problem?

Just tried that. It tosses an exception trying to delete a UUID that isn't there. I have submitted a fix to Kovid.

That said, this method is very slow. It tries to delete each of the "books" in the range, failing for most of them. It would be better to get a list of IDs in the DB and delete those. However ...

I discovered that it wasn't possible to get calibredb to give you a list of only ids. I submitted a fix for that as well. Assuming Kovid accepts it, you will be able to say --fields="" to show only the ids. You would then take that output and feed it into calibredb remove, perhaps something like this on a *nix box
calibredb remove `calibredb list --fields=""`

[BetterRed]

Originally Posted by chaley

Just tried that. It tosses an exception trying to delete a UUID that isn't there. I have submitted a fix to Kovid.

That said, this method is very slow. It tries to delete each of the "books" in the range, failing for most of them. It would be better to get a list of IDs in the DB and delete those. However ...

I discovered that it wasn't possible to get calibredb to give you a list of only ids. I submitted a fix for that as well. Assuming Kovid accepts it, you will be able to say --fields="" to show only the ids. You would then take that output and feed it into calibredb remove, perhaps something like this on a *nix box
calibredb remove `calibredb list --fields=""`

Thanks chaley - would it not be better to have a remove * and let calibredb get the list of IDs and do whatever

I gave you the wrong steer before, that error I mentioned was something else that I already solved - it was my error

I've not used the calibredb command before.

The copy of the test database I'm using is at "C:\_Sandpit\metadata.db" Where do I put the database path in the "calibredb remove ids" command. I tried putting the db path before 'remove' and after 'ids'

Could you spell it out in a code box so that I can cut and paste please - assume I want to remove books 1-10

BR

ps: I am not too bothered by how long it runs, it will be in a shutdown script.

[chaley]

Originally Posted by BetterRed

Thanks chaley - would it not be better to have a remove * and let calibredb get the list of IDs and do whatever

Perhaps, but I don't want to add functionality. Fixing something I think it broken is another thing.

Could you spell it out in a code box so that I can cut and paste please - assume I want to remove books 1-10

This works on my library. You can make the substitutions.

calibredb  remove 1-51 --library-path="D:\CBH_Data\calibre.git\Library.test_small"

ps: I am not too bothered by how long it runs, it will be in a shutdown script.

Depending on the upper limit, you might be talking about many hours.

[BetterRed]

Originally Posted by chaley

Perhaps, but I don't want to add functionality. Fixing something I think it broken is another thing.

See last 2 paras

Originally Posted by chaley

This works on my library. You can make the substitutions.

calibredb  remove 1-51 --library-path="D:\CBH_Data\calibre.git\Library.test_small"

I found the --library-path item in the PDF this morning, missed it in the online version both yesterday and last week. The CLI section of the manual would benefit greatly if it had some complete examples of each command.

Originally Posted by chaley

Depending on the upper limit, you might be talking about many hours.

I'll be asleep. Also it wouldn't be too onerous for me to set the upper limit at four weekly intervals, to something more sensible than the suggested 100000, eg 10% above max book ID.

However I anticipated my original question How to Create New Library from Existing Library at the Command Line would have had a straightforward answer - you do it like this or there is no way to do that.

I need to revisit why I asked the question and see if I can achieve what I want via a different route. Whilst I cringe when I think of Autoit (it upsets my puritanical mores) it's looking to be the best option. Isolating the Copy Structure from Library code into a CLI utility may have a limited lifetime given that Kovid is working on a new database schema.

Thanks for your suggestions, if nothing else our discussion did allow you to detect a defect which is a plus.

BR

[PeterT]

Why not just make one new library in the GUI and save it as a template?

Then possibly all you need to do is to create a new directory and copy the template metadata.db into the new directory?

[BetterRed]

Originally Posted by PeterT

Why not just make one new library in the GUI and save it as a template?

Then possibly all you need to do is to create a new directory and copy the template metadata.db into the new directory?

PeterT, but...

That's what I do now mate.

When I change library specific preferences or create a new VL or something like that, then I need to create a new template; but in the 'heat of the moment' I often forget to do that. Also I don't want the templates to be visible in Calibre until I really need them to effect a repair, so after creating fresh templates I then remove them from Calibre. I then I have to explain to those who look over my shoulder why I just did that

So what I want to do is to automate the creation of fresh templates as part of my end-of-day process. I don't care if they are the same as yesterday's, at least when tomorrow comes I will know they are up to date as at the end of the new yesterday .

If I have a disaster I want to be able to rebuild the Calibre libraries from the raw materials that I keep on my FTP server space. The templates are a part of that raw material inventory. And Calibre is a fraction of the data I would want to to rebuild in the event of disaster.

I should have stated what I was trying to do in the first instance, but given the responses I got to a 17 word sentence, I suspect another couple of sentences would only added to the confusion.

Cheers BR

[PeterT]

Ah well Just wasn't sure you had thought of the simple solution

[kovidgoyal]

https://github.com/kovidgoyal/calibr...10270c372a9d9d

[BetterRed]

Originally Posted by PeterT

Ah well Just wasn't sure you had thought of the simple solution

No worries mate, its always worth ensuring that I haven't overlooked the bleedin' obvious - as its something I do often do

BR

[BetterRed]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...10270c372a9d9d

- what can I say. Thank you is no where near enough. Maybe I can think of a better way

BR