Organizing library

[Inferno]

Originally Posted by eschwartz

It is a full backup. Hardlinks save space -- but the file is still there. No file deltas in sight. It leverages rsync --link-dest to perform full copies, hardlinking files to the dest folder if available.

It's not a matter of how easy is it, is a matter of how much of your hard drive do you want to use storing the same file hundreds of times (when it hasn't been changed -- the modified date is still the same).

Once you have it setup (you can schedule it automatically), it creates a snapshots folder that contains timestamped backup folders -- not zipped storage -- which can be easily browsed through the file explorer and you can copy the folder elsewhere and zip it and do whatever and and and...

You are by no means required to use it, or course. I just thought I'd make sure you know what kind of option it is.

My personal preference for backing up my library is to have multiple copies. I don't mind having a tb or two dedicated to my library and backups but that's me. I am a horder of the first degree of my ebooks. Thank goodness they don't take up the physical space my old paperback collection used to.

Having lost my electronic library in the past (back when it was text, html or pdb) due to drive failure and unreadable backup when my zip drives were not reliable and failed I became compulsive about backups. The problem with anything other than a full backup is the ability to restore quickly if there is a complete system failure and I like to rar it rather than just have a copy of the library because it is the library at that time so if a book turns up corrupted at a later date or if I somehow delete something in error I will be able to find it. It's purely a personal quirk.

For me it is a case of thanking the gods of technology for cheap usb drives .

[chaley]

Be careful about hardlinks. They do not make a copy. Instead the same physical file now has two names. If you open one name and change the file then the other sees the changes. If instead one name is deleted and then readded then the link is broken and you now have two copies.

The issue here is metadata.db. Hardlinks won't be broken when calibre opens and writes one of the names.

If I misunderstood what you are saying, sorry about the spam.

[eschwartz]

Originally Posted by chaley

Be careful about hardlinks. They do not make a copy. Instead the same physical file now has two names. If you open one name and change the file then the other sees the changes. If instead one name is deleted and then readded then the link is broken and you now have two copies.

The issue here is metadata.db. Hardlinks won't be broken when calibre opens and writes one of the names.

If I misunderstood what you are saying, sorry about the spam.

This is about an automated backup script, which takes into account things like needing to change the latest version of files. The backup folders are hardlinked to each other, but not to the "real" files. See: rsync manpages, the --link-dest switch. (not "cp -al")

[crescere]

Wow, I appreciate the input here, but most of this is over my head. I have over 1000 books in my library. I plan to use the method described here to manage them: http://manual.calibre-ebook.com/sub_groups.html

So to back up both the books and the tag structure are you all saying I just have to backup the Calibre library in the My documents folder on my computer? The tags and such will just go along with that?

I just plan to periodically put it on an external harddrive.

[crescere]

Originally Posted by BetterRed

@crescere - a calibre library has two main components the author/book folders which are the equivalent of library shelves, and a database (metadata.db), the equivalent of a library catalogue. It is the latter that is the heart and soul of calibre.

When you tag your books, correct author name, rate a book etc the database is updated automatically - you don't have to save it as such.

BR

In case something goes wrong with my computer I would like to backup both the author/book folders and the database.

[theducks]

Originally Posted by crescere

Wow, I appreciate the input here, but most of this is over my head. I have over 1000 books in my library. I plan to use the method described here to manage them: http://manual.calibre-ebook.com/sub_groups.html

So to back up both the books and the tag structure are you all saying I just have to backup the Calibre library in the My documents folder on my computer? The tags and such will just go along with that?

I just plan to periodically put it on an external harddrive.

Tags and other data belong to 'the current library' , so yes. what you see on the screen is contained in that folder.

OTOH
How it looks, plugins and some non-library specific rules are in Preferences (the configuration folder)

Backing Both allows a quick transfer to another system with only minor 'switch library' to take in account of user path changes.

[BetterRed]

Originally Posted by eschwartz

Not sure how you'd get that working on Windows.

You'd probably use a DeLorean Copy. There are shell and command line utilities available

BR

[BetterRed]

Originally Posted by crescere

In case something goes wrong with my computer I would like to backup both the author/book folders and the database.

A calibre Library is

A folder that contains author folders, the author folders contain book folders.

• The author folder names are the author names you enter - in 'purified ascii'
• The book folder names are the book titles you enter - in 'purified ascii'
• The book folders usually contain a cover image, a metadata.opf file, and one or more 'format' files, the format files are the ebook files themselves - eg .epub, .mobo, ,pdf etc.

And there is a file named metadata.db, this is the database I previously mentioned as being the heart and soul of calibre.

And there maybe a file called metadata_db_prefs_backup.json. Together with the metadata.opf files in book folders this provides the means via which the database can be rebuilt if something goes wrong - which is a very rare event.

You should backup the library folder in its entirety and restore it in its entirety. If you 'lose' an individual book then you have to import it back into the calibre via the Add process - which could be from a backup location.

If you already have backup/restore processes and tools in place for other data - eg documents, photos, software, source code etc - then you should probably just include the calibre libraries into those processes.

As I previously mention there's also the configuration data, which should also be backed up.

BR

[crescere]

Originally Posted by BetterRed

• The author folder names are the author names you enter - in 'purified ascii'
• The book folder names are the book titles you enter - in 'purified ascii'
• The book folders usually contain a cover image, a metadata.opf file, and one or more 'format' files, the format files are the ebook files themselves - eg .epub, .mobo, ,pdf etc.

And there is a file named metadata.db, this is the database I previously mentioned as being the heart and soul of calibre.

And there maybe a file called metadata_db_prefs_backup.json. Together with the metadata.opf files in book folders this provides the means via which the database can be rebuilt if something goes wrong - which is a very rare event.[/INDENT]
BR

I use Carbonite to backup my data, but I still want to put the right files on an external drive. Are you saying I have to find the metadata files and back them up separately?

[BetterRed]

Originally Posted by crescere

I use Carbonite to backup my data, but I still want to put the right files on an external drive. Are you saying I have to find the metadata files and back them up separately?

I already told you what you need to backup. I wont change my opinion on that unless the structure of a calibre changes, I doubt that's going happen any time soon.

Originally Posted by BetterRed

You should backup the library folder in its entirety and restore it in its entirety. If you 'lose' an individual book then you have to import it back into the calibre via the Add process - which could be from a backup location.

If you already have backup/restore processes and tools in place for other data - eg documents, photos, software, source code etc - then you should probably just include the calibre libraries into those processes.

As I previously mention there's also the configuration data, which should also be backed up.

I have no knowledge of Carbonite - but its cloud based then you should read Frequently Asked Questions. In particularly this paragraph

If you must share the actual library, use a file syncing tool like DropBox or rsync instead of a networked drive. If you are using a file-syncing tool it is essential that you make sure that both calibre and the file syncing tool do not try to access the calibre library at the same time. In other words, do not run the file syncing tool and calibre at the same time.

applies equally to cloud backup services used for backup.

BR

[pjmpjm]

Originally Posted by Inferno

My personal preference for backing up my library is to have multiple copies. I don't mind having a tb or two dedicated to my library and backups but that's me. I am a horder of the first degree of my ebooks. Thank goodness they don't take up the physical space my old paperback collection used to.

Having lost my electronic library in the past (back when it was text, html or pdb) due to drive failure and unreadable backup when my zip drives were not reliable and failed I became compulsive about backups. The problem with anything other than a full backup is the ability to restore quickly if there is a complete system failure and I like to rar it rather than just have a copy of the library because it is the library at that time so if a book turns up corrupted at a later date or if I somehow delete something in error I will be able to find it. It's purely a personal quirk.

For me it is a case of thanking the gods of technology for cheap usb drives .

Same here. Still remember the backups on 5 1/4 inch floppies. I need to have two complete backups to feel safe. Just bought a brand new 4TB WD My Book to have room for expansion.

[cvlowe]

Originally Posted by theducks

I use hierarchical Tags (very similar to what Hampshire Nanny showed, Except )

Use a dot and no space

You double click Parent (you get 2 Green +), you see ALL History
You select the sub group and you see just that one.

You do need to configure (set up) Hierarchical settings in Preferences: Look and Feel: Tag Browser

Using this method, would you use "fiction.science" or "science fiction" as it's own root tag?

[theducks]

Originally Posted by cvlowe

Using this method, would you use "fiction.science" or "science fiction" as it's own root tag?

Science Fiction is my root tag. (all the other slang names get merged: SciFi, SyFy, SF..). Science Fiction-Fantasy is seriously given scrutiny as it was used for shelving and many titles rarely truly bridged the genres (IMHO just tossing in a space ship/ray gun/sword does not establish a 'bridge')

I am smart enough to infer:
General ???
Fiction < used when 'I' have not assigned a root topic tag (usually because I have so few in that area as to make that a necessity)

I care less where (country/state) the Author lives or a raft of other 'garbage' tags. Do they write well , is how I choose books

[Inferno]

Originally Posted by eschwartz

It is a full backup. Hardlinks save space -- but the file is still there. No file deltas in sight. It leverages rsync --link-dest to perform full copies, hardlinking files to the dest folder if available.

It's not a matter of how easy is it, is a matter of how much of your hard drive do you want to use storing the same file hundreds of times (when it hasn't been changed -- the modified date is still the same).

Once you have it setup (you can schedule it automatically), it creates a snapshots folder that contains timestamped backup folders -- not zipped storage -- which can be easily browsed through the file explorer and you can copy the folder elsewhere and zip it and do whatever and and and...

You are by no means required to use it, or course. I just thought I'd make sure you know what kind of option it is.

Am I mistaken that if I delete the file the link points to it is gone? In the backup as well? or am I missing something?

[BetterRed]

Originally Posted by Inferno

Am I mistaken that if I delete the file the link points to it is gone? In the backup as well? or am I missing something?

Normal behaviour is that deleting a 'hardlink' (i-node) doesn't delete the 'file' (datastream) unless its the last i-node to the datastream in question.

I'm not sure about e..z's suggested usage of hardlinks and rsynch, maybe he can tell us if there are circumstances that the scripts do delete all the hard links for a given datastream.

BR