Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Related Tools

Notices

Reply
 
Thread Tools Search this Thread
Old 03-29-2010, 12:38 PM   #46
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by chaley View Post
I am part of a multi-country multi-partner project at my university, and we use dropbox to share code and papers under construction. We have frequently (several times per week) had cases where changes were simultaneously made to a file in different dropboxes. The result was the creation of 'conflict' files, all of which needed to be manually examined to determine which one was correct. In a couple of cases, neither were correct; we had to merge them together.
Interesting (and slightly horrifying) use for dropbox, why don't you use a version control system?

On the question of robustness. I'm not a dropbox user and I really don't know too much about it, but as Starson17 pointed out, calibre relies on an sqlite database. I would be rather surprised if dropbox was able to handle concurrent changes to the database file, which to it is presumably an opaque binary blob.

In addition, consider the following scenario of two users making concurrent changes to the same calibre library.

User 1 changes the author name of an entry. This updates the db for user 2. calibre does not rescan the db everytime the file changes, so User 2's calibre instance will continue to happily use its cached data. Now suppose user2 also decides to edit the same author name. calibre2 will use the old author name before user1's edit and will therefore not be able to find the files fo that book, since the file location depends on the author name. It will therefore think that entry has no files and create a new folder for that book with no files in it.

Now when dropbox syncs, both users will have lost the files of that entry, which will be in an orphaned folder based on the author name user 1 chose.

I can think of lots of problems like this caused by concurrent edits. Really the only safe way to support concurrent editing is with row level locking, the way it is done in databases.

So please dont use multiple calibre instances on the same dropbox share concurrently, or I am going to be dealing with a lot of reports about missing files
kovidgoyal is offline   Reply With Quote
Old 03-29-2010, 12:57 PM   #47
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,703
Karma: 6658935
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by kovidgoyal View Post
Interesting (and slightly horrifying) use for dropbox, why don't you use a version control system?
Short, common, story.

The project needed to produce a 'deliverable document' in a couple of weeks. Each partner was in charge of a section. One of the project members set up a partitioned document structure using latex sub-documents, put this into dropbox to simplify sharing the parts, and announced its existence to the partners. It worked quite well. So far, so good.

As days went on, the document structure started to morph. People found that 'their part' was actually distributed through the document. Clearly there would be simultaneous editing, but "it wouldn't be too much and anyway, who has time to set up a repository? We must finish soon!" This didn't work as well, but we could handle the conflicts.

Then someone decided that some code was needed to generate some of the figures, and as a side effect do some formal analysis and proofs on parts of the model. Great idea. Bang into dropbox it went along with a logic engine and a bunch of other stuff, and bang went the collisions. However, we were 2 days from deadline, so we lived with it.

This story is a classic example of why 'we' should think about what we are doing before charging off to use some shiny new doodad that seems to solve the problem. Although dropbox is trivially easy to use, distributed repositories are also easy to use and are more appropriate for the task. A subversion repository coupled with tortoiseSVN would have neatly solved the problem, and is arguably just as easy to use especially in this context.

Sigh.
chaley is offline   Reply With Quote
Old 03-29-2010, 01:02 PM   #48
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by daffy4u View Post
I think you're assuming that I live in Calibre. It is not something I use daily.
I'm not really assuming anything about you. I just hoped people would use care with setting up dropbox, regardless of how often they use Calibre.
For those who use it very seldom, the risk of concurrent use and conflicts seems to me to be very, very, low. That's one of those factors to be considered when weighing convenience against risk.
Starson17 is offline   Reply With Quote
Old 03-29-2010, 10:21 PM   #49
Sydney's Mom
Wizard
Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.Sydney's Mom ought to be getting tired of karma fortunes by now.
 
Sydney's Mom's Avatar
 
Posts: 2,895
Karma: 6995721
Join Date: Dec 2008
Location: Idaho, on the side of a mountain
Device: Kindle Oasis, Fire 3d Gen and 5th Gen and Samsung Tab S
Quote:
Originally Posted by kovidgoyal View Post
So please dont use multiple calibre instances on the same dropbox share concurrently, or I am going to be dealing with a lot of reports about missing files
Well, I had missing files tonight, but I knew it was dropbox, and not Calibre, so I didn't bother you, Kovid.

I did just what everyone said not to do. I had uplifted everything to the clouds, and about 80% was downloaded, and I just couldn't resist peeking at my netbook. My laptop was shut down, as was my desktop. Apparently, the metadata file hadn't downloaded yet, and so my library showed no books. Oh well, I thought, when it is 100% done, they will be there. Well, when I got home from work, dropbox was done syncing, but my laptop still showed XXX books, and my netbook showed 0. I was about to uninstall Calibre (the only thing I could think of was to reinstall), but chaley warned me about creating a metadata file by opening calibre on my netbook. Sure enough, when I opened the dropbox file, there was a metadata file with 75 kb, and a metadata conflicted file with 4000 kb. I deleted the smaller, and renamed the larger, and I was in business! Then to the desktop. Same issue (but all other Calibre libraries were closed). I opened the dropbox file on the desktop, and saw the 75 kb metadata file, but no larger file. I went to my file in the clouds, and downloaded and saved the larger file to my desktop. It worked! So Kovid, I hope I can keep using dropbox. I promise, if I lose my books again, I won't blame Calibre. I have a noncloud copy on an SD card and a portable hard drive. I should be okay. But I like to go into Calibre from a couple different computers. It is great not to have to email the book back and forth. And I finally feel I have enough backups (SD, HD, Dropbox, laptop, netbook, desktop). Sectuple rendundncy?
Sydney's Mom is offline   Reply With Quote
Old 03-29-2010, 10:42 PM   #50
ficbot
Wizard
ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.ficbot ought to be getting tired of karma fortunes by now.
 
Posts: 2,409
Karma: 4132096
Join Date: Sep 2008
Device: Kindle Paperwhite/iOS Kindle App
I am only going to use it on my Macbook. If I want to access my documents from work (it is unlikely I would need to access my ebooks there) I will log in through their website.
ficbot is offline   Reply With Quote
Old 03-29-2010, 10:47 PM   #51
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by Starson17 View Post
Make a separate backup of a good and complete copy of the Calibre library, or at least the metadata.db file, which is the most important file in Calibre.
I use Dropbox for syncing and I think you have been correct in your assumptions of possible pitfalls. I've used it now over a year and I have found most of the problems have been inflicted by me.

I have two backup strategies.

1. Log into the kids computer under my account and let Dropbox sync the database. When it is finished I log off. I never use Calibre on this machine and Dropbox does not automatically start up. Setting this up this way protects me from my own errors which are far worse then any hypothetical Dropbox conflict you can come up with.

2. The backup strategy I use outside of Dropbox doesn't involve backing up the metadata.db file at all.
Update: The below method will not backup custom recipes, they are stored in the metadata.db file. I save all of my custom recipes separate from Calibre but I wanted folks to know the limitation of this backup option.
  • I use the Save to Disk option in Calibre (ensuring both Save cover separately and Save metadata in OPF file are checked) to save my books in a backup directory.
  • Every time I add/update any books I select them and use Save to Disk to save them to my backup directory.
  • If I ever need to restore from a catastrophic failure I just reinstall Calibre and point it to a new Library folder and import the books. All the metadata, covers, tags, series etc... are recreated perfectly.
I have actually restored using 2 above a couple of times to test this setup and my library has always past the disaster recovery test.

Everyone here needs to know that a backup isn't a backup unless it can be restored. Far too many times folks thought they had a backup strategy only to find out is wasn't as good as they thought when they actually had to restore data.

Confession, like most other paranoids in this thread I do drag my library to an external drive about once a week for good measure and all of my source files are far away from Cailbre and in multiple locations in case of operator hardware error.

Last edited by DoctorOhh; 03-30-2010 at 12:59 AM.
DoctorOhh is offline   Reply With Quote
Old 03-29-2010, 11:33 PM   #52
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@sydney's Mom: Feel free to continue using drop box, but just be aware of the pitfalls
kovidgoyal is offline   Reply With Quote
Old 03-30-2010, 12:22 AM   #53
SirBC
Connoisseur
SirBC is on a distinguished road
 
Posts: 56
Karma: 64
Join Date: Nov 2007
Device: Kindle
Quote:
Originally Posted by kovidgoyal View Post
I can think of lots of problems like this caused by concurrent edits. Really the only safe way to support concurrent editing is with row level locking, the way it is done in databases.
Is it possible to have Calibre run in a "read only" mode, so that it didn't attempt to write to the DB? I guess I'm thinking of a scenario where, say, one family member is the one who maintains updating the ebooks in Calibre while the other family members just browse.
SirBC is offline   Reply With Quote
Old 03-30-2010, 12:52 AM   #54
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's what the content server is for.
kovidgoyal is offline   Reply With Quote
Old 03-30-2010, 10:12 AM   #55
SirBC
Connoisseur
SirBC is on a distinguished road
 
Posts: 56
Karma: 64
Join Date: Nov 2007
Device: Kindle
I haven't yet looked into the content server but I will. Do the people viewing via the server have the same full functionality as within Calibre? I keep my Calibre DB on my NAS, so other family members can access even when my computer is off. Would that still be true with the content server?
SirBC is offline   Reply With Quote
Old 03-30-2010, 10:34 AM   #56
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by SirBC View Post
Do the people viewing via the server have the same full functionality as within Calibre?
No. The content server provides access to the books via the web. You can search and download, but not change metadata or add/delete books.
Starson17 is offline   Reply With Quote
Old 03-30-2010, 10:45 AM   #57
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,703
Karma: 6658935
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by SirBC View Post
I keep my Calibre DB on my NAS, so other family members can access even when my computer is off. Would that still be true with the content server?
Note that a computer is required to run the content server. Unless your NAS server can run Calibre (perhaps it is a linux box?), you would need to leave the calibre server running on a NAS client.
chaley is offline   Reply With Quote
Old 03-30-2010, 11:00 AM   #58
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by SirBC View Post
I keep my Calibre DB on my NAS, so other family members can access even when my computer is off.
How do you prevent two people from using it at the same time?
Starson17 is offline   Reply With Quote
Old 03-30-2010, 04:32 PM   #59
SirBC
Connoisseur
SirBC is on a distinguished road
 
Posts: 56
Karma: 64
Join Date: Nov 2007
Device: Kindle
Quote:
Originally Posted by Starson17 View Post
How do you prevent two people from using it at the same time?
Currently I don't. That is why I was asking if it was possible to have Calibre start in a "read only" mode so that my other family members could use Calibre to access the DB when my computer is off, as well as do things like use tags to filter books. It sounds like the content server would kind of work, but we would have issues with limited functionality (if I am correctly understanding how content server works) as well as not being able to access the content server when my computer is off. It's not that a big a deal, I was just wondering if there was a startup command to run Calibre in read only mode to avoid possibly corrupting the database.
SirBC is offline   Reply With Quote
Old 03-30-2010, 09:39 PM   #60
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by SirBC View Post
Currently I don't. That is why I was asking if it was possible to have Calibre start in a "read only" mode so that my other family members could use Calibre to access the DB when my computer is off,
Currently it's not possible to start in a read only mode.

Code:
 as well as do things like use tags to filter books.
You can do this through the content server.

Quote:
It sounds like the content server would kind of work, but we would have issues with limited functionality (if I am correctly understanding how content server works)
What functions do you want? The content server can get all books/news and you can search by tags, author, title, etc. to find the content.

Quote:
as well as not being able to access the content server when my computer is off.
This is correct. OTOH, you gain the advantage of access anywhere in the world where there is internet access and access to your books via smartphone and other devices.

Quote:
It's not that a big a deal, I was just wondering if there was a startup command to run Calibre in read only mode to avoid possibly corrupting the database.
You could write a batch file that checks for the existence of a flag file, exits if it exists, creates it if it does not, then deletes it when you exit Calibre. If everyone runs it to start Calibre, they'd avoid stepping on each other.
Starson17 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiple computers one db? giddion Calibre 15 05-20-2011 08:11 PM
Syncing across multiple computers Sridhar Calibre 4 07-10-2010 11:27 AM
Sync Library on two computers BrianG Calibre 8 01-04-2010 06:59 PM
Duplicate books - multiple formats mranlett Calibre 5 09-26-2009 07:02 AM
Multiple computers and Calibre starrigger Calibre 11 03-19-2009 12:14 PM


All times are GMT -4. The time now is 07:32 PM.


MobileRead.com is a privately owned, operated and funded community.