Request for opinions on setting up read info syncing

chaley · 08-03-2014, 05:59 AM

The problem that Library Addict reported has raised some serious issues with the "first time" process after read information lookup names are entered. I have a way to fix it robustly, but there are side effects. Your comments are welcome. And apologies for the length of this post.

Currently, the first connect after a read info lookup name is entered into CC is special. The rules for this special sync are:

If there is a value in CC, then that value is copied to calibre,
otherwise if there is a value in calibre, that value is copied to CC,
otherwise do nothing.

On any subsequent connect, the "normal" rules apply:

If the value in CC differs from the value last received by CC from calibre then the CC value is copied to calibre
otherwise if the currrent value in calibre is copied to CC

The intent of the special sync was to try to recover any information that the user might have entered into CC's read fields while ensuring that CC's fields were properly initialized. A side effect is that CC is not required to scan books because that will happen "naturally" as part of the metadata sync process.

The problem is that this special processing must be done on the first connect to every library, not simply the first connect to some library. If the special processing is not done then there is a high probability that when connecting to a subsequent library, books that come from that library will have their calibre read information erased. This is what happened to Library Addict.

There are two ways to fix this problem.

The first way is for CC to keep track of whether it has connected to a library after the read info lookup names have changed, and to use the special processing if it is the first connect. This method respects the intent expressed above, but can lead to strange behavior. For example, if a person connects to a backup library two months later then s/he probably doesn't expect to have CC's read information take precedence over what is in the backup library, which is what will happen under this scheme. It is also possible for a calibre library to change its unique ID (think backup/restore), which would trigger special processing on the next CC connect.

The second is to change CC so that when a lookup name is entered the "special processing" is done locally by CC. If the CC value is empty then the calibre value is copied to the CC value, otherwise the CC value is left alone. On the next connect to any library the "standard" sync scheme is used, which is possible because the CC values have been properly initialized. This second scheme has the advantage of having predictable behavior no matter how many libraries are used or whether the library changed identities. It simplifies the connection process because there is no longer any need for calibre to do the special processing. The downside is that a dreaded database upgrade would be required if the lookup names are changed. This is the same processing that currently happens when you enter a lookup name into the "Make groups ..." box.

My feeling is that the advantages of the second scheme are worth requiring the DB upgrade process. I am a fan of predictability. Changing one of the lookup names should be a rare process. The DB upgrade process runs at around 2 to 3 books per second, so people with smaller libraries won't pay much. Unfortunately, for large libraries the process will take significant time.

Comments?

library addict · 08-03-2014, 07:38 AM

I think the second option is better as well.

Changing the lookup name would be rare as you said. And it seems to have less chance for side affects.

library addict · 08-03-2014, 09:21 AM

Oops wrong thread

PatNY · 08-03-2014, 11:30 AM

Hello Charles,

I much prefer the option that has the fewest unpredictable results. I've changed lookup names maybe once or twice over the last 4 years, so this wouldn't be a major problem. Long sync times are not a problem with me either, as long as I know it's a normal thing -- and not a freeze. So if you go with the second option, you may simply want to make sure all the instances that could cause long sync times are listed in your FAQ section.

--Pat

chaley · 08-03-2014, 12:17 PM

Quote:

Originally Posted by library addict

I think the second option is better as well.

Changing the lookup name would be rare as you said. And it seems to have less chance for side affects.

Quote:

Originally Posted by PatNY

Hello Charles,

I much prefer the option that has the fewest unpredictable results. I've changed lookup names maybe once or twice over the last 4 years, so this wouldn't be a major problem. Long sync times are not a problem with me either, as long as I know it's a normal thing -- and not a freeze. So if you go with the second option, you may simply want to make sure all the instances that could cause long sync times are listed in your FAQ section.

--Pat

Thanks to both of you for your opinions. I am leaning more strongly to the second option. One reason is that I figured out a way to make the DB upgrade run much faster in this case, around 8 or 9 books per second.

@Pat: the problem isn't sync times. Those won't change. The problem is that when you set or change a lookup name, CC must change the database. This process can be long, and while it is going on the user cannot use the grouping drawer or connect to calibre. If you have 4000 books in your library the process will take around 10 minutes.

If I go down this road then I will change the dialogs where you enter the information to be clear that tapping OK will trigger the upgrade, permitting the user to choose a convenient time to do it.

theducks · 08-03-2014, 05:28 PM

I just got the GP update and was worried, because CC icon disappeared from my desktop.

The DB update ran impressively fast (I did not time it, but I think it bested your estimates)

OTOH my first synch to Calibre is crawling at about 1% per minute.

chaley · 08-03-2014, 06:33 PM

Quote:

Originally Posted by theducks

I just got the GP update and was worried, because CC icon disappeared from my desktop.

That seems to happen for absolutely no discernible reason.

Quote:

The DB update ran impressively fast (I did not time it, but I think it bested your estimates)

OTOH my first synch to Calibre is crawling at about 1% per minute.

There isn't anything in the update that I can think of that should have affected the sync time. Did you change the cover size setting or any other setting? It might help me understand what is taking the time if you send me a CC debug log. Send it only to the charles dot haleys address, not the multipie address. There is no reason to bring my son into this discussion.

theducks · 08-03-2014, 07:28 PM

Quote:

Originally Posted by chaley

That seems to happen for absolutely no discernible reason.

There isn't anything in the update that I can think of that should have affected the sync time. Did you change the cover size setting or any other setting? It might help me understand what is taking the time if you send me a CC debug log. Send it only to the charles dot haleys address, not the multipie address. There is no reason to bring my son into this discussion.

I did something that touched every book in Calibre since last connect, that is why the time is long.
every book in CC is getting new metadata

08-03-2014, 05:59 AM	#1
chaley Grand Sorcerer Posts: 12,443 Karma: 8012886 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Request for opinions on setting up read info syncing The problem that Library Addict reported has raised some serious issues with the "first time" process after read information lookup names are entered. I have a way to fix it robustly, but there are side effects. Your comments are welcome. And apologies for the length of this post. Currently, the first connect after a read info lookup name is entered into CC is special. The rules for this special sync are: If there is a value in CC, then that value is copied to calibre, otherwise if there is a value in calibre, that value is copied to CC, otherwise do nothing. On any subsequent connect, the "normal" rules apply: If the value in CC differs from the value last received by CC from calibre then the CC value is copied to calibre otherwise if the currrent value in calibre is copied to CC The intent of the special sync was to try to recover any information that the user might have entered into CC's read fields while ensuring that CC's fields were properly initialized. A side effect is that CC is not required to scan books because that will happen "naturally" as part of the metadata sync process. The problem is that this special processing must be done on the first connect to every library, not simply the first connect to some library. If the special processing is not done then there is a high probability that when connecting to a subsequent library, books that come from that library will have their calibre read information erased. This is what happened to Library Addict. There are two ways to fix this problem. The first way is for CC to keep track of whether it has connected to a library after the read info lookup names have changed, and to use the special processing if it is the first connect. This method respects the intent expressed above, but can lead to strange behavior. For example, if a person connects to a backup library two months later then s/he probably doesn't expect to have CC's read information take precedence over what is in the backup library, which is what will happen under this scheme. It is also possible for a calibre library to change its unique ID (think backup/restore), which would trigger special processing on the next CC connect. The second is to change CC so that when a lookup name is entered the "special processing" is done locally by CC. If the CC value is empty then the calibre value is copied to the CC value, otherwise the CC value is left alone. On the next connect to any library the "standard" sync scheme is used, which is possible because the CC values have been properly initialized. This second scheme has the advantage of having predictable behavior no matter how many libraries are used or whether the library changed identities. It simplifies the connection process because there is no longer any need for calibre to do the special processing. The downside is that a dreaded database upgrade would be required if the lookup names are changed. This is the same processing that currently happens when you enter a lookup name into the "Make groups ..." box. My feeling is that the advantages of the second scheme are worth requiring the DB upgrade process. I am a fan of predictability. Changing one of the lookup names should be a rare process. The DB upgrade process runs at around 2 to 3 books per second, so people with smaller libraries won't pay much. Unfortunately, for large libraries the process will take significant time. Comments?

08-03-2014, 09:21 AM	#3
library addict Grand Sorcerer Posts: 13,115 Karma: 37197878 Join Date: Nov 2010 Location: USA Device: Nexus 7 2, Samsung Galaxy Tab A 8" (retired=Sony PRS650, Kobo Mini)	Oops wrong thread Last edited by library addict; 08-03-2014 at 09:47 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Rendering Targets. Awesome. Request for info	twobob	Kindle Developer's Corner	9	09-13-2013 06:02 PM
Info request	jbcohen	Amazon Kindle	1	10-05-2012 10:14 AM
Request for Nook screensaver info, please	jackie_w	Plugins	5	09-18-2011 07:58 AM
Feature Request For Exporting DB Info	RoninTech	Calibre	12	10-13-2008 06:20 PM
Hello and a request for opinions T2010 vs Iliad	zeigerpuppy	Introduce Yourself	4	11-29-2007 10:11 AM

08-03-2014, 07:38 AM	#2
library addict Grand Sorcerer Posts: 13,115 Karma: 37197878 Join Date: Nov 2010 Location: USA Device: Nexus 7 2, Samsung Galaxy Tab A 8" (retired=Sony PRS650, Kobo Mini)	I think the second option is better as well. Changing the lookup name would be rare as you said. And it seems to have less chance for side affects.

08-03-2014, 11:30 AM	#4
PatNY Zennist Posts: 1,022 Karma: 47809468 Join Date: Jul 2010 Device: iPod Touch, Sony PRS-350, Nook HD+ & HD	Hello Charles, I much prefer the option that has the fewest unpredictable results. I've changed lookup names maybe once or twice over the last 4 years, so this wouldn't be a major problem. Long sync times are not a problem with me either, as long as I know it's a normal thing -- and not a freeze. So if you go with the second option, you may simply want to make sure all the instances that could cause long sync times are listed in your FAQ section. --Pat

08-03-2014, 05:28 PM	#6
theducks Well trained by Cats Posts: 31,047 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	I just got the GP update and was worried, because CC icon disappeared from my desktop. The DB update ran impressively fast (I did not time it, but I think it bested your estimates) OTOH my first synch to Calibre is crawling at about 1% per minute.

Advert

Advert