content server caching delivery - not delivering latest edit- fresh evidnce

stumped · 01-29-2019, 07:58 AM

a few weeks ago I asserted this ws happening, and got shot down by Kovid who says dotes not happen - cannot happen
at tat time I was unsure whether to blame content server, calibre companion or my device, but here is fresh evidence that it is Content server which is caching
sequence
1. added a book to calibre - delivered it via content server and calibre companion to tablet 1.
2. then noticied that font size needed tweatking on one place, so I fixed that with the editor, then decided I y should not bother correcting it on tablet 1, but instead, I would send the revised version to tablet 2. So, picked up tablet 2 which was on overnight charge connected that to content server and got the book, only to find that I had been served the previous version with the wrong font size. THAT CANNOT BE BLAMED ON DEVICE OR ON CC APP as the tablet 2 target device was OFF until AFTER i fixed up the book and only then asked for it from the content server. the book had never been on that device before.

4. rechecking the book in calibre confirms that it HAS been edited, and the font size fixed.

so the content server had served up a cached version to tablet 2

now how do I convince Kovid to at least take a look at that & check it out ?

for the record I am running calibre v.3.3.7 in windows 10 latest 1809 update. PC is using wired ethernet to my LAN

Steps to reproduce the bug
1 request a book from content server & get it onto a reader
ANY EBUB WILL DO
2 then edit that book. Make some obvious easy to find alteration
3, then delete it from target device and request it again ( without shutting s down calibre. )the unedited version will be re-delivered
now shut down and restart calibre, delete the book from device and request it a 3rd time. the CS will now deliver the latest version

PS I know from previous testing that I can clear this "non existent" cache by shutting down calibre and restarting it. BUT I am seeking some acknowledgement and confirmation that the existing content server behavior is bugged and cannot be blamed on windows or, android, or Calibre

theducks · 01-29-2019, 10:48 AM

The content server is a Web host.
What were you using as the CLIENT? That might be the cache.

What NETWORKS (LAN only, your host end ISP1, client end ISP2) were between the Content server and the client

stumped · 01-29-2019, 10:59 AM

i explained how the client cannot possibly be the cache
i will walk you though it again
i retrieved a book from content server to calibre companion on tablet A
i spotted an error when reading so I edited the book back on the PC, using sicgil, thus updating the copy in calibre library - on the C drive , in default location
then I took tablet B, which was OFF, turned it ON, and used it to get the same book via the same connection. NB this table was off until AFTER i had edited the book, and has never had the book on it.
that tablet - call it tablet B received the old - un-edited copy from the content server
so it cannot be the client - the client was never on until after the book had been edited

I observed this with other books , some weeks ago, and posted about it then, but I got a blanket denial that the content server does not and cannot cache

so i set out steps to reproduce the problem in my previous post,
someone needs to take those and prove or disprove my claim on other equipment
it is a simple test which will only take a couple of minutes.

meanwhile I confirmed AGAIN, that if i shut down calibre and remove book from tablets A& B, then re-acquire it, I then get the updated version

[ last time around, I went even further, I did another edit, removed book from devices, re-acquired it a 3rd time, and did NOT get the most recent edit.. to get the most recent edit I have to stop / start calibre- stop /starting apps / clearing app caches, or even power cycling tablets in between content fetches makes no difference.

clearly ( to me ) content server or related processes are holding stuff in RAM and re-using that to fulfill requests. [ until such time as either calibre is stopped/ started, or cache is overwritten by other requests from server .)
I searched calibre app data and saw nothing that could be a cached book

ps you asked about network
the complete path is :PC with calibre library is wired to a GB switch, which is wired to my router
the tablets use wifi to reach an access point [ asus R56U ]which also wired to the same router[ virginmedia hub]
so the tablet app - calibre companion - sees the content server at local lan address 192.168.0.72 and asks for content to download from there. both tablets do this.

PS yet more confimation. if I mirror calibre library to dropbox then tell CC to get book from dropbox, it always gets the version that was most recently mirrored into dropbox. so I could edit book, sync to dropbox, get from dropbox, and always get the most recent version. it is only when I get from calibre content server , without stopping calibre, that i get previous cached versions

stumped · 01-29-2019, 11:15 AM

the even shoerter version

tablet A fetches book version A from content server
book is then changed (edited) to version B. so version A s no longer exists as a file on disc, anywhere on my LAN. calibre library has version B only.

tablet B is turned on and now asks for same book from same content server. it receives book version A. which by now exists only in RAM ???. tablet B has never before had the book.
calibre is now shut down and restarted, and tablet B then asks again for that book. it now gets version B of the book.

that is surely something that others can test. there's probably no need for tablets or other apps in the loop.
just get from content server, edit source , get again, see what is delivered 2nd time around.

stumped · 01-29-2019, 11:42 AM

the even shorter version:
tablet B was off when the book being added to calibre, the book being edited from version A to version B
tablet B was then and only then turned on, yet it acquired Book version A from content server.
at no time, in this universe ,was Tablet B switched ON , while book version A existed as a file in my library.

i will refer to it henceforth as Schrodinger's tablet

Cat says it might be valuable but he will not be sitting on it / in it/ near it any time soon

kovidgoyal · 01-29-2019, 11:07 PM

You want to convince me, you need to provide some set of steps that will unambiguously demonstrate the issue, something like:

1) Start calibre
2) Start the content server
3) Open Book 123 in browser
4) Edit book 123 in calibre
5) Open Book 123 in another browser, one that has never connected to the server (use a private profile or clear caches, etc)
6) Book 123 does not have the edits from step (4)

stumped · 01-29-2019, 11:55 PM

Quote:

Originally Posted by kovidgoyal

You want to convince me, you need to provide some set of steps that will unambiguously demonstrate the issue, something like:

1) Start calibre
2) Start the content server
3) Open Book 123 in browser
4) Edit book 123 in calibre
5) Open Book 123 in another browser, one that has never connected to the server (use a private profile or clear caches, etc)
6) Book 123 does not have the edits from step (4)

already done . read my previous posts and do the steps to reproduce. they are not identical to the above but are fit for purpose.
copy:
Steps to reproduce the bug
1 request a book from content server & get it onto a reader
ANY EBUB WILL DO
2 then edit that book [ back in calibre library]. Make some obvious easy to find alteration
3, then delete it from target device and request it again ( without shutting down calibre. )the unedited version will be re-delivered
now shut down and restart calibre, delete the book from device and request it a 3rd time. the CS will now deliver the latest version
or GIVE AN ALTERNATIVE EXPANATION FOR THE "SCHRODINGERS TABLET" outcome, if you want to stay in denial.

i know what i exerinced: a book (version A) that no longer existed as a file in the library was delivered to a device that had been previously OFF, OFF since before the book was placed in the library, and had remained off while the book was edited from version A to version B. I await your non-magical explanation of how that could have happened.

kovidgoyal · 01-30-2019, 12:11 AM

You appear to be extremely naive when it comes to how networks operate. You want a non-magical explanation, here is the most obvious one:

There exists a caching proxy between the software on your device and the content server.

And just FYI if you want to convince people of something, being rude is not going to get the job done.

stumped · 01-30-2019, 12:36 AM

Quote:

Originally Posted by kovidgoyal

You appear to be extremely naive when it comes to how networks operate. You want a non-magical explanation, here is the most obvious one:

There exists a caching proxy between the software on your device and the content server.

that would be a time travelling, space warping proxy on the device then, *capable of operating when device is off, and capable of retrieving files from some multi-verse.?
I prefer my explanaion, but if no one cares enough to test it., it can stay as a bug. I can work around it and will not ask you to waste your valuable time investigating....
* a book (version A) that no longer existed as a file in the library was delivered to a device that had been previously OFF, OFF since before the book was placed in the library, and had remained off while the book was edited from version A to version B

ilovejedd · 01-30-2019, 01:01 AM

Quote:

Originally Posted by stumped

that would be a time travelling, space warping proxy on the device then, *capable of operating when device is off, and capable of retrieving files from some multi-verse.?
I prefer my explanaion, but if no one cares enough to test it., it can stay as a bug. I can work around it and will not ask you to waste your valuable time investigating....
* a book (version A) that no longer existed as a file in the library was delivered to a device that had been previously OFF, OFF since before the book was placed in the library, and had remained off while the book was edited from version A to version B

Not in the library but data may have been cached in the router or switch so it doesn't really matter if device 2 was turned off at the time of caching. No warping of the space-time continuum necessary.

Only way to confirm if Calibre is serving stale data is by eliminating the middle man (that includes router/switch) and using a web browser installed on the same computer that's running the Calibre content server to download.

stumped · 01-30-2019, 01:12 AM

[QUOTE=ilovejedd;3804572]Not in the library but data may have been cached in the router or switch so it doesn't really matter if device 2 was turned off at the time of caching. No warping of the space-time continuum necessary.

/QUOTE]

Expect for the wnnoying factoid that stopping and restarting calibre but not stopping and restarting anything else, changes the outcome. So in your scenario the router needs some quantum entanglement ability to detect that a pc app has stopped and started... And wipe it's own cache accordingly

I still assert that it is bugged, but no one cares.
It could be independently tested in less time than it takes to post further arguments

ilovejedd · 01-30-2019, 01:39 AM

Quote:

Originally Posted by stumped

Expect for the wnnoying factoid that stopping and restarting calibre but not stopping and restarting anything else, changes the outcome. So in your scenario the router needs some quantum entanglement ability to detect that a pc app has stopped and started... And wipe it's own cache accordingly

I still assert that it is bugged, but no one cares.
It could be independently tested in less time than it takes to post further arguments

I've already tested it (or more like I have to deal with caching on a semi-regular basis due to fanfics with consecutive chapter updates). From my tests, it was never the Calibre server itself that was doing the caching. It was on-device caching on iOS which no amount of stopping and restarting of the Calibre server or device ever fixed. I have to either download and read the updated ebook on another device, use Dropbox to transfer, or just wait for the cache to expire eventually.

For me, using a different device always worked. I somehow doubt Calibre has explicit code that sees Android and decides to serve it an old version of a book just for kicks.

stumped · 01-30-2019, 01:57 AM

Quote:

Originally Posted by ilovejedd

I've already tested it (or more like I have to deal with caching on a semi-regular basis due to fanfics with consecutive chapter updates). From my tests, it was never the Calibre server itself that was doing the caching. It was on-device caching on iOS which no amount of stopping and restarting of the Calibre server or device ever fixed. I have to either download and read the updated ebook on another device, use Dropbox to transfer, or just wait for the cache to expire eventually.

For me, using a different device always worked. I somehow doubt Calibre has explicit code that sees Android and decides to serve it an old version of a book just for kicks.

But in this case, as I have repeatedly explained. the device was OFF from before the library entry was created until after it had been edited. So it is 100% impossible for the device to have accessed and cached the old version. That version while only in existence while the device was off.

The router and switch do not cache either. In fact I doubt the signal path includes the router, because the server pc and the wifi access point link to the same dumb GB switch. So they can communicate via lan without going through the router.

So it's either happenng in windows 10, or in the content server itself. And as stopping and restarting content server changes the outcome, it is much more likely to be the content server. Android is irrelevant my theory is that the server is seeing a request for the same file name that it has just supplied, so it fulfills that request from ram

kovidgoyal · 01-30-2019, 03:48 AM

I just tried the set of steps I posted, and could not replicate. Which conclusively demonstrates that the calibre server is serving updated books after editing them sing the editor, without needing to restart calibre.

stumped · 01-30-2019, 04:08 AM

thanks for testing
I am at a loss now for how to try to isolate a cause

were your tests done with content server running in windows 10 or does that not matter.
also, FWIW my edit was done using sigil to read from , and replace the book into, calibre library.
i looked also throughout app data for server logs but did not find any simple - this book was requested and delivered... info.

i could create a test book and go through my steps one more time, but use calibre editor in place of sigil - it is tedious though. i'd be more willing to do that if there's a way to tun on some verbose logging beforehand...

01-29-2019, 07:58 AM	#1
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	content server caching delivery - not delivering latest edit- fresh evidnce a few weeks ago I asserted this ws happening, and got shot down by Kovid who says dotes not happen - cannot happen at tat time I was unsure whether to blame content server, calibre companion or my device, but here is fresh evidence that it is Content server which is caching sequence 1. added a book to calibre - delivered it via content server and calibre companion to tablet 1. 2. then noticied that font size needed tweatking on one place, so I fixed that with the editor, then decided I y should not bother correcting it on tablet 1, but instead, I would send the revised version to tablet 2. So, picked up tablet 2 which was on overnight charge connected that to content server and got the book, only to find that I had been served the previous version with the wrong font size. THAT CANNOT BE BLAMED ON DEVICE OR ON CC APP as the tablet 2 target device was OFF until AFTER i fixed up the book and only then asked for it from the content server. the book had never been on that device before. 4. rechecking the book in calibre confirms that it HAS been edited, and the font size fixed. so the content server had served up a cached version to tablet 2 now how do I convince Kovid to at least take a look at that & check it out ? for the record I am running calibre v.3.3.7 in windows 10 latest 1809 update. PC is using wired ethernet to my LAN Steps to reproduce the bug 1 request a book from content server & get it onto a reader ANY EBUB WILL DO 2 then edit that book. Make some obvious easy to find alteration 3, then delete it from target device and request it again ( without shutting s down calibre. )the unedited version will be re-delivered now shut down and restart calibre, delete the book from device and request it a 3rd time. the CS will now deliver the latest version PS I know from previous testing that I can clear this "non existent" cache by shutting down calibre and restarting it. BUT I am seeking some acknowledgement and confirmation that the existing content server behavior is bugged and cannot be blamed on windows or, android, or Calibre Last edited by stumped; 01-29-2019 at 08:01 AM.

01-29-2019, 10:59 AM	#3
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	i explained how the client cannot possibly be the cache i will walk you though it again i retrieved a book from content server to calibre companion on tablet A i spotted an error when reading so I edited the book back on the PC, using sicgil, thus updating the copy in calibre library - on the C drive , in default location then I took tablet B, which was OFF, turned it ON, and used it to get the same book via the same connection. NB this table was off until AFTER i had edited the book, and has never had the book on it. that tablet - call it tablet B received the old - un-edited copy from the content server so it cannot be the client - the client was never on until after the book had been edited I observed this with other books , some weeks ago, and posted about it then, but I got a blanket denial that the content server does not and cannot cache so i set out steps to reproduce the problem in my previous post, someone needs to take those and prove or disprove my claim on other equipment it is a simple test which will only take a couple of minutes. meanwhile I confirmed AGAIN, that if i shut down calibre and remove book from tablets A& B, then re-acquire it, I then get the updated version [ last time around, I went even further, I did another edit, removed book from devices, re-acquired it a 3rd time, and did NOT get the most recent edit.. to get the most recent edit I have to stop / start calibre- stop /starting apps / clearing app caches, or even power cycling tablets in between content fetches makes no difference. clearly ( to me ) content server or related processes are holding stuff in RAM and re-using that to fulfill requests. [ until such time as either calibre is stopped/ started, or cache is overwritten by other requests from server .) I searched calibre app data and saw nothing that could be a cached book ps you asked about network the complete path is :PC with calibre library is wired to a GB switch, which is wired to my router the tablets use wifi to reach an access point [ asus R56U ]which also wired to the same router[ virginmedia hub] so the tablet app - calibre companion - sees the content server at local lan address 192.168.0.72 and asks for content to download from there. both tablets do this. PS yet more confimation. if I mirror calibre library to dropbox then tell CC to get book from dropbox, it always gets the version that was most recently mirrored into dropbox. so I could edit book, sync to dropbox, get from dropbox, and always get the most recent version. it is only when I get from calibre content server , without stopping calibre, that i get previous cached versions Last edited by stumped; 01-29-2019 at 11:07 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
calibredb does not work with content server running (content server is using --enable	Irvin	Server	3	01-08-2019 02:47 AM
Content (book) delivery question	atodorovic	General Discussions	10	01-30-2014 11:50 AM
Content Server edit help	selfdestroyer	Calibre	2	08-06-2010 04:47 AM
Why wireless only for content delivery?	wayner	Amazon Kindle	12	11-26-2009 04:32 PM
iLiad Web2iDS for undocumented communication with the iRex Delivery Server	Alexander Turcic	iRex Developer's Corner	22	11-24-2008 09:20 AM

01-29-2019, 10:48 AM	#2
theducks Well trained by Cats Posts: 29,891 Karma: 55267620 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	The content server is a Web host. What were you using as the CLIENT? That might be the cache. What NETWORKS (LAN only, your host end ISP1, client end ISP2) were between the Content server and the client

01-29-2019, 11:15 AM	#4
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	the even shoerter version tablet A fetches book version A from content server book is then changed (edited) to version B. so version A s no longer exists as a file on disc, anywhere on my LAN. calibre library has version B only. tablet B is turned on and now asks for same book from same content server. it receives book version A. which by now exists only in RAM ???. tablet B has never before had the book. calibre is now shut down and restarted, and tablet B then asks again for that book. it now gets version B of the book. that is surely something that others can test. there's probably no need for tablets or other apps in the loop. just get from content server, edit source , get again, see what is delivered 2nd time around.

01-29-2019, 11:42 AM	#5
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	the even shorter version: tablet B was off when the book being added to calibre, the book being edited from version A to version B tablet B was then and only then turned on, yet it acquired Book version A from content server. at no time, in this universe ,was Tablet B switched ON , while book version A existed as a file in my library. i will refer to it henceforth as Schrodinger's tablet Cat says it might be valuable but he will not be sitting on it / in it/ near it any time soon

01-29-2019, 11:07 PM	#6
kovidgoyal creator of calibre Posts: 43,933 Karma: 22669822 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You want to convince me, you need to provide some set of steps that will unambiguously demonstrate the issue, something like: 1) Start calibre 2) Start the content server 3) Open Book 123 in browser 4) Edit book 123 in calibre 5) Open Book 123 in another browser, one that has never connected to the server (use a private profile or clear caches, etc) 6) Book 123 does not have the edits from step (4)

01-30-2019, 12:11 AM	#8
kovidgoyal creator of calibre Posts: 43,933 Karma: 22669822 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You appear to be extremely naive when it comes to how networks operate. You want a non-magical explanation, here is the most obvious one: There exists a caching proxy between the software on your device and the content server. And just FYI if you want to convince people of something, being rude is not going to get the job done.

01-30-2019, 01:12 AM	#11
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	[QUOTE=ilovejedd;3804572]Not in the library but data may have been cached in the router or switch so it doesn't really matter if device 2 was turned off at the time of caching. No warping of the space-time continuum necessary. /QUOTE] Expect for the wnnoying factoid that stopping and restarting calibre but not stopping and restarting anything else, changes the outcome. So in your scenario the router needs some quantum entanglement ability to detect that a pc app has stopped and started... And wipe it's own cache accordingly I still assert that it is bugged, but no one cares. It could be independently tested in less time than it takes to post further arguments

01-30-2019, 03:48 AM	#14
kovidgoyal creator of calibre Posts: 43,933 Karma: 22669822 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I just tried the set of steps I posted, and could not replicate. Which conclusively demonstrates that the calibre server is serving updated books after editing them sing the editor, without needing to restart calibre.

01-30-2019, 04:08 AM	#15
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	thanks for testing I am at a loss now for how to try to isolate a cause were your tests done with content server running in windows 10 or does that not matter. also, FWIW my edit was done using sigil to read from , and replace the book into, calibre library. i looked also throughout app data for server logs but did not find any simple - this book was requested and delivered... info. i could create a test book and go through my steps one more time, but use calibre editor in place of sigil - it is tedious though. i'd be more willing to do that if there's a way to tun on some verbose logging beforehand...

Advert

Advert