News downloads and scheduled downloads mysteriously disappeared.

[mkgtu]

A mystery. Never had this happen before.

But...this morning only one of my scheduled newsfeed downloads succeeded, and was automatically emailed to kindle. The other two were not on my kindle so I checked the computer to find that it had restarted and was now on the "start" screen waiting for the PIN to be entered. (So calibre was not running)

After letting the PC finish restarting I checked calibre and found that in the "fetch news" section there were NO scheduled downloads, the whole "scheduled" category was gone. in addition, any actually downloaded issues of periodicals prior to yesterday had been deleted from the library. Yesterday's downloads and today's single issue were not deleted. I do have some of those scheduled feeds set to keep only back issue, other s to keep all issues. But basically all the back issues had been deleted.

I don't know why the computer restarted after only one scheduled download. That's usually due to a Windows update, I checked and found there were no updates today (It's Sunday!). I also got a warning from OneDrive (where my calibre library is stored) that 498 files had been deleted from the calibre library folder. It kept those deletions as "pending" and offered to restore them if I wanted to.

Fortunately I managed to repair the damage. With the OneDrive warning about the massive deletion, I was able to reverse the deletion of that batch of files and had OneDrive "restore" them. Interestingly, they were "restored" to the Calibre "trash" folder, so I presume that is where they came from. So all those 498 files had been deleted from the calibre library, presumably by calibre itself, and had been moved to calibre's trash folder, and were then immediately deleted from the trash folder. OneDrive could only have noted files deleted from the trash folder. If calibre removes library files and puts them in trash, OneDrive would consider that "moving" files, not deleting them. and so that would not raise any red flags.

I then used calibre to "restore" the removed files from trash.

I also have several backup programs on my PC and I used one of them find that calibre's "scheduler.xml" file had been deleted this morning a little after 5:00am, about the time of the first scheduled download (successful) of the Economist's Espresso. It must have been at the time the computer restarted itself. I copied that backed up file back to the calibre older in Users/AppData/Roaming. I then restarted calibre and all the "schedules" (all 14) were back in the "fetch news" section.

I have no idea why this all happened and may never know. But at least all the backup services I have worked as they should have and I was able to restore things.

btw this was calibre 6.17

[BetterRed]

Originally Posted by mkgtu

<snip>

I also got a warning from OneDrive (where my calibre library is stored) that 498 files had been deleted from the calibre library folder. It kept those deletions as "pending" and offered to restore them if I wanted to.

<snip>

See ==>> I am getting errors with my calibre library on a networked drive/NAS?

OneDrive is the equivalent of a network server, NAS, etc.

Version 6.16 changed the way deletes are handled see ==>> calibre - What's new

BR

[Quoth]

Short solution: Don't store Calibre Library on OneDrive.

[mkgtu]

Originally Posted by BetterRed

See ==>> I am getting errors with my calibre library on a networked drive/NAS?

OneDrive is the equivalent of a network server, NAS, etc.

Version 6.16 changed the way deletes are handled see ==>> calibre - What's new

BR

I'm not getting notice about any errors with calibre library which is stored locally in the OneDrive folder, and those files are always kept on the local drive and are incidentally also uploaded to the OneDrive cloud.

I doubt that OneDrive folders are the "equivalent" of an NAS server. For all intents and purposes files stored locally in the OneDrive folder function like being stored in any other folder, with the exception that a copy is also uploaded/synced to the OneDrivecloud. The computer only accesses the local file.

In this mysterious situation, to some extent Calibre and OneDrive operated as designed. Most files, except yesterday's, that had "calibre" as the author (all the downloaded news feeds), were removed from the library and put in, ie moved to, calibre's "trash". At that point OneDrive would have noted the "move", and consequently "moved" those files to their corresponding new "locations" in the cloud. This is just kind of a matter of bookkeeping for OneDrive in the cloud, just altering the file table or index. Nothing needed to actually be moved anywhere.

But for some strange reason, its seems that as soon as calibre "moved" the files to the trash folder, they were almost immediately deleted, not from the library, but from the trash folder, even though the trash folder was set to retain files for 14 days. (I've since changed that to 90 days). They shouldn't have been removed from the trash; actually they shouldn't have been removed from the library in the first place. I certainly didn't do anything. I was in bed asleep at the time!

It is at that point that OneDrive sent out an alarm, as it always does when large numbers of files are deleted at one time, because now an actual permanent deletion was happening. So I had OneDrive restore those files. They were restored to their previous location, which was the trash. They were then downloaded (1.4GB) from the cloud to the local drive.

So calibre's "remove to trash" and "restore from trash" (after OneDRive had redownloaded the files from the cloud) operated as designed. EXCEPT that I don't know what triggered them to be removed from the library in the first place, nor can I explain how and why they were the almost immediately deleted from the trash even though the retention was set for 14 days.

I don't know why the computer restarted right after downloading just one news feed; there were two thers scheduled right after that. I don't know why anything got removed and deleted, especially why only "calibre" authored items (and why not all of them; why did it leave the last day or so's worth of "caibre" files - about a dozen).

And why was the "scheduler.xml" which contains all the scheduled downloads info deleted from the calibre AppData folder. That is not stored anywhere near the calibre library folder.

[mkgtu]

Originally Posted by Quoth

Short solution: Don't store Calibre Library on OneDrive.

Sorry, I don't buy that. I've located the Calibre Library in the OneDrive directory (with the setting to always keep the local copy on the local drive) for well over 10 years. Never a problem. Nothing.

And if you'll read carefully, with what happened today OneDrive was not the problem. OneDrive didn't "remove" all the calibre authored newfeeds from the library folder and put them in the calibre trash. And OneDrive didn't immediately delete everything from the calibre trash folder instead of retaining the files for 14 days.

OneDrive worked exactly as it should have. As a matter of fact, if OneDrive had not alerted me the mass deletion of 498 files (1,4GB) I might never have known this had even happened. And it was OneDrive that enable me to easily recover those deleted files. I could also have recovered them with my three other backup programs (Second Copy, Acronis, and Macrium). But none of those programs alerted me to the mass deletion.

The only thing I noticed, before I noticed the OneDrive alert about the deletions, was that calibre failed to download The New York Times and The Washington Post. And when I checked on that I found that the entire "scheduled" list in the "fetch news" section had disappeared completely. It took a while to track down where calibre stored that file (I had to schedule a couple of items so that the file would become "recreated"). I finaly found it and then could hunt in my backup programs to find the file that was deleted around 5:00am this morning. And that file is NOT STORED IN THE ONEDRIVE FOLDER. And calibre itself is not installed in the OneDrive folder.

So OneDrive is not an issue. Calibre might be. Or Windows might be; why did it even restart - there was no update going on. Did calibre throw a big error and trigger a restart? Don't know any of that. But I am confident OneDrive was a boon, not a bane.

[ownedbycats]

For every "I've never had a problem in ten years," there's 4-5 other people posting on MR or the bugtracker after OneDrive or Dropbox or Google Drive broke their library even though it's clearly stated in the FAQ to not put their libraries there. And frankly, I think a few of us are getting a bit tired of it by now.

[theducks]

With NAS or other remote drive, there is always the chance that multiple accesses is possible (even if not intentional). Calibre caches data, so changes from other instances is not detected. That leads corruption.