[GUI Plugin] FanFicFare

[Atherton25]

I apologize for not including others, I figured that it would likely all be the same error.

The latest update fixes all the stories I was finding issues with.

Now, as it is, the only other things I can find that are not working quite right is the synopsis. When converting it into comments, it seems paragraphs are disappearing and everything is running together. I noticed this yesterday, but reconfirmed it today. Any story I look at is this way.

For example:
http://royalroadl.com/fiction/4293 should have a synopsis of 4 paragraphs, and it is turned into a single one in the comments section.
http://royalroadl.com/fiction/1439 should have 3 paragraphs, and again is turned into a single comment paragraph.
Both of these are picked from the top-rate list, just picking some that have several paragraphs in the description.

[Atherton25]

@wassname
I found one more that's giving an error. It's not the same as any of the others so far.
http://royalroadl.com/fiction/108
Error is: Missing required element.

[JimmXinu]

Originally Posted by Atherton25

@wassname
I found one more that's giving an error. It's not the same as any of the others so far.
http://royalroadl.com/fiction/108
Error is: Missing required element.

Chapter titled "good start" links to http://royalroadl.com/fiction/chapter/4375, which gives a 404 error. Which makes it an error in the site.

Can you show more than one story with that problem? As a one-off, I'm inclined to leave it, since the alternative is to silently continue downloading and fill in a failure message for that chapter.

I don't know about you, but I don't want to suddenly discover while reading a book that some chapters in the middle of it are missing.

You can still download the parts of that story that are there in two parts using URLs with chapter ranges:
http://royalroadl.com/fiction/108[-34]
http://royalroadl.com/fiction/108[36-]

[wassname]

Originally Posted by Atherton25

@wassname

...

I do appreciate you getting the plugins. Working for RRL ... I had been using another downloader and manually entering some info, but it doesn't have the same update feature as Fanficfare does plus it quit working with the new RRL site update a week or two ago. I'd only gotten about 30 stories down at the time and was cursing myself for not just downloading tons of them soni had them when I got to them.

No worries, I had the same problem so I decided to try to add royalroadl instead of making my own script. Now I can load up my kindle with royalroadl stories and feel rich with books. Next step is to stop collecting and actually read them

[wassname]

Originally Posted by Atherton25

I apologize for not including others, I figured that it would likely all be the same error.

The latest update fixes all the stories I was finding issues with.

Now, as it is, the only other things I can find that are not working quite right is the synopsis. When converting it into comments, it seems paragraphs are disappearing and everything is running together. I noticed this yesterday, but reconfirmed it today. Any story I look at is this way.

For example:
http://royalroadl.com/fiction/4293 should have a synopsis of 4 paragraphs, and it is turned into a single one in the comments section.
http://royalroadl.com/fiction/1439 should have 3 paragraphs, and again is turned into a single comment paragraph.
Both of these are picked from the top-rate list, just picking some that have several paragraphs in the description.

Good point, I didn't realise the synopsis was in HTML. So I've submitted a change to JimmXinu which makes it work http://royalroadl.com/fiction/4293. So it should have images and paragraphs in the next test version. (Thanks for the help JimmXinu).

[Atherton25]

Originally Posted by wassname

No worries, I had the same problem so I decided to try to add royalroadl instead of making my own script. Now I can load up my kindle with royalroadl stories and feel rich with books. Next step is to stop collecting and actually read them

It's always nice to help, though I can't program myself. It's always something I wondered if I should have done in the past but I'm not sure if I'd ever be able to do it very well.

I'm glad the kinks to the plugin are getting worked out. Feel free to employ me if you want someone to test any other sites you feel the need to add :P.

As for RoyalRoadL, I've only semi-recently been introduced to it, I started reading stories around 2 months ago, starting slow with just a few stories while I worked through others near the front of my queue. Thus the reason for the cursing when I really got into it and found stories I hadn't got to were no longer downloadable by that other program.

[DavidTC]

Fanficfare has a 'modified' column we can put in a custom column that gets updated if the story changes, and Calibre itself has a column called Modified that gets updated if the metadata changes.

But nothing appears to be changed if we check for updates and no updates are found, at least not with 'Update EPUB if New Chapters'. (Well, the Calibre metadata update column changes if some random thing like the author changes, but otherwise nothing.)

As someone who has a bunch of stories in progress, (and a virtual library that only shows uncompleted stories, unless I marked them as dead, so I know what to update) I'd like a way to sort and see the stories I haven't tried to update in a week, or whatever. Or I could sort and select the 10 oldest checked stories for update every date, or whatever I wanted to do.

So, can we get a new date value we can put in a column, called 'Checked' or something, that indicates when we last checked the story against the website?

[JimmXinu]

@DavidTC - You can do that now by adding a custom column for 'errors' and using the settings on the Custom Column tab to configure FFF to update it when checking updates (see attached). That will update Calibre's last_modified time.

[DavidTC]

Two things.

First, will that update if the 'error' is the same as the error already there?

I.e., if the error is 'Already contains 4 chapters.' and the error column *already has that value* (As it would have if this is the second time checking for an update), would the column still get updated and the metadata change?

Second...the Calibre Modified column gets updated by other stuff, too. For example, there is literally no book in my library modified before Sept 28, 2016. I don't know what I *did* Sept 28, 2016, maybe I added a field or something, but it touched every book in my library.

And at the bare minimum, most of us have a 'read' column, and modifying that updates the modified date.

So the modified date wouldn't *actually* tell you when a fic was checked last, it just tells you the fic can't have been checked *later* than that.

[JimmXinu]

Originally Posted by DavidTC

First, will that update if the 'error' is the same as the error already there?

I.e., if the error is 'Already contains 4 chapters.' and the error column *already has that value* (As it would have if this is the second time checking for an update), would the column still get updated and the metadata change?

Yes, it does.

Originally Posted by DavidTC

Second...the Calibre Modified column gets updated by other stuff, too. For example, there is literally no book in my library modified before Sept 28, 2016. I don't know what I *did* Sept 28, 2016, maybe I added a field or something, but it touched every book in my library.

That's also true.

I primarily hesitate about adding this because I don't want to encourage people doing mass updates.

However, you make good points that it isn't something that's easy to do otherwise. I'll look into it.

[JimmXinu]

New test version posted (direct link)

@DavidTC - See if this does what you want. Look at the bottom of the Custom Columns tab.

2016-10-28
- Fix for royalroadl.com Thanks wassname!
- Add Last Touch Column feature
- Bump test version.

(Suggestions on what to call it instead of 'Last Touch' welcome.)

[DavidTC]

Yes, it does.

Ah, okay. I had assumed it wouldn't.

I primarily hesitate about adding this because I don't want to encourage people doing mass updates.

Ah, the reason I want to know when I last updated is exactly so I *don't* do mass, or repeatedly unneeded, updates. As in, right now, I looking at a big list of fanfic, and I'm sure that *some* of it I've checked recently, but I'm not entirely sure which.

I was going to suggest, but deleted it due to not wanting to complicate things, an idea that having a 'Checked' field would allow the plugin itself to reduce updates. But I will suggest it now:


First, to use this idea, they have to have a Last Checked field, along with having set up a 'Completed' field to remove things from checking.

And then, once they have that, there could be a fanficfare command of 'Do updates' and only allow that to be run once every ten minutes or something, and only check about five of them, and only ones that are more than a week old and incomplete.

I'm not saying remove the manual update, I'm just saying, present the user with some option besides it, where fanficfare sorta 'rations' out updates, and no one has to deal with worrying about overloading anything or figuring out what to update, and that's the option people use 99% of the time. (While still checking that one fic they can't wait on manually.)

Could even have some logic to favor splitting across different servers, instead of just using the five absolute 'oldest'. Like if the oldest three are the same server, it then tries to find the next two that *aren't* on that server but haven't been checked in a week.

I would even suggest this would work best as a background process (Every ten minutes check one, until nothing is older than a week.), but I bet that's completely impossible with how Calibre works.

[JimmXinu]

Originally Posted by DavidTC

Ah, the reason I want to know when I last updated is exactly so I *don't* do mass, or repeatedly unneeded, updates. As in, right now, I looking at a big list of fanfic, and I'm sure that *some* of it I've checked recently, but I'm not entirely sure which.
...

That is why I encourage everyone to use email notifications and even added a feature to pull update URLs from email for you.

This sort of thing has been discussed more than once in the past.

I provide a few features to make life a little easier for those who want to poll for updates instead of use notifications. But I hesitate reluctantly over each one because I honestly believe that is the wrong way to maintain a fanfiction library.

I have implemented the last-touch datetime feature you requested. (You posted immediately after I announced it, so you have missed it.)

I did that much partly because I can see the point that mass-update people may do smaller updates, partly because I thought was a somewhat-interesting-yet-easy change, and mostly because I didn't have anything better to do today.

So I encourage you take it as a victory that you got last-touch, rather than a defeat that I'm saying 'no' to the rest.

[DavidTC]

That is why I encourage everyone to use email notifications and even added a feature to pull update URLs from email for you.

The problem there is that requires people to make an account on every site, and it also is a lot more work to set up for each work vs. just dragging a URL into the window. (Which you still *also* have to do.)

It would be amazing if there was some site where, when visiting, there was a button that said 'Email me this story URL right now, and also email me when the story updates', but that doesn't seem to exist. I have to go into the story (as opposed to just dragging it from the URL) and subscribe, and then *also* drag the URL into fanficfare.

(I guess it might be possible to find the monitored stories page and try to grab all the URLs on that page to add the new stories I subscribed to, but that would update all existing stories, defeating the point of not doing mass updates!)

----

That said..when talking about server load, I'm not entirely sure that sending an email is less load than loading a page, if the checking is sane and spaced out. That seems like the sort of assumptions about load it is easy to make from the outside, but could be entirely wrong.

If people were psychic and *only* checked for new updates *when* there was a new update, checking is less load than emails. (Obviously, as the email results in a check anyway.) Whereas checking every minute is obviously more load than notification emails.

So there is *some* crossover point, and I rather suspect it's actually a lot more biased against email than people suspect. Web servers are usually pretty damn efficient, whereas email introduces an entirely different server process managing a mail queue, and having to deal with all sorts of other internet servers in handing them mail.

And it's easy to assume that email notifications were introduced as a way to reduce load, but we don't know that. Email notifications might only exist for *user convenience*, not as an attempt to reduce load. It's entirely possible they're more load on average than sane checking behavior. (As in, for example, only stories with at least a 5% chance of having been updated since last check.)

And, honestly, if they really wanted to reduce load, the bigger sites would introduce some sort of Last-Modified header so you could just do a HEAD request, or some XML method, or something. The fact no site has stepped forward with some sort of 'Hey, we'll build you a quick way to check if the story is modified if you'll implement that, to reduce our load' rather implies they don't really care.

They'll block people who are crazy hammering on their server sometimes, yes, but honestly, I've been searching on AO3 before, opening tabs left and right, using a Chrome extension to copy all tabs to the clipboard, pasting them in Fanficfare, downloading three or four dozen stories, and never had a complaint or anything.

tl;dr - I think you might be worrying too much about mass updates...well, except for the whole 'I am going to select every book in my library and run the update over night' or nonsense like that. (Which for some reason you...allow? At least, I got to the point of being able to click Ok, although I didn't. Shouldn't there be some numeric limit on that?)

So I encourage you take it as a victory that you got last-touch, rather than a defeat that I'm saying 'no' to the rest.

Oh, I'm good, I was suggesting that more for other people, not for me. The rest of the stuff was pretty complicated. I already have a pretty good manual system that mostly worked, and this will make sure I don't completely skip ever updating some stuff ever.

I was just saying, if you're concerned about mass updates getting out of hand and pounding servers: Implement some easy-to-use slow-motion mass update thing that *isn't* out of hand. Or not. Whatever. _I'm_ good either way.

And...Last Touch? No offense, but that's a really confusing name. At least 'Last Checked'.

[JimmXinu]

New test version posted (direct link)

2016-10-28
- Rename Last Touch to Last Checked.
- Fixes for adapter_adultfanfictionorg.py from GComyn.
- New site: chosentwofanfic.com -- Thanks GComyn!
- New site: www.asexstories.com -- Thanks GComyn!
- New site: www.bdsmlibrary.com -- Thanks GComyn!
- New site: www.ficsite.com -- Thanks GComyn!
- Tweaks and fixes across several sites from GComyn.
- Site lotrfanfiction.com changed date format.
- Improve quotev.com not found reporting a little.
- Fix for potterfics.com layout change.
- Remove duplicate adapter for fanfiction.lucifael.com--still supported.
- Update translations.
- Bump test version.

Defunct sites?

The following sites are not working anymore for the reasons listed. If they are still down when the next release is made, they will probably be removed from FanFicFare.

www.restrictedsection.org - Parked domain

www.midnightwhispers.ca - No DNS domain found

www.psychfic.com - "Forbidden You don't have permission to access / on this server."

fanfiction.portkey.org - "website down We're in the process of trying to recover data from backup. ETA: 21 Oct 2016"