[GUI Plugin] FanFictionDownLoader

[Tanjamuse]

Is it possible when I add a whole page of stories (25) to get them shown in the opposite way they are on the page? Instead of from the top and down get them shown from the bottom and up?

[JimmXinu]

Originally Posted by Tanjamuse

Ok, It would have been nice

Do you know how to auto-sort on different columns, like Published date fx?

I use the View Manager plugin and have a view that sorts on my preferred three columns. Then I set a hot key to show that view when I hit space.

[JimmXinu]

Originally Posted by BookJunkieLI

Question about possibly updating/changing the way the Reject list behaves.

I'm wondering if it's possible to set/toggle FFDL to automatically either skip or reject any links that are already on the Reject list. Normally I don't mind if it double-checks if I want to re-download or reject a previously rejected story but I'm currently working on downloading roughly 54,000 stories from AO3. I'll set it to download 1,000 to 1,500 links at a time and wander away while it processes for an hour or so. Sometimes that's fine, other times I come back to find it stuck on a story waiting for a decision.

I did take a look at the personal.ini default settings but, honestly, I understood less than a 1/3 of what I was looking at.

There isn't such a feature, but there could be relatively easily. RL is swamping me right now, but maybe this weekend.

But seriously, 54,000 stories?

You really have that much reading time?

(And AO3 has that many stories?)

[BookJunkieLI]

Originally Posted by JimmXinu

There isn't such a feature, but there could be relatively easily. RL is swamping me right now, but maybe this weekend.

But seriously, 54,000 stories?

You really have that much reading time?

(And AO3 has that many stories?)

LOL AO3 actually has over a million stories. They passed the mark some time this spring, iirc.

Eh, I don't expect to read all of them. AO3, most archives actually, are great for letting you include certain tags in your search/filter but are complete crap at letting you exclude certain tags. Once I get them downloaded I'll be able to use Calibre to search on all the tags I don't want and I'll be able to add those stories to the reject list. And some of the fandoms I read have a large number of under 500 word "stories". I can blow through a dozen or more of them in a night.

When you get the time. I'm only 14,000 links done after a week. I'll be at this awhile.

[JimmXinu]

Here's a new test version that still has the Qt5 changes, but also adds a new 'Reject Silently' option. Look at the bottom right of the Basic config tab.

I'd also like to hear if anybody has better ideas for what text to use for the label and tooltip for that checkbox.

[duckblaster]

Something that might be useful to some people: an offline metadata cache of entire sites, updated from the "show all, sort by most recent first" pages or similar, would cut the to update all books in calibre a fair bit I think, certainly shouldn't be anything like the 1000+ http requests to check all stories individually, especially if you can choose which list pages it checks.

[aleyx]

Originally Posted by duckblaster

Something that might be useful to some people: an offline metadata cache of entire sites

Ah... No.

1) Those sites are _not_ built to make that easy or cheap (bandwidth-wise). It's not like they've got a SOAP/XMLRPC API for metadata queries (although goodness, how that would make our lives easier). Also, they'd follow FFN on the "cut off those leechers" bandwagon really, really fast.

2) I think you seriously underestimate the volume of data involved here. Especially when considering Kaiju-class sites like FFN. You'd have to either deal with a gigabytes-large metadata.db, or have to install a separate instance of MySQL/PostgreSQL. Not quite lightweight, certainly not portable.

3) Where do you get your 1000+ HTTP requests? You're supposed to only update those stories for which you receive an update notice from the hosting site, not have your entire library checked every n days/hours. Even then, I only update those stories I intend to read right away (or close enough), and leave the remaining notices for later.

So, yeah. It's a nice idea, but let's just put it aside for now, 'kay?

N.

[JimmXinu]

Originally Posted by aleyx

Ah... No.

...

I agree. Your calibre library is your offline metadata cache and you should be updating it selectively.

[BookJunkieLI]

Originally Posted by JimmXinu

Here's a new test version that still has the Qt5 changes, but also adds a new 'Reject Silently' option. Look at the bottom right of the Basic config tab.

I'd also like to hear if anybody has better ideas for what text to use for the label and tooltip for that checkbox.

Looks great! I haven't tested the functionality yet, been away until this afternoon, but thanks for getting to it so fast. The tooltip looks fine, but, maybe for the main text 'Reject without prompts(or prompting)?'

[BookJunkieLI]

Originally Posted by aleyx

Kaiju-class

Woohoo! Another Pacific Rim fan!

Originally Posted by JimmXinu

I agree. Your calibre library is your offline metadata cache and you should be updating it selectively.

I agree completely with aleyx and JimmXinu. Just using AO3 as an example, it has over a million stories on it. FFN has even more. I do not want a complete cache of those sites. AO3 I'd want maybe 5% of those stories. And of those 5% probably only 1% are incomplete and need to be checked periodically for updates.

This is my method for checking for updates:

I have 3 specific custom columns - Complete(Yes/No), Has URL(Yes/No), and Last Updated(Date).
I have a virtual library based on this search: #complete:no and #address:yes
I then sort based on Last Updated.
Depending on how many stories that produces I either select the latest 250 stories or go back roughly 6 months and then run FFDL.
Once that's complete I may or may not go back further. The oldest stories, some of them years since the last update, I only check every few months, if that.
This system works well for me and it doesn't overtax Calibre or FFDL.

[Tanjamuse]

How do you make a custom column for last updated? and can the plugin auto-fill that column?

and how do you make the "has URL" column?

[JimmXinu]

Originally Posted by Tanjamuse

How do you make a custom column for last updated? and can the plugin auto-fill that column?

and how do you make the "has URL" column?

Add a date type custom column, then after restarting, go into FFDL's config on the Custom Columns tab and choose to fill your new column with 'Updated'.

A 'has URL' column is really more of a library management question. There's a thread here that looks useful.

I believe you likely want a composite custom column with something like {identifiers:contains(url, Yes,)} for the template.

[JimmXinu]

Originally Posted by BookJunkieLI

Woohoo! Another Pacific Rim fan!

I think you'll find the word kaiju pre-dates Pacific Rim by several decades. Pacific Rim is an homage to the entire kaiju movie genre.

Originally Posted by BookJunkieLI

...
Depending on how many stories that produces I either select the latest 250 stories or go back roughly 6 months and then run FFDL.
Once that's complete I may or may not go back further. The oldest stories, some of them years since the last update, I only check every few months, if that.
This system works well for me and it doesn't overtax Calibre or FFDL.

Even that is a bit more excessive than I'm comfortable with for my own use. I continue to advocate using email notifications for updates and only updating stories that you know have updates.

[BookJunkieLI]

Originally Posted by JimmXinu

A 'has URL' column is really more of a library management question. There's a thread here that looks useful.

I believe you likely want a composite custom column with something like {identifiers:contains(url, Yes,)} for the template.

Spot on for the Last Updated column.

My Has URL column is actually literally just a Yes/No custom column that I manually set when I add new books. I've only just really started doing anything with composite columns in the last 3 or so months and I'm adding them slowly as I feel the need for them.

[BookJunkieLI]

Originally Posted by JimmXinu

I think you'll find the word kaiju pre-dates Pacific Rim by several decades. Pacific Rim is an homage to the entire kaiju movie genre.

Eh, true. I'd forgotten about that.

Originally Posted by JimmXinu

Even that is a bit more excessive than I'm comfortable with for my own use. I continue to advocate using email notifications for updates and only updating stories that you know have updates.

Everyone has a preference. I don't do daily checks for updates, weekly to every other week depending on when I remember. Plus not all sites I have stories from have email notifications.