[GUI Plugin] FanFicFare

[Atherton25]

Alright. I did some testing, and I get dups every time when I do this.
So I have several virtual libraries setup. These are all permanent, but I checked with a temp virtual library just simple filtering out something that makes all my FFF stories unique (I have a tag I add called FFF so it makes filtering out FFF stuff easier).

JimmXinu, I'm not sure how you performed your testing, but this is essentially my testing steps

1. Since this was a test to see if it would cause dups I just found a very short story on one of the sites (during testing I tried 3 different stories, one on Fanfiction.net, one on hpfanficarchive.com and one on ficwad) for the last test I did, I grabbed https://ficwad.com/story/278903 a 1k word, 1 chapter story on ficwad. Clicked the FFF icon, Downloaded the story.
2. Once the story was in my library I searched for "not tags:FFF" to filter it out (along with all my other FFF stories)
3. Went to virtual library button and clicked on 'current search' to use the current search as the filter for my temporary library.
4. Now, in the new virtual library I clicked on FFF button, added the story url. Though I don't think it matters, the selection for the dropdown is:
output: epub
if story exists: update epub if new chapters
Update Metadata checked, Update Epub Cover checked.
(I also checked a the drop down options and selected overwrite if newer in one of the tests in one of the tests)
5. Clicked okay and in a few seconds it wanted to add a new story to my library. Once I closed out of the temporary library I had duplicates. 4 in one case, as I tried a couple different options and refreshed the Virtual library to make sure the old one was filtered out before trying the new test.


Now, the way I see it (and this is what I meant when I brought it up the other day), this is a feature of calibre and there's not a lot that FFF can do about it. When in a Virtual Library what exists outside that library is considered a different library and since the filters I used removes anything downloaded with FFF, once I turn on the Virtual Library option those stories no longer exist in my library (on a temporary basis) so FFF can't see them.

It's not a bad thing, it just means that if I'm doing other work and using a Virtual Library tab, I shouldn't try and use the update email feature or add new stories that I might already have. ... or be watchful that I don't accidentally click one of my tabs before using those features, which has happened several times.

Adding more information that probably doesn't matter, some of my libraries are based on custom columns (I have a 'library' column too for different things like fanfiction/webfiction/fiction/nonfiction/etc) and I used the same thing to filter out a FFF story and it still did the same thing.

Edit:
Just for reference. And this probably doesn't matter, but I wanted to make sure. I'm using on Win7, using the latest version of Calibre (x64), the latest version of FFF. I have used the cli once or twice in the past, but nearly always use the gui version.

I've also had the same problem on my laptop in the past (Win10) a couple of times. Most of the time if I realize I've done it (generally because what should be a quick update is taking too long) I just stop the download, go into my full library and restart downloading correctly.

[JimmXinu]

Okay, NOW I think I see what's going on.

Atherton25, please look in your FFF settings, do you have "Search by Title/Author(s) for If Story Already Exists?" on or off?

If you have it off, please check it on and try your test again.

What I'm seeing is that the identifier search is 'honoring' Virtual Library and the title/author search doesn't. And because I always have "Search by Title/Author" on (it's on by default), I--and most users--never saw this.

## honors VL??
self.gui.current_db.search_getting_ids('identifiers:"~ur(i|l):~^%s$"'%re.sub(r'https?\\\:','https?(\:|\|)',re.escape(url)),None)

## ignores VL??
mi = MetaInformation(book['title'],book['author'])
identicalbooks = self.gui.current_db.find_identical_books(mi)

I'm going to go look in the Calibre code at these methods to see what's different between them.

[Atherton25]

Originally Posted by JimmXinu

Okay, NOW I think I see what's going on.

Atherton25, please look in your FFF settings, do you have "Search by Title/Author(s) for If Story Already Exists?" on or off?

If you have it off, please check it on and try your test again.

You're right, it fixed it.

But, I keep that unchecked for a reason, since I've run into sites that have 2 different versions of a story and generally I want both. Plus, this issue doesn't really bother me much, I can simply search for duplicates with duplicate finder (by url) and delete them when I do this. It was fathertyme that had brought it up. At least it helped (possibly) identify an issue he and others might be having. I'll try and remember to note even more calibre options next time if there's a possible issue.



Moving on, I know this has been brought up before in some form (probably by me) but I've had a lot of new issues/stories that I've had to make edits in my personal.ini for. My current personal.ini has 90+ (and growing) specific story's now in it to 'fix' this, I'm going to ask about it.

The issue actually stems is mostly due to the way the wuxia sites do things, and even worse on some than others. Generally due to translation, sometimes the title/author can vary a bit from site to site, and sometimes they even just use the Chinese/Japanase name rather than a translation. Also the description can be really screwy (on royal road as well), adding a lot of superfluous (update schedules for example, but a while back I had one wuxia story that had like 5 pages in the description that was not needed).

As far as I know (based on past questions) the only way to keep an edited version of a new title/author/description in the FFF metadata (specifically on the title page) is to manually add it into the personal.ini.

Am I incorrect? If I'm not, how difficult would it be to have an option to grab the calibre metadata instead of the website metadata? I know I can edit it in calibre and use options to keep the calibre data the same, but it never reflects inside the book title page for those specific options. If it's too difficult just ignore me, I'm probably the only person that gets so annoyed by this issue partly because I really like clean, proper metadata, partly because some wuxia sites don't always give a real description and I have to get it from another site (generally novelupdates.com) for something that actually makes sense.

On a side note, somehow getting a downloaded story to interconnect with novelupdates and pulling metadata from their would be awesome but likely way too much work.

[JimmXinu]

New Test Version Posted

2019-01-09
- Ignore current Virtual Library when checking for existing story ID.

Note that this could be change to the existing behavior for some Virtual Library users. I could potentially make this optional, but I'd much rather it just be consistent.

Originally Posted by Atherton25

You're right, it fixed it.

But, I keep that unchecked for a reason, since I've run into sites that have 2 different versions of a story and generally I want both. Plus, this issue doesn't really bother me much, I can simply search for duplicates with duplicate finder (by url) and delete them when I do this. It was fathertyme that had brought it up. At least it helped (possibly) identify an issue he and others might be having. I'll try and remember to note even more calibre options next time if there's a possible issue.

Many thanks to Atherton25 for helping figure this out.

Now I'll go read the rest of that last post.

[Atherton25]

Originally Posted by JimmXinu

New Test Version Posted

2019-01-09
- Ignore current Virtual Library when checking for existing story ID.

Note that this could be change to the existing behavior for some Virtual Library users. I could potentially make this optional, but I'd much rather it just be consistent.

Yeah... I don't know why people would keep 2 version of the exact same story, but there are some other possibilities.

I do know that some authors repost a 'revised' version under the same url sometimes, or remove chapters (particularly in the case of royalroad and people using kindle unlimited).

I generally just remove the url identifier and add [old] to the title or something, then add the story again in it's new/revised form. *But* I did for a while use sorting to allow me to update the new chapters only (before I figured removing the URL and unchecking title/author allowed me to do a normal update via url rather than highlighting)

This was before vlibraries, where I could have just auto filtered them out before trying to update. It is possible some people use that method for one reason or another.

[Atherton25]

Bah, for information purposes if anyone's interested and can't figure it out themselves. For anyone trying to 'fix' authors on webnovel.com so it doesn't add editor,

replace_metadata:
author=>^(.+?)Translator(.*)=>\1

placed in your personal.ini will fix this, removing the translator name completely, as well as editor if there is one in the metadata. It could be further modified to add translator as a second author name, but I just keep original author personally. Don't know if it helps anyone, but I figured I'd post it for those with little to no knowledge of regex.

[JimmXinu]

Originally Posted by Atherton25

... As far as I know (based on past questions) the only way to keep an edited version of a new title/author/description in the FFF metadata (specifically on the title page) is to manually add it into the personal.ini. ...

It's a little involved, but I think you can do at least part of this.

1. First, you'll need to identify (and/or create) the columns you want to hand edit.
2. If they are columns that are auto-populated, make sure under the FFF Standard Columns and/or Custom Columns tabs that they are checked 'New Only' so FFF doesn't change them.
3. Next, go to the personal.ini FFF tab. Towards the bottom, make sure "Pass Calibre Columns into FanFicFare on Update/Overwrite" is checked.
4. Just under that, click the "Show Calibre Column Names" button. That will show you a list of your custom and standard Calibre columns and their FFF metadata entry names. Copy the names for the columns you care about somewhere for reference. Eg, you might want calibre_std_title, calibre_std_comments & calibre_cust_characters.
5. Now go into personal.ini and add those (calibre_std_comments, etc) to titlepage_entries as you would like. Either by outright replacing the default titlepage_entries or using add_to_titlepage_entries, extra_titlepage_entries or even add_to_extra_titlepage_entries in appropriate sections. (I won't write an essay about the differences unless you need it.)

Example:

[defaults]
calibre_std_comments_label:My Comments
add_to_titlepage_entries:,calibre_std_title,calibre_std_comments,calibre_cust_characters

Some issues:

• During the download of a new story, there won't be any values for calibre_std_comments, etc, so they will be empty.
• The label for each of these (calibre_std_comments_label, etc) are automatically set to match the Calibre column name. You can also set them in personal.ini for your self.
• Trying to change the title page to use your calibre_std_title entirely (see wiki) would possible, but difficult--again, due to not having calibre_std_title available on initial download. You'd probably want/need to keep the site title even if you reduced its size.

Originally Posted by Atherton25

... I generally just remove the url identifier and add [old] to the title or something, then add the story again in it's new/revised form. *But* I did for a while use sorting to allow me to update the new chapters only (before I figured removing the URL and unchecking title/author allowed me to do a normal update via url rather than highlighting) ...

I do the same; remove identifier, change title.

And I can vaguely imagine someone deliberately using VL to get 'duplicates'. But because the id and title/author searches were inconsistent and people were reporting it as a bug, not a feature, I'm going to assume that consistent searches that can't make duplicates are the best way to go.

If somebody says they were using it that way, I'll revisit it then.

[Atherton25]

Originally Posted by JimmXinu

It's a little involved, but I think you can do at least part of this.

Excellent. Thank you very much.

I'll see about implementing this. I know I've seen (and done) some strange workarounds before. As long as they work, they work.

It's probably better than doing the (now semi constant) updates to the ini file for new books. Especially because people on wuxiaworld.co don't even put in a full description half the time. The just grab what's available and half the time it's cut off mid-point.

[Mara_Skywalker]

I have two problems with fanficfare:

What can I do to download fanfictions or quests from forums like sufficient velocity, spacebattles, questionable questing, alternate history that have neither threadmarks nor links in the first post?

I would like to download fics that have story/quest only threads like this one:

https://forums.sufficientvelocity.co...-archive.3519/

but on all the forums like this.

And I have problems with quests that are too long, for example, if I want to download this here:

https://forums.sufficientvelocity.co...r.27579/reader

It does not download all threadmarks, only the ones visible, so a lot of posts in the middle are not downloaded. Do you understand what I mean?

Would it be so hard to give the possibility to just download all the posts in a story only thread, by choosing that manually?

And to somehow also make it possible to download the quests, that have too many threadmarks?

Thank you for your help!

I do not know anything about programming, or I would try to make something myself.

I need the whole quest/story in epub format to open it with the freeware balabolka, that is capable of reading me the stories aloud because I really cannot read that long without my eyes starting to hurt.

[JimmXinu]

Originally Posted by Mara_Skywalker

What can I do to download fanfictions or quests from forums like sufficient velocity, spacebattles, questionable questing, alternate history that have neither threadmarks nor links in the first post?

This is an issue that's come up before. See those posts for why FanFicFare doesn't implement whole-thread downloading.

Originally Posted by Mara_Skywalker

And I have problems with quests that are too long, for example, if I want to download this here:

https://forums.sufficientvelocity.co...r.27579/reader

It does not download all threadmarks, only the ones visible, so a lot of posts in the middle are not downloaded. Do you understand what I mean?

Back in Sept SB & SV changed to 'hide' ranges of threadmarks behind '...' links. Code was added to FFF then to read past them. Do you have the current version of FFF installed?

I ran the chapter collection part of FFF on the story given and found 3,359 chapters, which is reasonably close to what the site reports(see below). I'll start it downloading and check it later.

FYI, SB/SV have an unfortunately tendency to not list all of the threadmarks for some categories when the list is small, or to report threadmark numbers that are off. There isn't much I can do when the site gives out bad data.

UPDATE: It took a couple hours with the slow_down_sleep_time:10 setting I threw in, but I downloaded that story successfully.

If you still have issues or want a copy of the epub I downloaded, let me know.

[Kammal]

Sufficient Velocity is giving me this error since yesterday, no idea why, I can access it just fine in my browser.

Spacebattles is working fine.

HTTP Error 503: Service Temporarily Unavailable

[JimmXinu]

Originally Posted by Kammal

Sufficient Velocity is giving me this error since yesterday, no idea why, I can access it just fine in my browser.

Spacebattles is working fine.

I'm not seeing this error.

But it says it right in the error: "HTTP Error 503: Service Temporarily Unavailable"

It could be that the site is simply having problems, just as the error says or that you (or your network) is behind a proxy of some kind that is having problems.

Regardless, there's nothing I can do to help you with server errors.

[Kammal]

Originally Posted by JimmXinu

I'm not seeing this error.

But it says it right in the error: "HTTP Error 503: Service Temporarily Unavailable"

It could be that the site is simply having problems, just as the error says or that you (or your network) is behind a proxy of some kind that is having problems.

Regardless, there's nothing I can do to help you with server errors.

I found out a way around it, if I use a VPN from my country I still get the same error, but if I use a VPN from Canada, it works just fine.
Really weird, on my browser it works just fine without a VPN.

[fanfictioneer]

I am trying to download this link: https://forums.sufficientvelocity.co...-litrpg.51514/

However, it seems it requires you to be logged in to view the thread at all, and FFF states that the story does not exist (I do have a section for forums.sufficientvelocity.com in my personal.ini). Is the adapter not logging in before checking whether a story exists at all?

[JimmXinu]

New Test Version Posted

2019-01-15
- Add always_login setting to base_xenforo for SV login-required story with 404 result.


I've never seen a user/pass required story on SV before.

We already have code for detecting stories that need login on QQ by the 403 - forbidden return code. But this is coming up 404 - file not found.

Because we still need to be able to detect and handle not-found, I've added always_login as an option for base_xenforo. But you should only set it for stories that need it.

Example:

[https://forums.sufficientvelocity.com/threads/small-medium-litrpg.51514/]
always_login:true

[forums.sufficientvelocity.com]
username:UserName
password:XXXXXXXXXX