[GUI Plugin] FanFicFare

[DarkMagda]

Thank You! I used edge instead of firefox and it worked just fine.

[mehetabelo]

I know that this is probably not an easy thing to do - but I have a suggestion for adding an option into the interface.

Somewhere, it would be *really* helpful to have a regex option for find/replace that directly applies to the html. This suggestion is based on spending several hours (maybe even days) of time working through some of the newer sites and fixing the html particularly the paragraph setup so that they show properly on my ereaders. It's not the fault of FFF, but rather the shitty HTML on the sites.

I've been running into having multiple <br> lines, or an extra <p> </p> between, or even a paragraph with <p> </p> as the separator (but not actually showing identifying the paragraph) an example of the last is:
<p> </p>
Paragraph X.
<p> </p>
Paragraph Y.
<p> </p>
Which in a kindle leads to a massive cluster because it doesn't read these as it is intended (correct or not). There are other things like &quote appears fairly commonly, instead of " symbols on some older html from webnovel. But it's weird, as it only does it sometimes and I've literally had a chapter that's good, one bad and the next good again.

Different sites do different things and I've worked out some regex replacements that I use in the convert dialog, or (mostly used) in the editor dialog.

Generally these regex are specific enough they should not effect each other, or screw up good text. My problem is anytime I get an update I have to fix the book again with the new chapters. That's why I'm asking for this option. I'm not sure exactly

Since the needed regex would likely vary a little from site to site, I'm Not sure exactly how it should be done, maybe within the preferences (probably best) or maybe separately, but it'd be good to have it available by site.

This is just a suggestion, but I believe it's a good one.

[JimmXinu]

Originally Posted by mehetabelo

Somewhere, it would be *really* helpful to have a regex option for find/replace that directly applies to the html.

Have you tried the replace_br_with_p option? That might help with some of it.

This idea has been raised before. I'm reluctant to implement such a feature in FanFicFare for a couple reasons:

First, it would be a very dangerous feature that could very, very easily be misconfigured to destroy the text it's trying to fix. It's a shotgun equipped with a stirrup on the end--you inevitably will shoot yourself in the foot with it. Plus you'll also end up hitting sites over and over again while fine tuning the settings.

Second, IMO, it's really outside of scope for FFF to be doing that level of editing on the story text. I know it can be done in Calibre's Edit Book, and while I haven't experimented with much, I know that Calibre's Convert has a 'Search & replace' regexp feature that can load and save lists of replacements.

You may then wonder about the replace_br_with_p option in FFF I just mentioned? That feature was contributed by another developer (Asbjørn Grandt). I was a bit hesitant about it, but I've since come to use it myself for some sites.

But earlier this year I discovered that it was accidentally corrupting the story text when run more than once on the same HTML during some updates. It's since fixed to only run once, but it was a reminder that such processing is dangerous.

[mehetabelo]

First, I don't want you to feel pressured to make the option. I want you to really consider it, and this post is for that specifically. Yes, I want the option but I'm not a programmer, so I'm not going to be the one doing any of the work (should you actually decide to do it).

I'll address the first comment by saying yes, I have tried the replace with br option. It doesn't always work, but it did cut down on the issues I had with several sites. And it works well enough most of the time for them. It's not perfect, and I'd have to test it to see why, but I know it doesn't fix all the <br> problems I did have.

Next, the dangerous part. Honestly you're right, it *could* be a dangerous feature.

As a note, I'd try implementing it myself if I knew how to program so that others wouldn't have to deal with the possibility, and I'd be content instead of trying to fix each book every time I try and read a new chapter. I'm not exactly OCD, but when I am reading and a paragraph is spaced far apart on my kindle (with a limited screen size to start with), it makes my mind go into fits. I HATE it, and thus have to fix it.

Back to be dangerous.

First, honestly how dangerous is it really? With a little time and effort the find/replace setup isn't that hard to figure out, and if it's screwed up, a story can be re-downloaded. If it's a chapter or two, it's easy to delete the chapter and re-download it after disabling (or fixing) the regex. As long as it's the same, or similar to a standard regex format (as used within the editor) I've already got 8-9 setup that haven't had any real problems when I've used them within the editor.

Second, you could specifically place a statement/note somewhere (on boards, in the INI, whatever) that this is unsupported. Beyond a basic rundown of the setup and an example regex maybe, noting if there are non-standard regex then point out the differences. People that want to use it can use it, and people that don't, don't need to bother with it. From past history on this thread, there are a minimum of a dozen (likely many more) people that have more than a little programming experience and have contributed in some way to the project. These people would likely find the regex feature nice when addressing formatting issues from certain sites. I have a fairly limited list I download from, and of those the only one that never has problems is fanfiction.net, maybe AOOO (though I don't download much from that site). Many of these site issues are fixed with replace_br_with_p option - but again it doesn't always work for the BR tags, and even then, it doesn't address other problems.

Next is your mention of being outside the scope of editing the story text. Yes, this could truly edit story text if needed, but the need called for is more of a formatting issue. Let me give you a couple examples of my regex find/replace statements I use. I don't have them all with me, as I'm traveling again, but I have some of the more common ones I use just for cleaning up spacing and major issues.

First is for empty paragraphs. I started with:
Find:<p>[^\S\r\n]{1,}<p>
But after many, many hours of use and modification, I believe (not 100% sure) the final one came to:
Find[\r\n])?<p[^>]*>(\s)?(<span[^>]*>)?(\s)?(<br/>|<br>)?(\s)?(</span>)?(\s)?</p>([\r\n])?
Replace:
Yes, replace with nothing. This specific function removes any empty paragraphs I have found so that I'm not faced with larges spaces between paragraphs. I've yet to see it fail or screw anything up, and I've used it in dozens, if not hundreds of different downloaded books. RoyalRoad, Wuxiaworld, and Webnovel are generally the targets of this specific F&R.
There are also some odder <p> setups I've seen used like:
<p class="p_line_space"> </p> that I believe this covers. Just to be clear, I'm pretty sure the <p class="p_line_space"> was in a fff download, but I can't be 100% sure as I was not keeping track of everything I've had to fix. Most of them (easily 95%+) have been FFF, but some were not.

Next
Find:&amp;quot
Replace:"
I previously mentioned this issue and it's a weird one on webnovel. I can find a handful of chapters with this problem, followed by good chapters, then bad chapters again. It's in no way consistent but is annoying as all get out when you run into it while reading. There was another that was an apostrophe problem that occurred as well, but it was rarer, and I don't have the current statement in my laptop.

Another, this on for BR stuff that pops up - these are specifically for places where I found the replace_br_with_p had not worked right (possibly because it was the older code you mentioned). Sometimes I download stories and put them on my kindle but don't get to them for months.
Find\s)?([\r\n])?(<br/>|<br>)(\s)?(<br/>|<br>)?(\s)?(<br/>|<br>)?(\s)?(<br/>|<br>)?(\s)?([\r\n])?
Replace:</p>\r\n<p>
this is also used in conjunction with:

Find:chapter-content">(<p>|<br/>|<br>)?(\s)?
Replace:chapter-content">\r\n<p>

Of note, I don't think that was the actual finished F&R statement, I had some modifications I made to it, but it cleaned up the paragraphs a little more so the first one (after chapter header) had a starting paragraph mark. I also had made one for the ending paragraph mark as well. However, I do not have it on this laptop - I was just using cleanup HTML as a quick fix for a while.

These were three of my most used ones and are therefore on my laptop for quick fixes when I was on the road. I don't have the others with me.

Now, looking at these, there is nothing here modifying any of the content, just spacing and fixing the quote mark. We're not doing any actual editing of the story, we're just cleaning the crap HTML that's downloaded. I have found that of all the stories I've had to edit to fix what they look like, probably less than 2% are actual fiction. Generally published stories have had a good editing job done and are clean without fixes. They're not perfect, and I have had to edit afew, but the vast majority of content requiring editing is specifically FFF downloaded stuff. Since this is the case, should it not be a part of the program to allow for something to edit and fix the formatting as they download?

You yourself said:

Originally Posted by JimmXinu

You may then wonder about the replace_br_with_p option in FFF I just mentioned? That feature was contributed by another developer (Asbjørn Grandt). I was a bit hesitant about it, but I've since come to use it myself for some sites.

Think about this for a little while, and consider again. Would this not be likely something similar? I don't know what sites you do use and if you have problems on them that are not addressed by the replace_br function, but if you do have some after spending a little while to make a correct F&R statement, you'd never have to worry about them again. I have had a few small issues with the F&R, such as the br statement generating too many empty paragraphs, (people using a combination of br and p in their html causes this) but I generally use both, with the br statement first and it fixes everything.

Now, I know some of this can be done through other functions of heuristic processing, as an option in the convert box or can be done in the search and replace function in the convert box, but both options do not work the way intended at all times. I very rarely find the heuristic 'Delete Blank lines between paragraphs' works well. That may be because it works a lot of the time, but I only see the times when it doesn't (and am forced to edit it myself). The second option using the search/replace function built into convert. I have set these up in the past, and it works pretty good. However, I find that I have to load this often when I convert for some reason, though not every time. This means that if I don't remember I have to go back and reconvert the document. If I could automate it, it'd be easier to use this function.

I know I rambled on a lot, but I'd like you to really consider the good and bad this could do - yes, people could really screw up their downloads, but realistically if they do some of their own research regex is not hard to learn. I've asked question in the past, but mostly I learned from researching through different boards. Most people will make a few statements, make sure it's working and just let it run in the background and never have to bother with it. Those people that do use it (and need it) will love the feature. I was not kidding when I stated how long I've spent manually editing files to get them to a place where they were readable on my kindle. The overall time is dozens and dozens of hours between all the different stories I've edited. I did use the built in S&R function, even replacing my original epub at times, but I've got to do it over and over again, and I think this would be a better way of doing it.

I don't know how much more work it would be, but if you're that worried about people screwing up downloads, you could even had a secondary button in the personal.ini tab that allows enabling/disabling with a specific warning about how messed up it could make your document. I honestly don't think it would require that level of caution, but it would make it very clear you're not responsible and make things easier to turn off/on for someone to check and see if that's the problem - should one occur.

Anyway, that's my thoughts. I spent a good week thinking this over before I even posted the suggestion. I hope you truly consider it, and maybe other people may want to chime in and say if they think it's a good idea. I know the board isn't terribly busy, and I don't know how many people keep up to date (I generally check every couple days) but I really do think it would be useful for lots of people.

[cryzed]

I admit I haven't read the post in its full detail, but modifying HTML with regular expressions is a bad idea for anything that exceeds trivial changes -- and this is quite clearly complex and handles many corner cases.

This would be much easier and more sane by simply using the parsed DOM tree (BeautifulSoup object) to make the desired changes from within FFF.

[JimmXinu]

As requested by mehetabelo, I have been thinking about this.

But I stand by my previous statement: It's a shotgun equipped with a stirrup on the end--you inevitably will shoot yourself in the foot with it.

It's just not something I want to have in software I support.

[Tanjamuse]

I tried searching to see if this has been answered before, but couldn't find anything.

Is there any way to collect the url for a series on AO3 when downloading metadata?

Example story url: https://archiveofourown.org/works/7695883

Example Series URL: https://archiveofourown.org/series/524566

[JimmXinu]

Yes. Just the series URL is available in seriesUrl. AO3, because it allows multiple series has series00Url (same as value seriesUrl), plus series01Url - series03Url.

[Ryune]

Right so... I'm still making my way through the thread for if this information was posted but can you guys recommend a e-reading software that can remain consistent with updating epubs? I've been using my Sony PRS-950 for years and years but I've gone into the market for a new reader and I'm curious if anyone has any recommendations for preserving things like page location when I upload a new book.

I'm currently looking at the Kobo Aura One or Kindle Oasis2. Can anyone tell me how well Koreader works on this front? Or how their native software handles serials from sideloading and wifi transfers?

I'm perfectly happy if you guys recommend some iPhone apps as well.

[JimmXinu]

Originally Posted by Ryune

Right so... I'm still making my way through the thread for if this information was posted but can you guys recommend a e-reading software that can remain consistent with updating epubs? ...

I tried a Kobo Glo HD in July 2015--I returned it because it did not recognize when ebooks(epubs) already loaded on it were updated. It was designed with the assumption that ebooks are static and don't change. I don't know if their newer devices/software have the same issue.

I've used three different Kindles (my current is a Voyage) and while all required mobi or azw conversion, they all handled updates and keep my place across updates as well or better than anything else I've used.

I've been very pleased with my Kindle Voyage for +2 years now.

My newer-older Kindle(5G) was not able to handle my full library when it passed ~3000 ebooks. It spent all it's cycles trying to index everything for searching and couldn't finish before I'd go and upload more updates. The Voyage is still working fine as I close in on 5000 ebooks. (Just realized I'm still on firmware 5.8.1 while current is 5.8.11--no idea what's changed.)

I can talk about older B&N Nook devices, but not since ~2015.

And I don't mean to speak for him, but our friend cryzed was telling me earlier this month that he purchased the new Kindle Oasis and was very happy with it.

[Ryune]

Originally Posted by JimmXinu

I've used three different Kindles (my current is a Voyage) and while all required mobi or azw conversion, they all handled updates and keep my place across updates as well or better than anything else I've used.

Thanks a lot, you just solidified my decision to get an Oasis. That waterpoofing and front light are things I wanted the moment I heard about them.

Do you sideload or send-to-kindle? I've noticed that using send-to-kindle for my iPhone typically ends up with just duplicating the thing and adding a new instance of it to my reader.

[JimmXinu]

Originally Posted by Ryune

Do you sideload or send-to-kindle? I've noticed that using send-to-kindle for my iPhone typically ends up with just duplicating the thing and adding a new instance of it to my reader.

I use side load exclusively.

I've added several features to FFF that integrate with Reading List to make managing updates to devices easier. It's described here. (It's been a while since I wrote that and I used a Nook at the time. Please let me know if you notice anything that needs updating.)

[Ryune]

Originally Posted by JimmXinu

I use side load exclusively.

I've added several features to FFF that integrate with Reading List to make managing updates to devices easier. It's described here. (It's been a while since I wrote that and I used a Nook at the time. Please let me know if you notice anything that needs updating.)

I actually already found that particular article. Everything worked just fine until I uploaded a book to my reader. Not sure if it was my PRS-950 or me screwing up the metadata plug but the metadata plugboard but every one of my books suddenly renamed themselves to "metadata plugboard error." You might want to consider putting the specific string of code in plain text so someone like me who doesn't exactly know what they are doing can just copy paste.

Don't worry though, I removed the metadata plug and reuploaded the new book and everything went back to normal.

[Ryune]

Sorry about the double post but it's been like 12 hours...

Right so, back with more questions!

Since I'm going to be getting an amazon reader, how exactly do I get fanficfare to update the AZW files automatically? I mean, yeah, you can set it up to only update the azw/mobi file but you recomend making an epub and then converting. Can we use fanficfare to set it up so that if we update an epub, that it then goes and converts it to the proper format?

EDIT: Never mind. I am dumb. Just ticking the "delete other formats when updating" should serve my purposes excellently.

[mehetabelo]

A couple of things.

@JimmXinu, thank you for considering it. I know that you have spent a lot of time with this program in the past, and just having it is a wonderful thing, I wont bring up that suggestion again.

@Ryune, I can verify what JimmX says, overall Kindle is a great option. I also know that my kobo has issues in the past. I do still own a Kobo, so I can check mine at home in a couple days, when I get home, to see if it has fixed the problem, but I wont be able to until then. I'm not sure how long you're willing to wait.

Kindle has it's own little issues that annoy me, though not as badly as before. One big one that drives me nuts is when sorting books into collections (ie folders) if you dont have a connection to cloud (wifi), and sideload everything (as I do - I'm a bit paranoid and hate amazon knowing everything I read), you get an error about collections not syncing to cloud and a second error when you try to load them into a folder. Now, this used to be every time you did it, but they fixed it and now it happens every ~24 hours. It's still annoying, and I have found no way to bypass it. But once those books are loaded into a folder, it stays there and updates accordingly. Most of the time when I update a book it only takes a minute or two for my kindle to update (though that depends on how many updates I put on).

If you do get a kindle and are even remotely rough on it (I take mine with me everywhere, in pockets, bags and I travel a lot). I'd highly recommend getting a case as well, it's a worthy investment. And, if the new oasis (which I would love to have) has a similar screen to the older paperwhites and other ereaders, they can crack fairly easily (just a little pressure in a small location). This is the same with any ereader I've ever had - so don't think kobo's are any better - but I noticed when I put a cover on it it got less cracks in it from being toted around in my backpack, briefcase, bags, etc. Generally the cracks aren't bad but they can be disruptive to reading as its a bright spot on the screen. I personally plan on buying a 3 year warranty next time I get an upgrade (Probably Oasis2) along with a case.

I don't like that kindle has its own format, but that's not a huge issue with calibre. If you do get a kindle, use the AZW format, rather than the old mobi whenever possible. It's very similar to epub and can handle most of the newer formatting issues mobi doesn't handle well.

There are other small issues on the kindle I've run into that I don't like, but mostly they're minor annoyances, like trying to get you to rate a book whenever you get o the end (just close the popup), but it happens anytime you get to the end, so if you update a downloaded fanfic chapter and the index isn't long enough (when it's placed at the end of the book) when you get to the end it wants you to rate a book. As I said, annoying, but rather minor. There's also no other way to keep track of length of book on a device outside your kindle. If you transfer kindles it's a matter of finding where you left off in every book - not a hard thing if you've only got a few books and your old device, or if you have great memory. Not so easy with thousands of downloaded fics, and so many crammed in your head they tend to overlap in your mind. I've had to do it a couple times over the years and used to be able to transfer the kindle files onto the new one so it would pick up the old downloaded info, not so anymore. Plus filing all the books into different 'folders' is the same way. New device? do it all over again.

But, I still buy them. Despite there issues, overall I feel they're the best ereader option and stay with them.

Now that I've quit rambling, let me know if you want me to look at that Kobo when I'm back home. I'm pretty sure they've had several updates since I last used it, so I'll update the firmware and check about how they handle updated books.