Automate Ad Page removal by filename

[LostOnTheLine]

I've found a bunch of books that I'm adding contain a distributor ad at the end of the book.

• The ad page is always the same filename signup.xhtml
• They contain different text but always have the words "Sign up for our mailing list" on the page

I ran into another distributor who has the same signup.xhtml but the page is completely different.

In any case I always want to remove the page. So far I do it by manually [right-click] > [delete] but since it's the same page name there should be an easier way for me to add it to my automation list.

I currently run an automation list on just about every book I add, with things like

• RunSavedSearchReplaceAll ' and ' to ' & '
• RunSavedSearchReplaceAll 'have ourselves a cake & eat a cake' to 'have ourselves a cake & eat it too'
• RunSavedSearchReplaceAll <<REPEATED WITH A DOZEN SIMILAR COMMON TRANSLATION ERRORS>>
• ImgShrinker
• MedPrettifyHTML
• StandardizeEpub

Is there any way to add a removal of all files named signup.xhtml?

I know the default tools I can add have options for DeleteUnusedMedia & DeleteUnusedStyles & RemoveNCXGuideFromEpub3 & there's the MendHTML & MendPrettifyHTML would it be possible to perhaps modify one of those to do it? I assume not since they won't even change books that have the files stupidly named .html instead of .xhtml but maybe it's possible. There's also the option to call plugins, maybe there's a plugin that has the ability to (even if it's not the intent of the plugin) delete a page by name?

My hope is that once I get this to work I'll put it in my automation list, then after it it'll remove unused media & get rid of the images on the ad pages as well, but even if it can't do that I'll consider it a success if I can delete the page.

[BeckyEbook]

Use a tiny plugin that will remove such a single file.

[LostOnTheLine]

Originally Posted by BeckyEbook

Use a tiny plugin that will remove such a single file.

Thanks a bunch. I looked but didn't find it. I'm surprised it doesn't have a preferences file to change what gets removed, but I guess enough people dislike the signup page specifically that it warranted a tool just for it, lol.

[KevinH]

I think BeckyEbook just wrote that for you (and others) to see how a simple plugin can be easily created and used.

[LostOnTheLine]

Originally Posted by KevinH

I think BeckyEbook just wrote that for you (and others) to see how a simple plugin can be easily created and used.

Ah. That'd explain why I couldn't find one, lol.

For me starting from scratch it's a no-go, I've never enjoyed programing, I can get by, but I don't know where to start. Editing & adjusting I can do. Like giving ImgShrinker more Options, I had something to start with & I can use that to make it do what I want that wasn't originally part of the options. I was hoping for something like that, a plugin that removed something as part of it's thing that I could extract the parts I needed & modify them to my desires, then get rid of anything extra I don't need.
Like I modified the plugin she provided (Even more thanks if you actually created it) to delete any page that contains the line "Sign up for our mailing list" so if a page has a different name it can work, & to work if the name is signup.xhtml or Signup.xhtml or SignUp.xhtml or SIGNUP.xhtml or SIGNUP.XHTML. I plan to tweak it further to work for .html files as well, & possibly add a preferences JSON file to add a list of page names to automatically remove, plus a list of lines that the page shoud be removed if it contains.

I'm also trying to work out a way to do something I was already told on here wasn't possible, remove blank pages. So far I've yet to succeed, but with this I feel like I'm a lot closer than anything before has ever been.

All that I can figure out so long as I have a starting point. But without a starting point I'm pretty much useless for things like that

[BeckyEbook]

This few-line script was supposed to be something like that – a starting point for your own extensions. I could only hope you’d be involved enough to develop it into something perfectly suited to your needs.
If you can automate repetitive tasks, that's what Sigil scripts are for

[Doitsu]

Originally Posted by LostOnTheLine

I'm also trying to work out a way to do something I was already told on here wasn't possible, remove blank pages. So far I've yet to succeed, but with this I feel like I'm a lot closer than anything before has ever been.

That requires only minimal code changes to Becky's plugin:

Spoiler:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from sigil_bs4 import BeautifulSoup

def run(bk):
    for html_id, href in bk.text_iter():
        html = bk.readfile(html_id)
        soup = BeautifulSoup(html, 'html.parser')
        body_text = soup.body.text.strip()
        if body_text == '' or len(body_text) <= 6:
            print('INFO: Removing {}... '.format(href))
            bk.deletefile(html_id)
    print('\nPlease click OK to close the Plugin Runner window.')
    return 0

def main():
    print("I reached main when I should not have\n")
    return -1

if __name__ == "__main__":
    sys.exit(main())

Please note that the plugin will crash Sigil if you use it with an epub that only contains blank pages.

[KevinH]

It should not crash Sigil. Sigil's plugin runner should make sure that at least one file remains at all times. The plugin runner code explicitly is supposed to detect and prevent that.

I will look at it.

The pluginrunner process can fail but nothing the plugin does should cause Sigil itself to actually crash in an unrecoverable way.

Update: The PluginRunner has a bug:

    // don't allow changes to proceed if they will remove the very last xhtml/html file
    if (m_xhtml_net_change < 0) {
        QList<Resource *> htmlresources = m_book->GetFolderKeeper()->GetResourceListByType(Resource::HTMLResourceType);
        if (htmlresources.count() + m_xhtml_net_change < 0) {
            Utility::DisplayStdErrorDialog(tr("Error: Plugin Tried to Remove the Last XHTML file .. aborting changes"));
            ui.statusLbl->setText(tr("Status: No Changes Made"));
            m_result = "failed";
            return;
        }
    }

The test should have been:

if (htmlresources.count() + m_xhtml_net_change <= 0) {

I will fix this in Sigil master.

[KevinH]

A fix for that bug was just pushed to master and will appear in the next release. This change will prevent a plugin from crashing Sigil by deleting the last xhtml file.

[LostOnTheLine]

Originally Posted by Doitsu

That requires only minimal code changes to Becky's plugin:

Spoiler:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from sigil_bs4 import BeautifulSoup

def run(bk):
    for html_id, href in bk.text_iter():
        html = bk.readfile(html_id)
        soup = BeautifulSoup(html, 'html.parser')
        body_text = soup.body.text.strip()
        if body_text == '' or len(body_text) <= 6:
            print('INFO: Removing {}... '.format(href))
            bk.deletefile(html_id)
    print('\nPlease click OK to close the Plugin Runner window.')
    return 0

def main():
    print("I reached main when I should not have\n")
    return -1

if __name__ == "__main__":
    sys.exit(main())

Please note that the plugin will crash Sigil if you use it with an epub that only contains blank pages.

This actually doesn't work...
I mean it does work, but it works too well...

It removes the blank pages, but it also removes non-blank pages, particularly pages that have only an image on them, which, in my case, is what I'm trying to get that sometimes ends up creating a blank page.

I have an automate that works perfectly for some books, particularly ones that I regularly get each volume of for a couple series as they all come from the same publisher. It find the images that are inline, which sometimes display not ideally, especially after I run ImgShrinker on them, & adds a split marker, then splits at the split marker.
Having these images on their own page is generally how I want my books, & I'll run the automate on other publishers stuff & check them, most of the time they work well, with only a few having images that are supposed to be part of a page getting split as well. But half the time the books have some, but not all images already on their own page, in which case they end up with a 2nd blank page.
Certain publishers always separate the full-page images so I have to have a 2nd automate for those that skips that part, but I'd prefer to have 1 that I can use more universally. That's my main goal for the "remove-blank-pages" thing. Unfortunately with this one it ends up making all the images on their own page, then removing all the pages that are blank as well as all those with just images.

Thanks for the attempt, but anyone looking into this should be aware that it will remove pages with just an image, like most cover pages, as well as the actual blank ones

When I have time I will try again to make it work & hopefully this will make that easier

[JSWolf]

There's no way to remove all ad pages by filename because the filenames are not all the same. It's a lot easier (and safer) to do the removal by hand.

[KevinH]

Yes, that is what the code Doitsu wrote does. If you do not want to delete pages with img tags and or svg tags then you need to modify that code to check for those tags before deleting.

Something like the following modification to Doitsu's code could be used to detect img and svg tags and prevent deletion if either is found:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from sigil_bs4 import BeautifulSoup

def run(bk):
    for html_id, href in bk.text_iter():
        html = bk.readfile(html_id)
        soup = BeautifulSoup(html, 'html.parser')
        body_text = soup.body.text.strip()
        node = soup.body
        if body_text == '' or len(body_text) <= 6:
            if not node.img and not node.svg:
                print('INFO: Removing {}... '.format(href))
                bk.deletefile(html_id)
    print('\nPlease click OK to close the Plugin Runner window.')
    return 0

def main():
    print("I reached main when I should not have\n")
    return -1

if __name__ == "__main__":
    sys.exit(main())

Please note: I did NOT test this at all because all I have access to is an ipad.

[LostOnTheLine]

Originally Posted by JSWolf

There's no way to remove all ad pages by filename because the filenames are not all the same. It's a lot easier (and safer) to do the removal by hand.

I didn't say I want it to detect & remove all ad pages, I specifically said that I noticed that the signup page was used across multiple distributors, & that since it has the same name, & I've yet to see a signup page that needed to be kept, I want to delete those pages specifically. The end result, when I finish making the modifications I want, is to have a way to add the file names that are to be automatically deleted, & to add strings of words that, if present on a page, the whole page should be deleted.

It isn't "a lot easier" to manually do it. I add at least 2 books a month that have the same page that needs to be deleted. Sometimes I start a series that already has 20 volumes & get all the books in bulk at the same time. Manually going in & deleting the pages is tedious & unnecessary as I now have, from this thread, an automation step that removes them automatically for me.

As far as "safer" well yeah, but I'm fairly confident that I'm not going to find a book with a signup page that I DON'T want to delete, & if I accidentally delete one, I always have the original I can use or just re-download a fresh copy from my account. I understand the risk, & it's a known risk, taken willingly & I'm okay with that

[Hitch]

Originally Posted by LostOnTheLine

I didn't say I want it to detect & remove all ad pages, I specifically said that I noticed that the signup page was used across multiple distributors, & that since it has the same name, & I've yet to see a signup page that needed to be kept, I want to delete those pages specifically. The end result, when I finish making the modifications I want, is to have a way to add the file names that are to be automatically deleted, & to add strings of words that, if present on a page, the whole page should be deleted.

Alrighty, well...you need a clip for that, pretty much.

It isn't "a lot easier" to manually do it. I add at least 2 books a month that have the same page that needs to be deleted. Sometimes I start a series that already has 20 volumes & get all the books in bulk at the same time. Manually going in & deleting the pages is tedious & unnecessary as I now have, from this thread, an automation step that removes them automatically for me.

Wait....this is about two books a month? TWO books a month? Because it's more time-consuming to click, select, delete?

As far as "safer" well yeah, but I'm fairly confident that I'm not going to find a book with a signup page that I DON'T want to delete, & if I accidentally delete one, I always have the original I can use or just re-download a fresh copy from my account. I understand the risk, & it's a known risk, taken willingly & I'm okay with that

Meh, safer, you're always working in a copy anyway, ight? So even if you screw the pooch, you can revert and undo the damage. I have working files in Dropbox for that reason (I have a Enterprise account) that allows me to rollback versions, for that reason, too--just in case. Just in case I screw up and overwrite the "live" file, or whatever, I can version my way back to a safe shore.

You could take several of these clips and modify them and be done with it, but at this point, you've probably put more time in, trying to figure it out than you would click-select-deleting 200 of them. :-) I know the feeling, Regex hasn't always been my buddy, either.

Hitch

[JSWolf]

The thing is, eBooks with this file titled signup has other files I'll want to delete. So it's just easier to delete all the files you don't want by hand (IMHO).