Plugin Development

[slowsmile]

@Diapdealer...Your suggestion worked a treat. The plugin is now working properly. Thanks again.

[slowsmile]

(Please see release notes for a description of the above plugin)

Hi everyone...I'm currently working, together with Doitsu, to fix various problems with the above plugin when div tags are used in various formats with the image tag lines.

The current problem -- and the last problem -- that I'm having is that when I run Granny Grump's Irving Washington epub through this plugin I'm still getting one error with this problem which is a missing end </div> tag for just one image only in dual format. All the other images in the ebook which are formatted in exactly the same way are OK. And when I test other ebooks with and without the div tag formatting I never get this problem. So this is both a consistent problem occurring with one image only in one epub and it's also a quirky problem because I can't reproduce this problem in any other epub ebook on test.

I also cannot understand why this missing div tag problem is occurring.

The plugin itself is quite small -- all it does is add the relevant media queries to the selected stylesheet and then dual format all images in the epub HTML files for KF7(in pixels) and KF8(as % values).

Here is the main driver code for the program:

Spoiler:

def processAllTasks(bk, wdir, t_ids, t_fnames, s_ids, s_fnames):
    print('\n -- Processing automatic tasks...')
    for fname in t_fnames:
        if 'cover.xhtml' in fname or 'TOC.xhtml' in fname:
            continue
        else:    
            fname = os.path.join(wdir, fname)
            reformatImageLayout(wdir, fname)   # put all the possible forms of div/img formatting into a standard input format
            reformatSmallImages(wdir, fname)   # create two separate img lines to enable dual format for Kindle KF7 and KF7 devices
            
            if options.SYS_EXIT == True:
                return(0)
                
            prettifyXHTMLFile(wdir, fname)     # prettify the html display
            fixBrokenTags(wdir, fname)         # fix any broken tags
        
    # now add the media queries to the selected stylesheet
    for file in s_fnames:
        if file == options.SET_CSS_FNAME:      
            file = os.path.join(wdir, file)
            prettifyCSS1(wdir, file)
            addMediaQueries2CSS(wdir, file)
            prettifyCSS2(wdir, file)          
         
    # write css files back to epub         
    writeFiles2Epub(bk, wdir, t_ids, t_fnames)
    writeFiles2CSS(bk, wdir, s_ids, s_fnames)        
    return(0)

And here is the main function that does all the dual formatting for the html image lines in GG's ebook(please also note that there is no div tag formatting at all in this function). This function appears to be working without any problems:

Spoiler:

def formatImages(wdir, line):
    
    # needed for later dual format processing
    original_line = line
    
    # find the image name & location
    soup = BeautifulSoup(line, 'html.parser')
    tag = soup.find('img')    
    if tag.has_attr('src'):
        text = tag['src']      
    else:
        return(line)  
    
    #get the img file name from the img path     
    file_name = os.path.basename(text)
    file_path = os.path.join(wdir, file_name)
     
    # get the image dimensions
    width, height = getImageSize(file_path)                             # uses the PIL 'image' method to get image dimensions from the bitmap
    perc_width = round(width/int(options.BASE_WIDTH_VALUE) * 100)       # calculates width as a percentage of screen width
    perc_height = round(height/1200 * 100)                              # calculates height as a percentage of screen height
    
    # reduce image widths to 100% screen width 
    # for all outsize images
    if perc_width >= 100:
        perc_width = 100
        
    if perc_height >= 100:
        perc_height = 100
        
    # precaution in case image html name contains spaces(not allowed in epubs)    
    file_name = os.path.split(file_name)[1]
    file_name = file_name.replace(' ', '_')
    
    ### Now dual format the image line ###
    
    # insert the % height and width image values for KF8 devices
    soup = BeautifulSoup(line, 'html.parser')
    img = soup.img
    if img.has_attr('class'):
        del img['class']                      # remove all classes from the img tags 
    if not img.has_attr('style'):             
        if img.has_attr('width'):
            del img['width']
        if img.has_attr('height'):
            del img['height']            
        img['style'] = 'width: ' + str(perc_width) + '%;height: auto;'     # add the calculated % values into the inline style
    else:
        img['style'] = 'width: ' + str(perc_width) + '%;height: auto;'     # delete then re-add the calculated % values into the inline style   
    line = str(soup)
    
    # include css @media classes for kf8 devices only
    liner = BeautifulSoup(line, 'html.parser')
    img1 = soup.img
    if img1.has_attr('class'):
        del img1['class']            # remove any classes
        img1['class'] = 'kf8only'    # add the appropriate @media class
    else:    
        img1['class'] = 'kf8only'    # add the appropriate @media class if there are no classes present
        
    if img1.has_attr('id'):          # remove ids from the image tag -- not required(avoids id duplication errors on Epubcheck)
        del img1['id']    
        
    l = str(soup)
    line2 = l + '\n'
    
    # format pixel image values for mobi(KF7) version only using 
    # the img tag html attributes for height and width in pixels
    soup = BeautifulSoup(original_line, 'html.parser')
    img2 = soup.img
    
    # remove all inline styling from the img tag
    if img2.has_attr('style'):
        del img2['style']
    
    # remove all classes and then add the appropriate @media classes for mobi only
    if img2.has_attr('class'):
        del img2['class']
        img2['class'] = 'mobionly'
    else:
        img2['class'] = 'mobionly'
    
    # remove and then add the appropriate 'px' values for h nd w
    # in pixels for mobi version only 
    if img2.has_attr('width'):
        del img2['width'] 
        img2['width'] = str(width) + 'px'
    else: 
        img2['width'] = str(width) + 'px'   
    
    if img2.has_attr('height'):
        del img2['height']
        img2['height'] = str(height) + 'px'
    else:
        img2['height'] = str(height) + 'px'    

    if img2.has_attr('id'):
        del img2['id']            
    
    # now join these two line as one(they already contain newline end chars)
    line1 = str(soup)
    line = line1 + line2
    
    return(line)

Also, both the reformatDivLayout() and fixBrokenTags() functions were added after we got the missing div tag problem. Similarly the prettifyXHTMLFiles() function also has nothing to do with this problem because I've already used this function in all my other plugins without any problems whatsoever. So these functions have nothing to do with the missing div tag problem.

Below you can also download the new version of this plugin(v0.1.5 -- which is only for test purposes only and not released yet) as well as Doitsu's test epub so that you can see the missing div tag problem on Epubcheck for yourself. The test epub by Granny Grump is called Irving,Washington-LegendOfSleepyHollow-illus-Rackham.epub.

[slowsmile]

The above missing div tag problem can be fixed simply by running Sigil's Tool > Mend All HTML Files. But it would still be nice to know why this problem is occurring.

I'm also pretty sure that I could cure the above problem by using Tidy and setting the wrap option to off. My own suspicions about this problem are that it might be happening because of wrapping problems. Just a wild guess. But I'm trying to avoid using Tidy because libtidy and the associated dll are large files and Tidy also has its own set of peculiar quirks. And if I use Tidy then it would be like using a sledgehammer to crack a walnut to cure this problem.

But any other ideas or thoughts concerning this problem would be welcome.

[Doitsu]

I haven't looked at your code in detail, but it looks like you're using both bs4 and regular expressions, which, IMHO, isn't a good idea.

The invalid code that Sigil is reporting is caused by the following lines in LSH.xhtml

before:

<div class="illus">
<img alt="" class="img065" src="../Images/lsh-23-065.png"/></div>

after:

<div class="illus">

<img alt="" class="mobionly" height="441px" src="../Images/lsh-23-065.png" width="630px"/>

<img alt="" class="kf8only" src="../Images/lsh-23-065.png" style="width: 100%;height: auto;"/>

For some reason your code removed the closing </div> tag.

I don't know what the exact cause is, but since you seem to be using regular expressions, it might help to normalize the HTML input using the Sigil bs4 prettyprint_xhtml function. When I ran the following plugin prior to running yours, I didn't get any error messages:

Spoiler:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from sigil_bs4 import BeautifulSoup

def run(bk):
    html_id = 'LSH.xhtml'
    html = bk.readfile(html_id)
    soup = BeautifulSoup(html, 'html.parser')
    normalized_html = str(soup.prettyprint_xhtml(indent_level=0, eventual_encoding="utf-8", formatter="minimal", indent_chars="  "))
    bk.writefile(html_id, normalized_html)

    return 0

def main():
    print('I reached main when I should not have\n')
    return -1

if __name__ == "__main__":
    sys.exit(main())

[slowsmile]

@Doitsu...Yes, I agree with you on all counts. You really shouldn't use regex in code, couldn't agree more. But this is not a standard fault that can be spotted or cured easily -- so I thought that I would just try the regex and see if it would work(it didn't work). Remember, there are alot of other image tags in GG's LHS.xhtml file that are also formatted with divs around image tags in the same way. So, if it was my code(and I'm not saying its not), then why are all those other image tags dual formatted correctly in GG's epub after running the plugin? And why can't I reproduce this fault in any other ebook using image tags within div tags? That's what makes this problem so weird and strange.

I've also had strange wrap problems like this before in the past where it wasn't the code but was caused by wrap errors. And I really don't want to use Tidy again to cure this, not keen on that either because that's overkill. The wrapping thing is a wild guess by the way. But the only way way I can prove that that problem is caused by wrapping problems is by using the Tidy module and setting the wrap option to OFF to see if that cures the problem. I might do that later(bit busy now). But I'll let you know the outcome of this testing with Tidy when I can.

[DiapDealer]

I'll try to take a peek if I get some time. The gist is that the original (x)html is well-formed, and that the plugin is dropping a closing div tag under certain conditions, right?

[slowsmile]

I think that I might have found the cause of the missing div tag problem when the plugin is run on GG's epub. To be clear about my conclusions, I now believe that the HTML Parser module, which is always used in my own plugin code with bs4 is the main reason and culprit for the missing div tag. See below for reasons.

My bs4 code always uses 'html.parser' and its the only parser that I use for epub html. I use this parser for two reasons. First, this parser is the correct parser to use because it uses the same version of html -- html 4.01 -- as epub 2. Second, 'html.parser' is the only bs4 parser that actually fixes html errors.

Here's what I did in the test. In turn, I changed all my BeautifulSoup parser declarations in the reformatSmallImages() and formatImages() functions in my code to using the 'lxml', 'html5' and no parser and ran the plugin on GG's epub. In all instances of testing these other parsers with GG's epub I got a whole bunch of errors. But the missing </div> tag error was not among any of the errors for any of these other parsers. When I changed the parser back to using bs4 with 'html.parser' and ran the test again then that single missing </div> tag problem returned after the plugin was again run. I also ran all these same test on Bundled Python and ran them all again using my own Python version on my computer(Python 3.4.4) with exactly the same results.

Therefore it would seem that the HTML Parser module that bs4 uses is the cause of this missing div tag problem.

If someone can also confirm the above test result then obviously this problem can't be fixed and I should perhaps advise another caveat in my plugin release advising the plugin user to run Sigil's Tools > Mend All HTML Files option which will fix any missing div tags problem if it occurs when running the plugin.

[Doitsu]

Originally Posted by slowsmile

Therefore it would seem that the HTML Parser module that bs4 uses is the cause of this missing div tag problem.

I don't think that the HTML Parser module causes this problem, because, when I preprocessed the HMTL with the very same module, your plugin no longer created invalid HTML files. Also I used it in a couple of plugins and it never produced invalid HTML code.

Run the following prepossessing code, before running your plugin and you'll see that missing <div> error no longer occurs:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from sigil_bs4 import BeautifulSoup

def run(bk):
    html_id = 'LSH.xhtml'
    html = bk.readfile(html_id)
    soup = BeautifulSoup(html, 'html.parser')
    normalized_html = str(soup.prettyprint_xhtml(indent_level=0, eventual_encoding="utf-8", formatter="minimal", indent_chars="  "))
    bk.writefile(html_id, normalized_html)

    return 0

def main():
    print('I reached main when I should not have\n')
    return -1

if __name__ == "__main__":
    sys.exit(main())

[DiapDealer]

I've not investigated extensively yet, but I'm able to change the source code:

<div class="illus">
<img alt="" class="img065" src="../Images/lsh-23-065.png"/></div>

to:

<div class="illus">
<img alt="" class="img065" src="../Images/lsh-23-065.png"/>
</div>

(simply dropping the closing div tag to its own line) to make the plugin work just fine. Even a quick mend and prettify BEFORE running the plugin ensures that your plugin doesn't drop the closing div.

It handles all conditions where the opening and closing divs around images are on their own lines just fine.

[DiapDealer]

In formatImages() in cutils.py, you're souping a single line of xhtml that has an invalid closing div tag at the end of it. What did you think the parser would do with the extraneous closing tag?

soup = BeautifulSoup(line, 'html.parser')

where "line" is:

<img alt="" class="img065" src="../Images/lsh-23-065.png"/></div>

Will definitely drop the closing div.

This code:

line = str(soup)

commits the divless soup back to the xhtml.

You need to find a better way of isolating the img tag and writing the modified one(s) back without affecting the surrounding code.

The attached python script should make what's happening clear.

[slowsmile]

@DiapDealer and @Doitsu...Thanks for testing and for your opinions.

You are both right -- it isn't the BS html parser causing the problem. I did some further checks to confirm this. After I ran the plugin I then checked the plugin working directory containing all the epub files. When I ran the LHS.xhtml(in the working directory) in Chrome and looked at the source code this showed the same missing div error as before. So this does seem to confirm that this problem is somewhere in my plugin code.

To help confirm the problem area in my code, I think it would perhaps be helpful if I show recently added functions that are not involved with the missing div problem.

I have recently added the following functions to my code:

reformatImageLayout()
fixBrokenTags()

I added these two functions in after getting the missing div tag error. I decided to initially reformat the div tags as a possible way of curing the missing div problem and standardize the input. Similarly, I wrote a regex function that repaired broken tags to fix this problem.These two functions made no difference regarding this problem(ie they have neither caused nor cured the problem). The only function that actually formats div tags in the plugin is the reformatImageLayout() function and, as I've already said, that function was added in as a possible fix after I started getting the missing div problem in GG's epub.

I've also thoroughly checked my plugin code and there is nowhere else in the plugin code where the code changes or removes div tags. So I am still in the dark as to why the plugin code only gives one missing div tag error on line 322 of the LHS.xhtml file while the other images in GG's epub, which are all formatted in exactly the same way as the image with the missing div tag, have no missing div tag errors. This fact would seem to actually contradict the belief that it is the plugin code that is causing the single missing div tag problem because if it was my code then wouldn't all the aforementioned images have the same missing div tag problem as well? After all, couldn't this missing div problem also be due to a problem in a used module or a wrapping problem that really has nothing to do with my actual plugin code?

I might try using libtidy to confirm whether missing div tag error is a wrapping problem. As I've already said, this isn't going to be an easy error to locate or fix. So later I will try adding Tidy with the wrap option set to OFF in the plugin to see if this cures the problem. Can't do it now -- I have to dash off again.

[DiapDealer]

I told you exactly where the problem lies. It's in formatImages(). Was I unclear? You're using bs4 to parse single lines of an xhtml file with the assumption that they will always be self-contained, well-formed snippets of xhtml. Doing so will cause data-loss when it is not. The problematic line of code in LHS.xhtml will not parse without data-loss. In this case, the closing div tag in question gets clobbered.

[KevinH]

Yes, just as DiapDealer said, you need to extract just the image tag not the entire line before passing it to soup to parse/fix. Having a closing div tag on the end of that line and passing it to soup will remove it since there is no matching starting div in the piece of code you passed to it.

[slowsmile]

@DiapDealer and Doitsu...Yes, of course DiapDealer was right. In fact I had already implemented his suggestion because reformatImageLayout() was specifically written to provide a standard input image line format for the main formatImages() function.

But when I went back and saw the offending image line (with the missing image tag) in a fresh copy GG's epub in Sigil, I saw this:

<div class="illus">
<img alt="" class="img065" src="../Images/lsh-23-065.png"/></div>

Unfortunately, my code in the reformatImageLayout() function assumed that you should expect only one form of inline div/img tags i.e. inline only. I've now changed the code to expect all forms and GG's epub now goes through the plugin without any missing div tag problems.

My grateful thanks to you both for helping me out with this last problem.

@Doitsu...I will release the new version -- AddKindleMediaQueries(v0.1.5) --as soon as I can after some more testing. I'll let you know when the new version is released so you can test it in your own way. Thanks again.

[Doitsu]

My simple epub2 output plugin uses ncxgenerator.py from the python3lib folder to optionally generate a new toc.ncx file from nav.xhtml.
(I know that the Python files in the python3lib folder are not intended for use with plugins, however, some of the functions might actually be helpful in special cases, such as my epub2 output plugin.)
Since I couldn't figure out a good cross-platform method for adding python3lib to sys.path, I simply bundled ncxgenerator.py with my plugin.
Even though I don't think that ncxgenerator.py will be significantly updated in future Sigil versions, I think that it'd be better to add python3lib to sys.path instead of bundling ncxgenerator.py with the plugin.

1. Would it be possible to add python3lib to sys.path in one of the next Sigil versions?

or

2. Is there a robust cross-platform method for adding python3lib to sys.path that I overlooked?