Calibre 6.11: Conversion from AZW3 to EPUB3 Removing Used Images

[enuddleyarbl]

The conversion process in Calibre 6.11 seems to be removing an image that is actually in use in a book. This is the third book in a row, now that it's happened. I wasn't sure about the previous two times, but I paid more attention this time. This particular book is Alexander McCall Smith's "Morality for Beautiful Girls" (the third in his No. 1 Ladies' Detective Agency series):

https://www.amazon.com/gp/product/B0...w_myk_ro_title

The previous two times were for the first and second books in the series.

I downloaded a fresh copy of the AZW3 from my library at Amazon, Added it to my working Calibre library and converted it individually to an EPUB3. Immediately afterward, I edited the result and noted broken image links (specifically, those used in the Chapter titles). Comparing the list of images in the AZW3 to those present in the EPUB3 I get:

00013.jpeg (in use in chapter titles, but missing from EPUB)
00014.jpeg (in use in title page, present in EPUB)
00015.jpeg (in use as the cover page, renamed to cover.jpeg in EPUB)
00016.jpeg (in use in About the Author file, present in EPUB)
00019.jpeg (in use in end of book ads, present in EPUB)
00020.jpeg (this and all the rest not used and correctly removed)
00021.jpeg
00022.jpeg
00023.jpeg
00025.jpeg

So, it's just the image used in the chapter titles that incorrectly removed (ditto for the previous two books). The lines in the source chapters referencing that image look fine to me:

<p class="cn"><span id="c02"></span><b>CHAPTER TWO</b><br/><img src="../images/00013.jpeg" alt="image"/></p>

If I manually copy that image file from the AZW3 editor instance to the EPUB3 editor instance, it immediately shows up correctly in the chapter titles of the EPUB.

The Job Details for the conversion follows. But, it looks to me like it's just correctly handling the cover and trimming the unused images. I don't see any reference to 00013.jpeg in there (but, I might have missed it):

Convert book 1 of 1 (Morality for Beautiful Girls)
DeDRM v10.0.3: Trying to decrypt rjbj_mx1.azw3
Using Library AlfCrypto DLL/DYLIB/SO
MobiDeDrm v1.1.
Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: Morality for Beautiful Girls (No 1. Ladies' Detective Agency Book 3)
Found 6 keys to try after 0.0 seconds
Crypto Type is: 0
This book is not encrypted.
Decryption succeeded after 0.0 seconds
DeDRM v10.0.3: Finished after 0.0 seconds
Conversion options changed from defaults:
  replace_scene_breaks: '<hr />'
  margin_bottom: 0.0
  enable_heuristics: True
  margin_top: 0.0
  markup_chapter_headings: False
  verbose: 2
  cover: 'C:\\Users\\Dave\\AppData\\Local\\Temp\\calibre_0q52gfog\\ugbrj2s9.jpeg'
  output_profile: 'tablet'
  smarten_punctuation: True
  remove_first_image: True
  expand_css: True
  minimum_line_height: 0.0
  insert_metadata: True
  filter_css: 'line-height'
  read_metadata_from_opf: 'C:\\Users\\Dave\\AppData\\Local\\Temp\\calibre_0q52gfog\\7mq1buiz.opf'
  disable_font_rescaling: True
  epub_version: '3'
  margin_right: 0.0
  preserve_cover_aspect_ratio: True
  margin_left: 0.0
Resolved conversion options
calibre version: 6.11.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0.0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': "//*[((name()='h1' or name()='h2') and re:test(., "
            "'\\s*((chapter|book|section|part)\\s+)|((prolog|prologue|epilogue)(\\s+|$))', "
            "'i')) or @class = 'chapter']",
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': 'C:\\Users\\Dave\\AppData\\Local\\Temp\\calibre_0q52gfog\\ugbrj2s9.jpeg',
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': True,
 'dont_split_on_page_breaks': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': True,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_toc_at_end': False,
 'epub_version': '3',
 'expand_css': True,
 'extra_css': None,
 'extract_to': None,
 'filter_css': 'line-height',
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001212ADDDED0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': True,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0.0,
 'linearize_tables': False,
 'margin_bottom': 0.0,
 'margin_left': 0.0,
 'margin_right': 0.0,
 'margin_top': 0.0,
 'markup_chapter_headings': False,
 'max_toc_links': 50,
 'minimum_line_height': 0.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.TabletOutput object at 0x000001212ADDC9D0>,
 'page_breaks_before': "//*[name()='h1' or name()='h2']",
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': True,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': 'C:\\Users\\Dave\\AppData\\Local\\Temp\\calibre_0q52gfog\\7mq1buiz.opf',
 'remove_fake_margins': True,
 'remove_first_image': True,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '<hr />',
 'search_replace': '[]',
 'series': None,
 'series_index': None,
 'smarten_punctuation': True,
 'sr1_replace': None,
 'sr1_search': None,
 'sr2_replace': None,
 'sr2_search': None,
 'sr3_replace': None,
 'sr3_search': None,
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': '[]',
 'transform_html_rules': '[]',
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
DeDRM v10.0.3: Trying to decrypt jc2h9k1y.azw3
MobiDeDrm v1.1.
Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: Morality for Beautiful Girls (No 1. Ladies' Detective Agency Book 3)
Found 6 keys to try after 0.0 seconds
Crypto Type is: 0
This book is not encrypted.
Decryption succeeded after 0.0 seconds
DeDRM v10.0.3: Finished after 0.0 seconds
InputFormatPlugin: MOBI Input running
on C:\Users\Dave\AppData\Local\Temp\calibre_0q52gfog\dmzu44me.azw3
Found KF8 MOBI of type 'standalone'
Extracting text...
Parsing all content...
Parsing text/part0000.html ...
*********  Heuristic processing HTML  *********
flow is too short, not running heuristics
Parsing text/part0018.html ...
*********  Heuristic processing HTML  *********
There are 4 blank lines. 0.042105263157894736 percent blank
Hard line breaks check returned False
Median line length is 100, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0025.html ...
*********  Heuristic processing HTML  *********
flow is too short, not running heuristics
Parsing styles/0002.css ...
Parsing text/part0013.html ...
*********  Heuristic processing HTML  *********
There are 2 blank lines. 0.022988505747126436 percent blank
Hard line breaks check returned False
Median line length is 141, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0022.html ...
*********  Heuristic processing HTML  *********
There are 1 blank lines. 0.0196078431372549 percent blank
Hard line breaks check returned False
Median line length is 100, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0005.html ...
*********  Heuristic processing HTML  *********
There are 0 blank lines. 0.0 percent blank
Hard line breaks check returned False
Median line length is 94, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0009.html ...
*********  Heuristic processing HTML  *********
There are 1 blank lines. 0.025 percent blank
Hard line breaks check returned False
Median line length is 123, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0015.html ...
*********  Heuristic processing HTML  *********
There are 2 blank lines. 0.025974025974025976 percent blank
Hard line breaks check returned False
Median line length is 161, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0019.html ...
*********  Heuristic processing HTML  *********
There are 3 blank lines. 0.03614457831325301 percent blank
Hard line breaks check returned False
Median line length is 114, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0027.html ...
*********  Heuristic processing HTML  *********
flow is too short, not running heuristics
Parsing text/part0014.html ...
*********  Heuristic processing HTML  *********
There are 3 blank lines. 0.046153846153846156 percent blank
Hard line breaks check returned False
Median line length is 107, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing styles/0005.css ...
Parsing text/part0001.html ...
*********  Heuristic processing HTML  *********
There are 0 blank lines. 0.0 percent blank
Hard line breaks check returned True
Median line length is 48, calculated with html format
Unwrapping required, unwrapping Lines
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0002.html ...
*********  Heuristic processing HTML  *********
There are 0 blank lines. 0.0 percent blank
Hard line breaks check returned True
Median line length is 91, calculated with html format
Unwrapping required, unwrapping Lines
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0010.html ...
*********  Heuristic processing HTML  *********
There are 4 blank lines. 0.07142857142857142 percent blank
Hard line breaks check returned False
Median line length is 190, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0016.html ...
*********  Heuristic processing HTML  *********
There are 1 blank lines. 0.015384615384615385 percent blank
Hard line breaks check returned False
Median line length is 116, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0003.html ...
*********  Heuristic processing HTML  *********
flow is too short, not running heuristics
Parsing text/part0007.html ...
*********  Heuristic processing HTML  *********
There are 1 blank lines. 0.020833333333333332 percent blank
Hard line breaks check returned False
Median line length is 100, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0021.html ...
*********  Heuristic processing HTML  *********
There are 3 blank lines. 0.045454545454545456 percent blank
Hard line breaks check returned False
Median line length is 124, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing styles/0003.css ...
Parsing styles/0004.css ...
Parsing text/part0006.html ...
*********  Heuristic processing HTML  *********
There are 1 blank lines. 0.012195121951219513 percent blank
Hard line breaks check returned False
Median line length is 137, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0008.html ...
*********  Heuristic processing HTML  *********
There are 3 blank lines. 0.023076923076923078 percent blank
Hard line breaks check returned False
Median line length is 159, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0011.html ...
*********  Heuristic processing HTML  *********
There are 2 blank lines. 0.021052631578947368 percent blank
Hard line breaks check returned False
Median line length is 126, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0024.html ...
*********  Heuristic processing HTML  *********
There are 0 blank lines. 0.0 percent blank
Hard line breaks check returned True
Median line length is 43, calculated with html format
Unwrapping required, unwrapping Lines
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0017.html ...
*********  Heuristic processing HTML  *********
There are 2 blank lines. 0.028985507246376812 percent blank
Hard line breaks check returned False
Median line length is 108, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing styles/0006.css ...
Parsing text/part0023.html ...
*********  Heuristic processing HTML  *********
flow is too short, not running heuristics
Parsing text/part0028.html ...
*********  Heuristic processing HTML  *********
flow is too short, not running heuristics
Parsing text/part0026.html ...
*********  Heuristic processing HTML  *********
There are 0 blank lines. 0.0 percent blank
Hard line breaks check returned False
Median line length is 164, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing styles/0001.css ...
Parsing text/part0004.html ...
*********  Heuristic processing HTML  *********
There are 2 blank lines. 0.024096385542168676 percent blank
Hard line breaks check returned False
Median line length is 155, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0012.html ...
*********  Heuristic processing HTML  *********
There are 0 blank lines. 0.0 percent blank
Hard line breaks check returned False
Median line length is 118, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Parsing text/part0020.html ...
*********  Heuristic processing HTML  *********
There are 1 blank lines. 0.017241379310344827 percent blank
Hard line breaks check returned False
Median line length is 104, calculated with html format
Fixing hyphenated content
Formatting scene breaks
Reading TOC from NCX...
Removed first image
Merging user specified metadata...
Detecting structure...
Inserting metadata into book...
Flattening CSS and remapping font sizes...
Filtering CSS properties: line-height
Source base font size is 11.99880pt
Removing fake margins...
Found 30 items of level: div_1
Found 39 items of level: div_2
Found 2 items of level: div_3
Found 4 items of level: p_4
Found 1496 items of level: p_2
Found 45 items of level: p_5
Found 2 items of level: p_1
Ignoring level p_1
Ignoring level p_4
Ignoring level div_3
div_1  left margin stats: Counter({'': 25, '0': 1})
div_1  right margin stats: Counter({'': 25, '0': 1})
div_2  left margin stats: Counter({'': 1})
div_2  right margin stats: Counter({'': 1})
Negative text indent detected at level  p_2, ignoring this level
p_5  left margin stats: Counter({'0': 45})
p_5  right margin stats: Counter({'0': 45})
Cleaning up manifest...
Trimming unused files from manifest...
Trimming 'images/00022.jpeg' from manifest
Trimming 'images/00020.jpeg' from manifest
Trimming 'images/00025.jpeg' from manifest
Trimming 'images/00021.jpeg' from manifest
Trimming 'images/00023.jpeg' from manifest
Creating EPUB Output...
Splitting markup on page breaks and flow limits, if any...
	Looking for large trees in text/part0000.html...
	No large trees found
	Looking for large trees in text/part0018.html...
	No large trees found
	Looking for large trees in text/part0025.html...
	No large trees found
	Looking for large trees in text/part0013.html...
	No large trees found
	Looking for large trees in text/part0022.html...
	No large trees found
	Looking for large trees in text/part0005.html...
	No large trees found
	Looking for large trees in text/part0009.html...
	No large trees found
	Looking for large trees in text/part0015.html...
	No large trees found
	Looking for large trees in text/part0019.html...
	No large trees found
	Looking for large trees in text/part0027.html...
	No large trees found
	Looking for large trees in jacket.xhtml...
	No large trees found
	Looking for large trees in text/part0014.html...
	No large trees found
	Looking for large trees in text/part0001.html...
	No large trees found
	Looking for large trees in text/part0002.html...
	No large trees found
	Looking for large trees in text/part0010.html...
	No large trees found
	Looking for large trees in text/part0016.html...
	No large trees found
	Looking for large trees in text/part0003.html...
	No large trees found
	Looking for large trees in text/part0007.html...
	No large trees found
	Looking for large trees in text/part0021.html...
	No large trees found
	Looking for large trees in text/part0006.html...
	No large trees found
	Looking for large trees in text/part0008.html...
	No large trees found
	Looking for large trees in text/part0011.html...
	No large trees found
		Splitting on page-break at id=calibre_pb_0
Adjusted split point to ancestor
		Splitting on page-break at id=calibre_pb_1
		Splitting on page-break at id=calibre_pb_2
		Splitting on page-break at id=calibre_pb_3
		Splitting on page-break at id=calibre_pb_4
		Splitting on page-break at id=calibre_pb_5
		Splitting on page-break at id=calibre_pb_6
		Splitting on page-break at id=calibre_pb_7
		Splitting on page-break at id=calibre_pb_8
	Looking for large trees in text/part0024.html...
	No large trees found
	Split into 9 parts
	Looking for large trees in text/part0017.html...
	No large trees found
	Looking for large trees in text/part0023.html...
	No large trees found
		Splitting on page-break at id=calibre_pb_0
	Looking for large trees in text/part0028.html...
	No large trees found
	Split into 2 parts
	Looking for large trees in text/part0026.html...
	No large trees found
	Looking for large trees in text/part0004.html...
	No large trees found
	Looking for large trees in text/part0012.html...
	No large trees found
	Looking for large trees in text/part0020.html...
	No large trees found
Encrypting font: fonts/00001.otf
Encrypting font: fonts/00002.otf
Encrypting font: fonts/00003.otf
Encrypting font: fonts/00004.otf
Encrypting font: fonts/00005.otf
Encrypting font: fonts/00006.otf
Encrypting font: fonts/00007.otf
Encrypting font: fonts/00008.otf
Encrypting font: fonts/00009.otf
Encrypting font: fonts/00010.otf
Encrypting font: fonts/00011.otf
Encrypting font: fonts/00012.otf
Encrypting font: fonts/00017.otf
Encrypting font: fonts/00018.otf
Upgrading to EPUB 3...
EPUB output written to C:\Users\Dave\AppData\Local\Temp\calibre_0q52gfog\i1aiot12.epub

Computer: GIGABYTE B550 VISION D-P
CPU: AMD Ryzen 5 5600X (Vermeer, VMR-B0)
3700 MHz (37.00x100.0) @ 3593 MHz (36.00x99.8)
Motherboard: GIGABYTE B550 VISION D-P
BIOS: F15d, 07/20/2022
Chipset: AMD B550 (Promontory PROM19 C)
Memory: 16384 MBytes @ 1597 MHz, 14-14-14-34
- 8192 MB PC25600 DDR4 SDRAM - G.Skill F4-3200C14-8GFX
- 8192 MB PC25600 DDR4 SDRAM - G.Skill F4-3200C14-8GFX
Graphics: AMD Radeon RX 6800 XT
AMD Radeon RX 6800 XT, 16384 MB GDDR6 SDRAM
Drive: KINGSTON SKC3000D2048G, 2000.4 GB, NVMe
Sound: ATI/AMD Navi2x - High Definition Audio Controller
Sound: AMD Family 17h/19h - HD Audio Controller
Network: RealTek Semiconductor RTL8125 Gaming 2.5GbE Family Ethernet Controller
Network: Intel Wi-Fi 6 AX200 160MHz
OS: Microsoft Windows 10 Professional (x64) Build 19045.2364

[DNSB]

Could you try using the KindleUnpack plugin to convert the azw3 to epub and see if the images survive?

[enuddleyarbl]

Originally Posted by DNSB

Could you try using the KindleUnpack plugin to convert the azw3 to epub and see if the images survive?

Yes. it survives the KindleUnpack KF8 to Epub process.

Surprisingly, I ran the AZW3 through ScrambleEbook and the resulting AZW3 also loses the corresponding image in the Calibre conversion process. And, like the real book, that images comes across just fine in KindleUpack.

https://drive.google.com/file/d/1s9f...ew?usp=sharing

[kovidgoyal]

Open a bug report and attach the file there. Mark it private.

[jhowell]

Your log shows that you have the option "Remove first image" set under the Structure Detection tab. That appears to be the source of the problem.

Unrelated to this, you also have Heuristic Processing enabled which can cause a variety of problems if used indiscriminately.

[enuddleyarbl]

Originally Posted by jhowell

Your log shows that you have the option "Remove first image" set under the Structure Detection tab. That appears to be the source of the problem.

Unrelated to this, you also have Heuristic Processing enabled which can cause a variety of problems if used indiscriminately.

I think you're right. I've always had that "Remove First Image" option on and it's never given me a problem. But, with this book, the first image by position in the book (as well as position in the Images section) is that very same image that they use in the chapter titles. I'd always assumed that the first image would be something used as a cover page. Probably, up until now, that happened to be true for my cases.

I'm going to consider this to be "works as designed" and not bother opening any kind of bug report. I'll have to consider whether I want to turn that option off or keep it on and just keep an eye out for unexpectedly disappearing images.

Thanks.

BTW: what is considered to be the "First Image?" The first image that shows up in the book (usually the cover) or the first image that's listed in the Images section?

[kovidgoyal]

the first image that shows up in the book.