User Error in Search/Replace?

[slantybard]

When copying paragraphs of text from #blurb to #comments, Calibre is inserting <br> tags into the html of the text. I have attached 3 images of this issue - the first is the regex being used to copy text and the other 2 show the results (normal view and html view). I am interested in *not* having Calibre insert <br> tags into the copies text.

Thanks for any thoughts.

[chaley]

S & R doesn't add any text or html tags.

Your first screen capture shows that #blurb has html markup, which is copied to comments. The end of the comments matches the end of #blurb. Have you "proved" that the <br> tags are not already in #blurb?

[slantybard]

Originally Posted by chaley

S & R doesn't add any text or html tags.

Your first screen capture shows that #blurb has html markup, which is copied to comments. The end of the comments matches the end of #blurb. Have you "proved" that the <br> tags are not already in #blurb?

Yes, I have "proved" that the <br> tags are not in the #blurb - if you look at the 3rd image, it shows the html for both comments and the blurb - the blurb html has no <br> whereas the text copied to the comments now has <br> tags. The <br> tags are being added by Calibre when doing the S&R.

[chaley]

It isn't search & replace that is changing it.

Experiment: appending a custom HTML comment to the standard comment

• Here is an image of the html of the comments field before running S&R.
  
  
• Here is the contents of a custom HTML column that will be copied to comments.
  
  
• Here is the S&R used.
  
  
• Here is the contents of the "Test text" for book 1, same as shown in the above image.
  

  <div>
  <p>This is a line in the comment.</p>
  <p>This is another line in the comment.</p></div>

• Here is the contents of the "Test result" for book 1. The result is the two fields concatenated. No BR tags are added.
  

  <div>
  <p>{This is the only text in this comment}</p></div><div>
  <p>This is a line in the comment.</p>
  <p>This is another line in the comment.</p></div>

• Here is an image of the contents of comments after the S&R. *Something* (not S&R) cleaned up the </div><div>, joining the two blocks together. Other than that the HTML is the same as what S&R wrote into the column. No BR tags are added.
  

To determine what is "cleaning" the HTML, I tried pasting the "after" text into a custom HTML column.

• After the paste it looks like this:
  
  
• Switch to Text mode:
  
  
• Switch back to HTML mode. The internal DIVs are gone. Nothing to do with S&R
  

These experiments show that the Qt RichText (HTML) widget changes the widget's contents. Nothing to do with S&R.

[slantybard]

Ok, thanks so much for showing me that. I have run some tests on an empty test epub book that has "1000" Words in the #Words metadata and the Calibre workflow I have used is introducing <br> tags somehow. Can you look at what I'm doing and give your thoughts please and thanks:

Image 1: Empty epub test file with: 1000 words, no comments, test paragraphs in #blurb (no <br> tags) - I have attached it if someone wants to try this on their system

Image 2: Run S&R regex to copy "Words" count to the comment field:
Search field #wordcount

(\d+)

Replace append mode field comments

\1 Words

See Image 2 and the results of this S&R in Image 3
Notice that the results image does not have any <br> tags anywhere

Image 4: Run S&R regex to append the #blurb paragraphs to the comment field:
Search field #blurb

(.*)

or

^(.*)$

Replace append mode field comments

\1

See Image 4 and the results of this in Images 5 (normal view) and 6 (html view) where <br> tags have somehow been introduced???

If I do NOT run the 1st S&R regex copying the word count into the comments, <br> tags are not introduced by the 2nd regex copying the blurb

If I have an empty comments field OR have manually typed in the word count into the comments field first (ie again, not running the 1st regex) before running the 2nd regex, <br> tags are not introduced.

I guess my final question would be if this is a Calibre issue or QT5/6 issue?

[chaley]

I tried your experiment. Jumping to the end, I found that the problem is caused by the "1000 Words" not being enclosed in HTML.

My apologies for the length of what follows. I wrote down all the steps so I could be sure I knew what I was doing/had done. The steps I ran to discover this:

1. Empty the comments field.
2. Use your S&R to copy int Words to the comments field. After the S&R comments *but before Edit Metadata* contains
   

   1000 Words

   I used a database manager to look directly at the data in the comments column.
3. Look at the book with the Metadata Editor.
   

   <div>
   <p>1000 Words</p></div>

   Some HTML has magically appeared. I checked the database. That HTML isn't actually in the column data, according to the database.
4. Set a custom comment to the following HTML
   

   <div>
   <p>This is a line in the comment.</p>
   <p>This is another line in the comment.</p></div>

   Closed the metadata editor and reopened. The text was still correct.
5. Checked the blurb field in the database. It is the same.
6. Ran the S&R to append the blurb to comments. S&R says it is putting the following in comments.
   

   1000 Words<div>
   <p>This is a line in the comment.</p>
   <p>This is another line in the comment.</p></div>

7. Checking the database, the comments column contains the same text.
   

   1000 Words<div>
   <p>This is a line in the comment.</p>
   <p>This is another line in the comment.</p></div>

8. Edit metatada on the book. The comments field shows
   

   <div>
   <p>1000 Words</p>
   <p><br></p>
   <p>This is a line in the comment.</p>
   <p><br></p>
   <p>This is another line in the comment.</p></div>

   This is very different from what is in the database.
9. Checking the database, it still contains the original text.
   

   1000 Words<div>
   <p>This is a line in the comment.</p>
   <p>This is another line in the comment.</p></div>

   Lets look at what HTML is being used when showing comments in book details. I changed to code to show it.
10. Here is the output being sent to book details. It has all the cruft and more.
    

    <p class="description">1000 Words</p>
    <div><br/><p class="description">This is a line in the comment.</p>
    <br/><p class="description">This is another line in the comment.</p>
    </div>

    Why?
    
    Looking at the code, calibre does this because it thinks the comments aren't already HTML because the content doesn't start with a '<'. Lets try fixing that by changing the original "1000 Words" to be "<div>1000 words</div>" so calibre thinks it is HTML. After the S&R, the comments field contains
    

    <div>1000 Words</div>

11. edit metadata shows it as
    

    <div>
    <p>1000 Words</p></div>

    Not the same but not as different as above. The database hasn't changed, though.
12. Run the second S&R to add the blurb. The comments field now contains
    

    <div>1000 Words</div><div>
    <p>This is a line in the comment.</p>
    <p>This is another line in the comment.</p></div>

13. The comments field HTML in edit metadata now shows:
    

    <div>
    <p>1000 Words</p>
    <p>This is a line in the comment.</p>
    <p>This is another line in the comment.</p></div>

    This isn't what is in the database but it is much closer.
14. The output in book details is
    

    <div>1000 Words</div><div>
    <p>This is a line in the comment.</p>
    <p>This is another line in the comment.</p></div>

    This is actually what is in the database. No extra <br> or what-have-you.
    
    We now know how to "fix" it. Ensure that *everything* that is added to comments is correct HTML. In this case that means surrounding the "words" stuff with tags.
15. We see that using <div> causes calibre to add <p>. For fun, try it using <p></p> around the word count instead of div. Without showing the intermediate results, we get in edit metadata:
    

    <div>
    <p>1000 Words</p>
    <p>This is a line in the comment.</p>
    <p>This is another line in the comment.</p></div>

    This isn't bad!
    
    We see in book details:
    

    <p>1000 Words</p><div>
    <p>This is a line in the comment.</p>
    <p>This is another line in the comment.</p></div>

    which is what we want.

Bottom line: you must ensure that the resulting comments field contains valid HTML. As such, your first S&R should be like this:


Looking at the code I thought about why calibre was adding all the stuff if the contents didn't appear to be HTML. The answer (I think) is that calibre has no idea what is in there. It must protect itself against strange and malformed html to avoid having book details get scribbled on. One could argue it is being overzealous, but Kovid arrived here after many years of experience with strange things.

[slantybard]

Thank you so much for looking into this and taking the time to explain what is happening. I really appreciate your effort!