Searching comments in calibre

franknight · 09-09-2018, 11:18 AM

I use calibre a lot and keep a fairly large library. I like to have abstracts in the comments for each book. It is tedious to go through each comment to check its integrity. So I decided comments less than 200 characters i would replace. But when I did a search on short comments the answers were erratic. I used the rational expression ^.{0,200}$ in the calibre search comments box from "edit metadata, edit metadata in bulk, search and replace." I replaced the short comment fields with blanks (because I don't know how to mark them otherwise) and many comment fields are not blanked that only hold 10 or 20 characters. Any help? Thanks if you can help me. frank

kovidgoyal · 09-09-2018, 11:05 PM

You'd probably need to get the . to match all characters including newlines, other wise itll only match text whoose first line is less than 200 chars. Prefix the expression with (?s) for that.

franknight · 09-10-2018, 03:35 PM

Thank you for your quick response. I'm sorry to be such a noob but I just don't understand. It seems to me most all first lines would be less than 200 chars so I would have matched most comment fields. But it didn't - It matched about 3% of them and didn't match some with 20 chars total. Could you feed my feeble mind with a little more info. Many thanks. frank

davidfor · 09-10-2018, 08:47 PM

Quote:

Originally Posted by franknight

Thank you for your quick response. I'm sorry to be such a noob but I just don't understand. It seems to me most all first lines would be less than 200 chars so I would have matched most comment fields. But it didn't - It matched about 3% of them and didn't match some with 20 chars total. Could you feed my feeble mind with a little more info. Many thanks. frank

The problem is that your regex is matching against each line in the comment. That is, it is matching against lines of comment that end in a new-line character. If the comment has a single line with less than 200 characters in it, then it will be matched. And from what I have seen, that will be fairly common.

Kovid's suggestion is to change the regex to :

Code:

(?s)^.{0,200}$

But, I'm not sure if that will do exactly what you want. I can't test it at the moment.

franknight · 09-11-2018, 11:54 AM

You are exactly right. It does have something to do with new line (carriage returns) but it doesn't act as you would think. You would expect it to delete any comment field with a line (chars between carriage returns) less than 200 chars. It does not. I tried a comment field with six lines of 30 chars each and it did not delete them. But put them together into one 180 char line and it does delete it. I don't understand all this but I don't think I can do what I want unless there is a way to add the number of chars in a line together with all the other lines number of chars in the comment field. Is this possible? Or is there another way or is this just too silly for you to continue with? lol. frank PS: BTW I really enjoyed learning a bit about regular expressions - something I didn't even know existed being a computer engineer from the 60's.

franknight · 09-12-2018, 08:02 AM

Thank you for your 'dotall' suggestion of (?s) -- I have prefixed the expression as you suggest but it makes no difference. If I put one carriage return in a short calibre comment neither expression will catch the comment and delete it. In fact it seems the (?s) doesn't modify the ^.{0,200}$ expression at all. i.e (?s)^.{0,200}$ = ^.{0,200}$ in calibre RE at least in the comment fields. Please ignore my last post when I didn't understand the dotall modifier. Am I nuts here or starting to figure it out? thanks for any reply, frank

kovidgoyal · 09-12-2018, 09:02 AM

It's possible the dot all modifier does not work (it might be overriden) In which case you can do something like

Code:

[^|]

instead of using the dot. As long as your comments dont use the | character it should work.

franknight · 09-14-2018, 07:12 PM

Hmmm. That's a bit cryptic. Would that make the expression [^|](?s)^.{0,200}$ My comments don't use the | character to my knowledge.

BetterRed · 09-14-2018, 08:58 PM

Cryptism is regex's raison d'être.

kovidgoyal · 09-14-2018, 09:21 PM

No, just

Code:

[^|]{0,200}$

franknight · 09-15-2018, 05:01 AM

Very clever BeterRed! Thanks Kovid - I'll run some tests.

franknight · 09-19-2018, 02:16 PM

I'm still fooling with my little short comment problem. It seems the first suggestion (?s)^.{0,200}$ works the best. After more reading I can't understand why it isn't perfect but it is very good. When I get a match now I prepend the string SHORT COMMENT to the tags column. Works fine and I can then search for them.

BUT the comments that fail prepends the entire comment into the tags field. I can't imagine why. Help?

franknight · 09-20-2018, 04:41 PM

I think I understand why the RE (?s)^.{0,200}$ fails sometimes. The following comment fails with this formula: <div><div style="text-align: justify;">1234567890</div></div> This only has 10 digits and the test case in calibre works fine for this short comment but the actual comment fails to be recognized as less than 200 chars because of the punctuation. Is there any way to get calibre or the RE engine to ignore the punctuation?

kovidgoyal · 09-21-2018, 12:31 AM

No I'm afraid not, the regex engine has no knowledge of html markup

franknight · 09-21-2018, 02:14 AM

Ahhhh! THANK YOU Kovid. This may have solved the mystery. Do you think the RE engine is getting hung up on these markup descriptors. And if so is there any way to make my calibre comments have less of them? If not I will need to find another way or just 'thumb' though 60 thousand books. Ughhh! Thanks so much for your patience. I'm sure the average mobileread forum participant has no interest in this stuff.

09-09-2018, 11:18 AM	#1
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	Searching comments in calibre I use calibre a lot and keep a fairly large library. I like to have abstracts in the comments for each book. It is tedious to go through each comment to check its integrity. So I decided comments less than 200 characters i would replace. But when I did a search on short comments the answers were erratic. I used the rational expression ^.{0,200}$ in the calibre search comments box from "edit metadata, edit metadata in bulk, search and replace." I replaced the short comment fields with blanks (because I don't know how to mark them otherwise) and many comment fields are not blanked that only hold 10 or 20 characters. Any help? Thanks if you can help me. frank

09-12-2018, 09:02 AM	#7
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's possible the dot all modifier does not work (it might be overriden) In which case you can do something like Code: [^\|] instead of using the dot. As long as your comments dont use the \| character it should work.

09-14-2018, 09:21 PM	#10
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, just Code: [^\|]{0,200}$

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Searching in comments for html-tags	Kuhnke	Calibre	1	05-23-2017 09:33 PM
Searching in comments	Joanna	Library Management	5	07-19-2016 04:47 AM
searching comments	pinky62	Library Management	1	12-01-2015 04:10 AM
Searching comments, or lack thereof	BookJunkieLI	Library Management	2	02-23-2012 11:28 PM
Searching for books with no comments/cover	Bertrand	Calibre	3	12-28-2010 09:14 PM

09-09-2018, 11:05 PM	#2
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You'd probably need to get the . to match all characters including newlines, other wise itll only match text whoose first line is less than 200 chars. Prefix the expression with (?s) for that.

09-10-2018, 03:35 PM	#3
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	Thank you for your quick response. I'm sorry to be such a noob but I just don't understand. It seems to me most all first lines would be less than 200 chars so I would have matched most comment fields. But it didn't - It matched about 3% of them and didn't match some with 20 chars total. Could you feed my feeble mind with a little more info. Many thanks. frank

09-11-2018, 11:54 AM	#5
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	You are exactly right. It does have something to do with new line (carriage returns) but it doesn't act as you would think. You would expect it to delete any comment field with a line (chars between carriage returns) less than 200 chars. It does not. I tried a comment field with six lines of 30 chars each and it did not delete them. But put them together into one 180 char line and it does delete it. I don't understand all this but I don't think I can do what I want unless there is a way to add the number of chars in a line together with all the other lines number of chars in the comment field. Is this possible? Or is there another way or is this just too silly for you to continue with? lol. frank PS: BTW I really enjoyed learning a bit about regular expressions - something I didn't even know existed being a computer engineer from the 60's.

09-12-2018, 08:02 AM	#6
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	Thank you for your 'dotall' suggestion of (?s) -- I have prefixed the expression as you suggest but it makes no difference. If I put one carriage return in a short calibre comment neither expression will catch the comment and delete it. In fact it seems the (?s) doesn't modify the ^.{0,200}$ expression at all. i.e (?s)^.{0,200}$ = ^.{0,200}$ in calibre RE at least in the comment fields. Please ignore my last post when I didn't understand the dotall modifier. Am I nuts here or starting to figure it out? thanks for any reply, frank

09-14-2018, 07:12 PM	#8
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	Hmmm. That's a bit cryptic. Would that make the expression [^\|](?s)^.{0,200}$ My comments don't use the \| character to my knowledge.

09-14-2018, 08:58 PM	#9
BetterRed null operator (he/him) Posts: 20,570 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	Cryptism is regex's raison d'être.

09-15-2018, 05:01 AM	#11
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	Very clever BeterRed! Thanks Kovid - I'll run some tests.

09-19-2018, 02:16 PM	#12
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	I'm still fooling with my little short comment problem. It seems the first suggestion (?s)^.{0,200}$ works the best. After more reading I can't understand why it isn't perfect but it is very good. When I get a match now I prepend the string SHORT COMMENT to the tags column. Works fine and I can then search for them. BUT the comments that fail prepends the entire comment into the tags field. I can't imagine why. Help?

09-20-2018, 04:41 PM	#13
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	I think I understand why the RE (?s)^.{0,200}$ fails sometimes. The following comment fails with this formula: <div><div style="text-align: justify;">1234567890</div></div> This only has 10 digits and the test case in calibre works fine for this short comment but the actual comment fails to be recognized as less than 200 chars because of the punctuation. Is there any way to get calibre or the RE engine to ignore the punctuation?

09-21-2018, 12:31 AM	#14
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No I'm afraid not, the regex engine has no knowledge of html markup

Advert

Advert

09-21-2018, 02:14 AM	#15
franknight Member Posts: 16 Karma: 10 Join Date: Sep 2018 Device: kindle	Ahhhh! THANK YOU Kovid. This may have solved the mystery. Do you think the RE engine is getting hung up on these markup descriptors. And if so is there any way to make my calibre comments have less of them? If not I will need to find another way or just 'thumb' though 60 thousand books. Ughhh! Thanks so much for your patience. I'm sure the average mobileread forum participant has no interest in this stuff.