Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 03-21-2016, 11:10 PM   #1
Sidetrack
Enthusiast
Sidetrack began at the beginning.
 
Posts: 39
Karma: 10
Join Date: Jan 2009
Location: South Pacific
Device: Kindle DX
regex/search help needed to remove redundant metadata in different fields

So I've got some redundant data across fields, such as ISBNs listed in identifiers and in tags. or Authors listed in Tags, or Series Names listed in Tags

Once I identify them I can to a regex search and replace something like this

template: {identifiers}, @@ {tags}
search for: .*\bisbn[0-9x-]+),.*@@(.*)\1(.*)
replace with: \2\3
field: tags

so if an ISBN in a tag matches the ISBN in identifiers it will delete the tag. However, if there is no matching tag, that regex dumps the whole "{identifiers}, @@ {tags}" into the tags field.

If I could a) search for matching items across fields, or b) refine the regex so it doesn't corrupt the tags field when the pattern doesn't match... I'd be happy.

Is there an elegant solution that I'm missing? I don't want a lot of built template columns slowing down operations.
Sidetrack is offline   Reply With Quote
Old 03-21-2016, 11:49 PM   #2
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 17,633
Karma: 20473671
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@Sidetrack - see Mufti Column Search

I'm not certain not sure it will do what you want, but worth a look

BR
BetterRed is offline   Reply With Quote
Advert
Old 03-26-2016, 08:06 PM   #3
Sidetrack
Enthusiast
Sidetrack began at the beginning.
 
Posts: 39
Karma: 10
Join Date: Jan 2009
Location: South Pacific
Device: Kindle DX
Quote:
Originally Posted by BetterRed View Post
@Sidetrack - see Mufti Column Search

I'm not certain not sure it will do what you want, but worth a look

BR
That does help, doesn't seem to handle the idenitfiers field that I can see.
Sidetrack is offline   Reply With Quote
Old 03-26-2016, 08:37 PM   #4
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 17,633
Karma: 20473671
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by Sidetrack View Post
That does help, doesn't seem to handle the idenitfiers field that I can see.
Why don't you post something in the PI's thread outlining what you need, the developer is usually up for making kaizen style enhancements.

BR
BetterRed is offline   Reply With Quote
Old 03-27-2016, 11:07 AM   #5
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 1,880
Karma: 1804640
Join Date: Aug 2013
Location: Texas
Device: 10" Win10 Tablet w/Calibre64, CalibreSpy & Freda+
MCS and Identifiers

Quote:
Originally Posted by Sidetrack View Post
That does help, doesn't seem to handle the idenitfiers field that I can see.
@Sidetrack:

Both the 'Final Filters' Tab and 'SQL Queries' Tab handle identifiers. For your purposes, the 'SQL Queries' Tab should be used.

See the image below of an example using the 'SQL Queries' Tab.

Copy and paste this into that Tab:

Quote:
SELECT book FROM _mcs_tags_by_book WHERE tagname LIKE '%978%' OR tagname LIKE '%045%' OR tagname LIKE '%isbn%' OR tagname REGEXP '^[0-9][-]*[0-9][-]*[0-9][-]*[0-9][-]*[0-9][-]*[0-9][-]*[0-9][-]*[0-9][-]*[0-9][-]*[0-9]+$'
This will catch both 10-digit and 13-digit ISBNs with and without hyphens.

Table _mcs_tags_by_book is an MCS "Search Accelerator" table.

In lieu of using the latter table, you could JOIN standard Calibre tables tags and books_tags_link. However, if you have a lot of books with a lot of tags, that query will take much longer to execute. Hence, the MCS "Search Accelerator" table.

By the way, the (industrial-strength) QuarantineAndScrub add-on automatically fixes the issues you described in your first post. The example in the image is from an actual Q&S test library with ISBN test cases.


DaltonST


Attached Thumbnails
Click image for larger version

Name:	mcs_example_isbn_search.jpg
Views:	94
Size:	263.8 KB
ID:	147429  
DaltonST is offline   Reply With Quote
Advert
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Regex search and replace in bulk metadata help slantybard Library Management 7 08-20-2015 07:30 PM
Pruning redundant and partially redundant tags Sidetrack Library Management 0 03-01-2013 06:05 PM
invert search for custom fields (GUI) rogue_librarian Library Management 4 07-25-2011 12:36 PM
Search and Replace or remove Metadata jazzcat007 Calibre 5 05-21-2010 12:35 PM
Searching for.... in fields help needed mezme Calibre 8 02-07-2010 11:32 AM


All times are GMT -4. The time now is 02:43 PM.


MobileRead.com is a privately owned, operated and funded community.