Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 08-03-2012, 08:48 AM   #1
TMSxMR
Member
TMSxMR began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2012
Device: Impression
Question Search/Replace from Title Field to Pubdate

Hi

I spent for now more than an hour trying to use search/replace function using regular expression to extract publishing dates from Title or Authors fields and push them into Publishing date field, I used the expression \d{4} to extract the publish year, the preview of the results is OK, but after applying it, it corrupts the library most of the times (and strangly sometimes works just as supposed) , when it corrupts, calibre thinks that there is unrecognized sting in the pubdate field, and some time recognize it just right (all years that I extracted are valid), is there a way to convert or force calibre to understand that the result of regular expression is a number not a string? or I should do that in some other way?
TMSxMR is offline   Reply With Quote
Old 08-05-2012, 04:21 AM   #2
_Em
Junior Member
_Em began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Sep 2010
Device: iPod
Quote:
Originally Posted by TMSxMR View Post
Hi

I spent for now more than an hour trying to use search/replace function using regular expression to extract publishing dates from Title or Authors fields and push them into Publishing date field, I used the expression \d{4} to extract the publish year, the preview of the results is OK, but after applying it, it corrupts the library most of the times (and strangly sometimes works just as supposed) , when it corrupts, calibre thinks that there is unrecognized sting in the pubdate field, and some time recognize it just right (all years that I extracted are valid), is there a way to convert or force calibre to understand that the result of regular expression is a number not a string? or I should do that in some other way?
I've spent quite some time trying to figure this out, and I always get a "unicode values" error resulting in too many values of the wrong type reported.

Eg:
Search Field: title
Search Mode: Regex
Search For: (\d{4})-(\d{2}).*
Replace With: \2:::\1 (or \2-\1, which works when IMPORTING books)
Destination Field: pubdate

has these results:
TypeError:QDateTime(): arguments did not match any overloaded call: overload 1: too many arguments overload 2: argument 1 has unexpected type 'unicode' overload 3: argument 1 has unexpected type 'unicode' overload 4: argument 1 has unexpected type 'unicode' overload 5: argument 1 has unexpected type 'unicode'

So, obviously, field pubdate is expecting an ISO date value instead of a unicode value. But importing automatically typecasts 08-2012 into the apropriate iso value -- so what do we have to do to typecast the unicode value in here? Shouldn't Python be able to do that automatically? Shouldn't QDateTime() be doing this? do I have to somehow pre-transform it via regex?

Help please

(oh yes, and this results in my database being corrupted every time; so now I make a copy immediately before attempting this, and replace it after killing the process (the above error continuously loops until I kill the process)).

Last edited by _Em; 08-05-2012 at 04:25 AM.
_Em is offline   Reply With Quote
 
Advertisement
Old 08-05-2012, 11:27 AM   #3
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,250
Karma: 6020307
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Peeking into the DB

pubdate is shown as (Type: TIMESTAMP)
Code:
2005-02-24 07:00:00
or
2005-02-24 07:00:00+00:00
I don't see the space between the date and time.
Yuo might place a literal time value in the replace eg 01:00+00:00
theducks is offline   Reply With Quote
Old 08-06-2012, 03:24 AM   #4
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,902
Karma: 1216548
Join Date: Jan 2010
Location: France
Device: Many android devices
I was reluctant to make the change to allow search/replace to update date fields precisely because of the possibility of this sort of problem. Calibre was not designed to auto-convert between types and apparently the database calibre uses (SQLite) permits writing trash values into date fields. Sigh ...

The next release (0.8.64) will attempt to convert the string to a date in this case and some other cases that I found.
chaley is offline   Reply With Quote
Old 08-06-2012, 04:26 AM   #5
_Em
Junior Member
_Em began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Sep 2010
Device: iPod
Quote:
Originally Posted by chaley View Post
I was reluctant to make the change to allow search/replace to update date fields precisely because of the possibility of this sort of problem. Calibre was not designed to auto-convert between types and apparently the database calibre uses (SQLite) permits writing trash values into date fields. Sigh ...

The next release (0.8.64) will attempt to convert the string to a date in this case and some other cases that I found.
Thanks! It looks like the unicode date string to ISO date value conversion doesn't work either... so that'd be nice.

How come the import conversion works correctly? Could the same code be used for the search/replace?
_Em is offline   Reply With Quote
Old 08-06-2012, 04:40 AM   #6
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,902
Karma: 1216548
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by _Em View Post
How come the import conversion works correctly? Could the same code be used for the search/replace?
No, at least not in a way that is maintainable. The import code is special cased and handles only pubdate. Search/replace must handle all fields of all types, custom or standard. It knows it has a source and destination field but doesn't care what the base types are; the cross-product of types make such knowledge impractical. Instead it uses metadata for the destination field to look up a db "setter" function for that field, then calls that function hoping that it can handle any necessary type conversion from the source type. There was a path where the pubdate setter did not do the conversion.

FWIW: there was also a path where S/R attempted to set the timestamp. This would never work, so timestamp is no longer a permitted destination field.
chaley is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
generate series from title using search and replace richreads Calibre 5 06-14-2012 10:36 AM
Replace field with match or empty string? charlweed Library Management 2 03-29-2012 01:44 AM
Cleaning Title Names using Search and Replace JTAL604622 Calibre 2 05-24-2011 06:23 AM
search and replace - drops blanks in replace ? cybmole Conversion 10 03-13-2011 04:07 AM
Variable Search, Replace & Transfer to a new field flopis Library Management 10 03-09-2011 12:41 PM


All times are GMT -4. The time now is 10:45 AM.


MobileRead.com is a privately owned, operated and funded community.