Importing Organization

b00kiest · 01-25-2021, 12:57 AM

So, I finally decided to get a handle on my pattern library but wrangling it into a neat form is seeming impossible. I have all the fancy plugins but i'm not technie enough to know how to use them without point by point instructions and i can't seem to find any thing like that. This is partially why I've let that library go wild but 85Gb is silly and I am wasting time finding patterns I already have :/ much like my books used to be.
Anyhow, I spent the last hour trying to figure out how to import a folder of files that are messy...some all uppercase, some mixedcase, some all lowercase with some having funky pipes to separate title form creator other with _ and still others with . or - to parse the title. Yes, it's a nightmare.

But I was thinking somewhere there was a plugin with magic buttons or instructions how set up expressions to force conformity based on rules I could choose.

Am I SOL or is there something out there that someone could point me to? I have JobSpy and Quality Check and others but again even after reading and searching online I can't find a video showing me how to set the up or run them or directions for stupid. Oh I found information but i'll be honest if it's more than a few sentences I get lost unless I have pictures or a video to follow.
Thanks in advance if there is help someone can point me to. I'm giving up for the night.
Bookiest

ownedbycats · 01-25-2021, 01:38 AM

If it's taking the metadata from the filenames, check out Preferences > Adding Books for the filename metadata regex.

Unfortunately, if the data's pulled from the file metadata itself, it depends on whatever the creator of the file shoved in there (I got that problem when importing game manuals, random names and useless titles). But you can uncheck "read metadata from file contents" if you'd like.

theducks · 01-25-2021, 10:49 AM

The QC PI looks for the (tested) condition and simply Marks (a push pin) those books. YOU need to review and take action. This is a tool to HELP identify metadata issues.

This is what some refer to as 'Normalization', making a standard way of recording data to make searches work well.

FIX section does a few REALLY COMMON adjustments (it uses the same configuration setting as the test) eg make all Author Initials the same. Flip Ln, Fn to Fn Ln (or the other way)
It does NOT handle variations in Names (some with MI, others without), The same Publisher BUT listed differently eg Baen, Baen Books, Baen Publishing Enterprises Note: I prefer to use (cover) IMPRINT rather than the House when ever possible.
Word of caution: Auto-incorrect

Beware of auto fixing everything. LOOK AT the selections.
The SAME Series name may be used by multiple UNRELATED Authors. You may need to modify some to have a tie breaker eg Magic (AN) . Also beware of variants of the same over time

When I first started, I added (empty books) all (940) of my paper Library info by using the ISBN on the book. It took over 2 years of fix a few per day, to beat the metadata into shape.
OH! as you do, be sure to BACKUP before doing any MASS change, just in case AUTO does goes south.

b00kiest · 01-26-2021, 09:42 PM

So, it's a pipe dream? I'm just going to apologize now for the following stupid questions.

I read the replies and I still am unclear if what I want to happen can or cannot. So, I'm going to try one more time.

So if I look at the reg. expression that calls the order to put the filename: Author, series, title.

(?P<author>[^_-]+) -?\s*(?P<series>[^_0-9-]*)(?P<series_index>[0-9]*)\s*-\s*(?P<title>[^_].+) ?

I can't add in something like:

Find: ([a-z])([A-Z])
Replace: $1\l$2

Remove camel case (e.g. cAmelCAse => camelcAse => camelcase)

*note that I only superficially understand this may work but I am not sure how as I tried adding the find and replace strings and only can create spectacular

or did I understand that the regex is only used in the areas open to Calibre for search patterns and for one that change would need to be built in a plugin and currently there isn't one that does this?

thanks in advance.
Bookiest

P.S. I am not certain how many patterns I have but its definitely too many :/ since at the rate I create or use one i'd have to live another 17000 years. I'm thinking that might be out as an option.

davidfor · 01-27-2021, 04:18 AM

You will probably be able to do what you want, but, it might take a couple of passes, and the function to do it isn't all in the same place.

The add template is about how to split the file name up to populate the title, author and so on. It doesn't have any options to change the the select parts. There is function to do this for the author on the "Adding actions" tab of the "Adding books" preference page. There you can set rules for what to do with tags or the author. That includes changing the case.

For the title, you can do this after the import with the bulk metadata editor. Select the new books, open the metadata editor and the title change options are near the bottom of the first tab. You can also do this with the search-and-replace.

Also, when adding books, the metadata can come from the filename or the internal metadata. If the metadata has already been set, the latter is the better option.

Some of the problems can be found in the library using the Quality Check plugin as @theducks suggests. The plugin can fix some of the problems.

Some of this is manual. There is the Action Chains plugin to do things, but, I haven't needed to look at it, so I don't know if it can help with what you want.

b00kiest · 01-27-2021, 11:00 PM

Ok so I was sorta right. Thanks for explaining it more. I think I have a basic understanding. I'll take a look at the autoadd. I had before but I didn't quite see how that work as I recall it was for things imported to a folder that Calibre polls for entries to add. Which I didn't quite grasp when the polling takes place so I thought well if it there it has to be other places.. that is the functionality. Silly me.

I looked at the plugin you suggested but unfortunately the TTS to MP3 plugin I can't do without hasn't ported to python 3 as far as I know and if I upgrade I'll lose that option. So, I'll keep puttering around and see if the AutoAdd has a video for the uninitiated to help out.

Or honestly, with my patience and attention span these days, i might be barking up the wrong tree. Cleaving out some of the content with a Russian roulette method might do me some good. Then I could hand curate the lucky ones that get to stay. it's not like the things I want to organize I can't get again if I really need them. It's just 15 years of unplanned foldering and forced imports over computer changes and poor decisions on structure many times over with no appetite to do the work to clean it out though in my flimsy excuse I think I thought on some level I'd get to it just not 14 years too late

This is turning out to be a lot more effort than I may be able to give.

Thanks everyone. If you hear swearing and other odd noises just ignore it. Any deletions are done with health and sanity of the local computer in mind.

davidfor · 01-27-2021, 11:19 PM

Quote:

Originally Posted by b00kiest

I looked at the plugin you suggested but unfortunately the TTS to MP3 plugin I can't do without hasn't ported to python 3 as far as I know and if I upgrade I'll lose that option. So, I'll keep puttering around and see if the AutoAdd has a video for the uninitiated to help out.

From the version history, TTS to MP3 was made compatible with Python 3 in May last year. And the minimum calibre version for the current version of the plugin is 5.4.0.

b00kiest · 01-28-2021, 08:17 PM

Well color me stupid. I thought when I reviewed the list a week or so ago it was not one crossed out. I do check now as I don't want another issue like I had last year with kindle not working etc etc. Thanks for point it out. I'm updating now.

01-25-2021, 12:57 AM	#1
b00kiest Member Posts: 16 Karma: 36 Join Date: Mar 2019 Device: iphone, yoga 3 thinkpad	Importing Organization So, I finally decided to get a handle on my pattern library but wrangling it into a neat form is seeming impossible. I have all the fancy plugins but i'm not technie enough to know how to use them without point by point instructions and i can't seem to find any thing like that. This is partially why I've let that library go wild but 85Gb is silly and I am wasting time finding patterns I already have :/ much like my books used to be. Anyhow, I spent the last hour trying to figure out how to import a folder of files that are messy...some all uppercase, some mixedcase, some all lowercase with some having funky pipes to separate title form creator other with _ and still others with . or - to parse the title. Yes, it's a nightmare. But I was thinking somewhere there was a plugin with magic buttons or instructions how set up expressions to force conformity based on rules I could choose. Am I SOL or is there something out there that someone could point me to? I have JobSpy and Quality Check and others but again even after reading and searching online I can't find a video showing me how to set the up or run them or directions for stupid. Oh I found information but i'll be honest if it's more than a few sentences I get lost unless I have pictures or a video to follow. Thanks in advance if there is help someone can point me to. I'm giving up for the night. Bookiest

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Hacks Organization from a computer: Possible?	mb2u	Amazon Kindle	5	12-21-2017 05:15 AM
Organization problem	teraflame	Amazon Kindle	2	03-09-2011 12:26 AM
K3 Legacy Organization	Gfinch	Amazon Kindle	7	10-22-2010 05:18 PM
Any suggestions for organization?	kl_0	Calibre	4	02-08-2010 01:44 PM
Library organization	crAss	Bookeen	5	01-27-2009 03:39 PM

01-25-2021, 01:38 AM	#2
ownedbycats Custom User Title Posts: 10,961 Karma: 74999999 Join Date: Oct 2018 Location: Canada Device: Kobo Libra H2O, formerly Aura HD	If it's taking the metadata from the filenames, check out Preferences > Adding Books for the filename metadata regex. Unfortunately, if the data's pulled from the file metadata itself, it depends on whatever the creator of the file shoved in there (I got that problem when importing game manuals, random names and useless titles). But you can uncheck "read metadata from file contents" if you'd like.

01-25-2021, 10:49 AM	#3
theducks Well trained by Cats Posts: 31,047 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	The QC PI looks for the (tested) condition and simply Marks (a push pin) those books. YOU need to review and take action. This is a tool to HELP identify metadata issues. This is what some refer to as 'Normalization', making a standard way of recording data to make searches work well. FIX section does a few REALLY COMMON adjustments (it uses the same configuration setting as the test) eg make all Author Initials the same. Flip Ln, Fn to Fn Ln (or the other way) It does NOT handle variations in Names (some with MI, others without), The same Publisher BUT listed differently eg Baen, Baen Books, Baen Publishing Enterprises Note: I prefer to use (cover) IMPRINT rather than the House when ever possible. Word of caution: Auto-incorrect Beware of auto fixing everything. LOOK AT the selections. The SAME Series name may be used by multiple UNRELATED Authors. You may need to modify some to have a tie breaker eg Magic (AN) . Also beware of variants of the same over time When I first started, I added (empty books) all (940) of my paper Library info by using the ISBN on the book. It took over 2 years of fix a few per day, to beat the metadata into shape. OH! as you do, be sure to BACKUP before doing any MASS change, just in case AUTO does goes south.

01-26-2021, 09:42 PM	#4
b00kiest Member Posts: 16 Karma: 36 Join Date: Mar 2019 Device: iphone, yoga 3 thinkpad	So, it's a pipe dream? I'm just going to apologize now for the following stupid questions. I read the replies and I still am unclear if what I want to happen can or cannot. So, I'm going to try one more time. So if I look at the reg. expression that calls the order to put the filename: Author, series, title. (?P<author>[^_-]+) -?\s(?P<series>[^_0-9-])(?P<series_index>[0-9])\s-\s(?P<title>[^_].+) ? I can't add in something like: Find: ([a-z])([A-Z]) Replace: $1\l$2 Remove camel case (e.g. cAmelCAse => camelcAse => camelcase) note that I only superficially understand this may work but I am not sure how as I tried adding the find and replace strings and only can create spectacular or did I understand that the regex is only used in the areas open to Calibre for search patterns and for one that change would need to be built in a plugin and currently there isn't one that does this? thanks in advance. Bookiest P.S. I am not certain how many patterns I have but its definitely too many :/ since at the rate I create or use one i'd have to live another 17000 years. I'm thinking that might be out as an option.

01-27-2021, 04:18 AM	#5
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	You will probably be able to do what you want, but, it might take a couple of passes, and the function to do it isn't all in the same place. The add template is about how to split the file name up to populate the title, author and so on. It doesn't have any options to change the the select parts. There is function to do this for the author on the "Adding actions" tab of the "Adding books" preference page. There you can set rules for what to do with tags or the author. That includes changing the case. For the title, you can do this after the import with the bulk metadata editor. Select the new books, open the metadata editor and the title change options are near the bottom of the first tab. You can also do this with the search-and-replace. Also, when adding books, the metadata can come from the filename or the internal metadata. If the metadata has already been set, the latter is the better option. Some of the problems can be found in the library using the Quality Check plugin as @theducks suggests. The plugin can fix some of the problems. Some of this is manual. There is the Action Chains plugin to do things, but, I haven't needed to look at it, so I don't know if it can help with what you want.

01-27-2021, 11:00 PM	#6
b00kiest Member Posts: 16 Karma: 36 Join Date: Mar 2019 Device: iphone, yoga 3 thinkpad	Ok so I was sorta right. Thanks for explaining it more. I think I have a basic understanding. I'll take a look at the autoadd. I had before but I didn't quite see how that work as I recall it was for things imported to a folder that Calibre polls for entries to add. Which I didn't quite grasp when the polling takes place so I thought well if it there it has to be other places.. that is the functionality. Silly me. I looked at the plugin you suggested but unfortunately the TTS to MP3 plugin I can't do without hasn't ported to python 3 as far as I know and if I upgrade I'll lose that option. So, I'll keep puttering around and see if the AutoAdd has a video for the uninitiated to help out. Or honestly, with my patience and attention span these days, i might be barking up the wrong tree. Cleaving out some of the content with a Russian roulette method might do me some good. Then I could hand curate the lucky ones that get to stay. it's not like the things I want to organize I can't get again if I really need them. It's just 15 years of unplanned foldering and forced imports over computer changes and poor decisions on structure many times over with no appetite to do the work to clean it out though in my flimsy excuse I think I thought on some level I'd get to it just not 14 years too late This is turning out to be a lot more effort than I may be able to give. Thanks everyone. If you hear swearing and other odd noises just ignore it. Any deletions are done with health and sanity of the local computer in mind.

01-28-2021, 08:17 PM	#8
b00kiest Member Posts: 16 Karma: 36 Join Date: Mar 2019 Device: iphone, yoga 3 thinkpad	Well color me stupid. I thought when I reviewed the list a week or so ago it was not one crossed out. I do check now as I don't want another issue like I had last year with kindle not working etc etc. Thanks for point it out. I'm updating now.

Advert

Advert