[GUI Plugin] Import List

[theducks]

Originally Posted by Floating Quietly

It's a fair point. But I'm curious how you think about my situation.

I used a different program that had a date field with the identical semantics of Calibre. The date field represented when the record was created in that program.

I exported that program's information to a CSV and now want to import it to Calibre.

I'd like to exactly re-hydrate the information from the other program in Calibre so that I can order by the Date column and know it represents when I added the content (which is not the same as when I read the content, in many cases that was months before the add date).

How would you suggest I create that experience in Calibre? And ideally how can I create it without having to constantly type dates in an extra field for every new book I add forever. That is just going to be really annoying. Especially when I forget to do it and a book gets "lost". So now I have to go write some kind of logic to detect missing dates and fix them.

Thoughts?

If you are doing this as an initial import, I can see doing the change.
As a 1 time work around.
Import that date into a custom column
Then use the Bulk mode Metadata Edit (MDE) S&R: to effect a copy of just those with custom dates (find: #mydatename:true) into the {date}
Backups are ALWAYS a good idea when doing bulk MDE operations

Do this using an Intake Library , ALSO SET Preferences: Adding Books: Adding actions (a tab) tick the preserve date
Once things have settled. Use the Copy TO Library (on context menu) to move TO the Target library

[capink]

After importing them, select all the books and choose the metadata bulk edit tool > search and replace tab:

Search field: title
Search for: (.+)
Replace with: \1

Search mode: Regular Expressions
✔ Case sensitive
Apply function after replace: Lower Case (or Title Case)
Mode: Replace Field

You can also save the above settings for easily reusing them

[kcar181]

You're the best!

[Floating Quietly]

Originally Posted by capink

Create a composite column with the following template:

{#custom_date:'ifempty($, field('timestamp'))'}

replace #custom_date with the name of your custom column. And choose sort column by date. This column will show the value of the #custom_date and if not found will substitute it with timestamp.

Thank you! I keep forgetting how ridiculously powerful Calibre is. I'm used to thinking of book cataloging software as being really limited. But that just doesn't apply here.

And in fact, this is actually the perfect solution for me. With my old program I only added a book when I was ready to review it. So the timestamp == review date. But I actually put ebooks into Calibre before I review them. So my date was always going to be a bit off. With this solution I can choose to set a more exact date if I want to (and will set it automatically for older books from import) or leave it empty and let it be calculated from the timestamp if I do not. This appeals to the fastidious nitpicker in me.

But I am also grateful for the pointer to the metadata bulk update. Because if I were slightly less... well me.... this would have been a very sensible solution.

[capink]

Version 1.4.10
Fix: Highlighting and moving between matches in webpage tab not working with python3.
Fix: Disable automatic preview when a new file is selected, to allow users to select proper field mappings first.
Fix: Stop the tag view from updating while the progress bar is on.
Fix: Move previous and next buttons in the web tab out of the scrollarea to make them visible again.

This version is mostly fixes or minor improvements. I do not have any new features I want to add to this plugin, and I hope that eventually this version will be good enough to be released officially. So I like everyone to test this as much as they can and report any problems. If all goes well, I will contact the moderators to see what we can do about releasing it.

I tried my best when adding features not to change the original behavior of the plugin. I made an exception here by stopping the automatic preview whenever a new csv file is selected. The rationale here is that the plugin used to go into preview while the columns were mapped according to the old csv file, often putting the data in the wrong fields (It can be caught by datatype validation only if there are errors in datatypes). Now it gives the user the chance to first remap the columns either manually or using the "from headers" button, and then press preview. This is also consistent with the traditional way the plugin operates when you first open it, as it remembers the last csv file you used, but it does not preview it until the user presses the preview button. I would like to hear your feedback on whether to keep this new behavior or revert to the old one.

[davidfor]

@capink: The search in the webpage tab is working now in Python3. But, the page source is displayed like:

b'<html class="dark prog ads" lang="en">\n <head>\n  \n  <base href="https://www.fantasticfiction.com/">\n  <title>\n   Joe Abercrombie\n  </title>\n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n </head>\n <body>\n  <div class="main" id="mainframe">\n   \n

From a quick test, line 529 in tab_webpage.py needs to be changed to:

self.source_tedit.setPlainText(unicode(raw, 'utf-8'))

That seems to work in both Python 3 and the released version.

[capink]

@davidfor: Thanks for the heads-up.

A new version with davidfor's fix is attached here. The version in the old post removed.

[Rellwood]

I am not sure if this is because the imported dates are always set to dd MMM yyyy (the same as the columns) but whenever I import a list containing dates (ie. read dates) the dates in the column after the list has been imported are one day later.

Imported list says that a book was read Jun 08 2015, the column ends up having June 09 2015. This is for every date in every column. Apparently this has been the way it always has been for this plugin, it is only today that I made the connection that it was caused by it.

For years I have been importing lists with dates and so many times I have noticed that the dates in the list were one off from the column that I was importing them into. Only today I was checking against a list that I had just imported that It came to my attention that EVERY date in the freshly imported column was off by the list that I just used.

Any way of getting around this?

Also....

If I import a page column into a column formatted as intengers {0:,d}, I have to format the column in the list as being numbers and make sure that there is at least a 0 in every line or else it won't import.

Am I going about it wrong? I would like to import page columns that may have books without a listing, but as it stands I have to go into each blank space and mark it with a 0.


Would a Moderator please move this to the Import List Plugin page?

Done

[capink]

Originally Posted by Rellwood

Am I going about it wrong? I would like to import page columns that may have books without a listing, but as it stands I have to go into each blank space and mark it with a 0.

Done

This was fixed in the first release I've done nearly two months ago, have you tested this recently, or are you speaking from memory?

I will have a look into the date problem.

[Rellwood]

Originally Posted by capink

This was fixed in the first release I've done nearly two months ago, have you tested this recently, or are you speaking from memory?

I will have a look into the date problem.

Sorry, I didn't realize you were talking about the page numbers, I am going by memory. Version 4.1.9
Calibre 4.18 64

sorry for all the edits, I type then re-read and realize mistakes.

[capink]

Regarding the date problem reported by Rellwood, from my own testing, it happens only in these situations:

• If the day in the date string is the first day of month (=1) it is incremented one day.
• If the day in the date string is the last day of the month (28, 29, 30, 31 depending on month and year), it is decremented one day.

(The presence of absence of time or timezone in the date string does not change this behavior)

The plugin call a function in Calibre (field_from_string > parse_only_date > fix_only_date), which is responsible for the behavior described above. I don't know why Calibre does this, but I am sure there is good reason for it.

This can be bypassed by calling parse_date directly, but since I don't understand the problem of dates very well, I cannot be sure whether this will lead to any unforeseen problems. So, I will leave this problem to someone who understands it better to deal with it.

Edit: Rellwood reported dates where the day is neither the first nor the last day of the month, this does not happen for me.

[davidfor]

Originally Posted by capink

Regarding the date problem reported by Rellwood, from my own testing, it happens only in these situations:

• If the day in the date string is the first day of month (=1) it is incremented one day.
• If the day in the date string is the last day of the month (28, 29, 30, 31 depending on month and year), it is decremented one day.

(The presence of absence of time or timezone in the date string does not change this behavior)

The plugin call a function in Calibre (field_from_string > parse_only_date > fix_only_date), which is responsible for the behavior described above. I don't know why Calibre does this, but I am sure there is good reason for it.

This can be bypassed by calling parse_date directly, but since I don't understand the problem of dates very well, I cannot be sure whether this will lead to any unforeseen problems. So, I will leave this problem to someone who understands it better to deal with it.

Edit: Rellwood reported dates where the day is neither the first nor the last day of the month, this does not happen for me.

This was asked a long time ago. At the time, I suggested time zone issues. I don't remember if I looked at it then, or some other time for another plugin. If I remember rightly, it is calibre trying to compensate for a timestamp that in one place has a timezone and in another place doesn't, or doesn't need the timezone.

[capink]

Originally Posted by davidfor

This was asked a long time ago. At the time, I suggested time zone issues. I don't remember if I looked at it then, or some other time for another plugin. If I remember rightly, it is calibre trying to compensate for a timestamp that in one place has a timezone and in another place doesn't, or doesn't need the timezone.

This is interesting, because the link talks about this problem occurring in pubdate. In My testing, populating pupdate did not cause this problem. When I looked at the code to see why, I found that field_from_string was not used when populating pubdate, instead it calls parse_date (through a function called parse_pubdate).

So maybe someone fixed this for pubdate but did not do the same for custom date columns.

[thiago.eec]

Originally Posted by capink

This is interesting, because the link talks about this problem occurring in pubdate. In My testing, populating pupdate did not cause this problem. When I looked at the code to see why, I found that field_from_string was not used when populating pubdate, instead it calls parse_date (through a function called parse_pubdate).

So maybe someone fixed this for pubdate but did not do the same for custom date columns.

When I was working on Skoob Books plugin (Jun. 2019), I've noticed this.

Skoob website only give us the year of publication, so I chose to set the pubdate to YYYY-01-01, but this made dates to be set as (YYYY-1)-12-31. As a workaround I chose to set it as YYYY-01-02. Since the real date was only the Year, this would make no difference.

Now, I've just checked and setting pubdate as YYYY-01-01 works normally. It must have been fixed in this interval.

[capink]

Originally Posted by davidfor

This was asked a long time ago. At the time, I suggested time zone issues. I don't remember if I looked at it then, or some other time for another plugin. If I remember rightly, it is calibre trying to compensate for a timestamp that in one place has a timezone and in another place doesn't, or doesn't need the timezone.

Originally Posted by thiago.eec

When I was working on Skoob Books plugin (Jun. 2019), I've noticed this.

Skoob website only give us the year of publication, so I chose to set the pubdate to YYYY-01-01, but this made dates to be set as (YYYY-1)-12-31. As a workaround I chose to set it as YYYY-01-02. Since the real date was only the Year, this would make no difference.

Now, I've just checked and setting pubdate as YYYY-01-01 works normally. It must have been fixed in this interval.

I asked Kovid about this issue and he said that it is done so that the month is one in all time zones, and that it is safe to bypass it using parse_date.

Thanks for your input.