Importing and converting html fles

[citac]

I checked the forum to see if there was a similar question to mine, but couldn't find one. I understand why calibre converts a file saved as web page, complete to a zip file since it contains photos and css and whatnot, but why does it to the same thing to a pure html file without any css or images attached? Just a simple html file containing text? The files in question are fanfiction files saved through a fanfiction downloader.

I just tried importing a few files just to check whether I was mistaking simple html files for the ones with images attached, but I'm not.

I also have a problem with the author of the story being confused with the author of the dowloader program when imported into calibre. Looking at the markup of the file, the downloader inserts <meta name='author' content='name_of_programmer' /> into the file which is read as author name by calibre. I could correct this for a few files, but I have thousands of files I plan to import and convert. Is there a way to automate this?

Thank you in advance.

[Starson17]

Originally Posted by citac

I understand why calibre converts a file saved as web page, complete to a zip file since it contains photos and css and whatnot, but why does it to the same thing to a pure html file without any css or images attached?

Consistency?

I also have a problem with the author of the story being confused with the author of the dowloader program when imported into calibre. Looking at the markup of the file, the downloader inserts <meta name='author' content='name_of_programmer' /> into the file which is read as author name by calibre. I could correct this for a few files, but I have thousands of files I plan to import and convert. Is there a way to automate this?

What "downloader program" are you referring to? Are you saying it modifies the html? If it does, you can strip it during conversion.

[citac]

What "downloader program" are you referring to? Are you saying it modifies the html? If it does, you can strip it during conversion.

Alright, consistency. Can I make it stop doing that? I was going to leave html files as is and only convert to epub web pages that had covers and banners; will I have to convert them too?

What "downloader program" are you referring to? Are you saying it modifies the html? If it does, you can strip it during conversion.

I was using Graffer, but it seems to have been abandoned. There is no longer a website connected to it. I'm going to switch to Fanfiction Downloader which has improved recently and added several fic sources, and it handles meta name="author" correctly (and yes, they both modify the html during download). How do I strip or change the incorrect author information in calibre?

[Starson17]

Originally Posted by citac

Alright, consistency. Can I make it stop doing that?

Disable the HTML to Zip filetype plugin.

I was using Graffer, but it seems to have been abandoned. There is no longer a website connected to it. I'm going to switch to Fanfiction Downloader which has improved recently and added several fic sources, and it handles meta name="author" correctly (and yes, they both modify the html during download). How do I strip or change the incorrect author information in calibre?

You can do it in the conversion settings to change the html, but you'll need to change the metadata stored for the file anyway to the correct author.

[citac]

Disable the HTML to Zip filetype plugin.

And that will stop it from (um, what's the term?) enveloping web pages with images into zip, correct? Would that affect the conversion from html to epub? Will I end up with an epub file with missing images? I should probably say that I don't intend to keep the files in the Calibre library after conversion.

You can do it in the conversion settings to change the html, but you'll need to change the metadata stored for the file anyway to the correct author.

Where will I find these particular conversion settings? I'm looking at Preferences > Input options, and the only input formats offered are fb2, pdf and txt. Output options has htmlz, but only offers "How to handle css" and "How to handle class based css". Unless I should go to Common options > Search and replace? I am not familiar with regexes and made a mess of things last time I tried using them. Can I just paste the text I want to change with the text I want to change it into?

I know I'll have to change the metadata to the correct author. I presume there's a way to do it for several files at once for those fics that share an author; it will be more work for those of individual authors. I definitely have some reading up to do.

ETA OK, so I just read a few threads on regexes; am I right to assume they're not used to replace text within a file? There was a thread where Kovid recommended using Sigil for this. Or am I misunderstanding regexes?

[Starson17]

Originally Posted by citac

And that will stop it from (um, what's the term?) enveloping web pages with images into zip, correct?

It will stop the zip processing of all html files. Note that it's not just images that are affected. Any html page that uses a relative link to another file will be affected whether that other file is html, jpg, png or whatever. The processing does other work. Look at the options for that plugin to see what you can control and look at the zip content after processing with that plugin to see some of the additional files it creates.

Would that affect the conversion from html to epub? Will I end up with an epub file with missing images? I should probably say that I don't intend to keep the files in the Calibre library after conversion.

I don't recommend disabling it if you are processing anything other than single html files, and even there, I'm not sure why you'd bother to disable it.

Where will I find these particular conversion settings?

Search and replace.

I am not familiar with regexes and made a mess of things last time I tried using them. Can I just paste the text I want to change with the text I want to change it into?

Yes, you can just search for text1 and replace with text2, but knowing some regex would help.

ETA OK, so I just read a few threads on regexes; am I right to assume they're not used to replace text within a file?

In Calibre, most regex is used to change metadata, not book content, however, you can use regex S&R to change content during conversion from one format to another (or from one format to the same format). If what you are going to do is complex editing, then Sigil is the right tool.

[citac]

It really gets on my nerves that it imports even plain html as zip files, even though it makes sense to me that it should do so with web pages. I haven't really been processing anything yet: I've imported several fics, each one a different format depending on whether it was saved from an ancient archive active in the early years of fandom (.txt), whether it was done for a Big Bang or fest exchange on LJ or DW (accompanying artwork = web page), or saved with a downloader from current fic archives (plain html that contains bold, italics and no images). I don't keep several different formats of the same file as I gather many users do. I want my plain html to stay plain html, I would like to convert my txt files to epub, and probably web pages too. I don't understand the need for it.

So, I've been trying out the program again, trying to understand how it works because I wanted to see if I can bulk edit some things in my fics, like this problem with incorrect authors, and I've been reading the tutorials on regexes ad search&replace, and I still don't understand how thy work. I tried following them, but what I'm doing doesn't seem to have any results.

If I choose Edit metadata individually, there's no search&replace; if I choose bulk editing, there's a search&replace tab, and I have several options which again don't work for me. I enter the <meta name='author' content='Grzegorz Hordynski' /> string, and replace it with the proper author, it shows up in the test text, but hitting Apply does absolutely nothing. How do I achieve what I'm after? I have authors who have written over 40 works and I cannot edit them all. What am I doing wrong?

[Starson17]

Originally Posted by citac

It really gets on my nerves that it imports even plain html as zip files, even though it makes sense to me that it should do so with web pages.

You might prefer that calibre use .zip for some html files and .html for others, but I certainly would not. I want the same extension for all files of the same format. I don't think the file extension should be dependent on the ebook content.

Also, be aware that the zip processing of even a single file html ebook does more than just compress the single file. Look inside the zip file produced by the plugin to see the other files created and stored in the zip file.

I want my plain html to stay plain html

Then disable the processing plugin.

I would like to convert my txt files to epub, and probably web pages too. I don't understand the need for it.

I'm not really sure what the problem is other than that you don't like the way Calibre has to handle multiple file html ebooks? Is it interfering with how you use the html files?

If I choose Edit metadata individually, there's no search&replace;

Correct. "Metadata" isn't content, and calibre isn't an editor, so you won't find any way to edit content on that page. You also won't find any way to search and replace metadata since you've only got one book. Just change whatever needs to be changed.

if I choose bulk editing, there's a search&replace tab,

Yes. It's for searching and finding/changing metadata, not content.

I have several options which again don't work for me. I enter the <meta name='author' content='Grzegorz Hordynski' /> string, and replace it with the proper author, it shows up in the test text, but hitting Apply does absolutely nothing.

It sounds like you are trying to change content. You can't do that there. It's on the Conversion screen. It will do it, but it's not really intended for what you are doing.

How do I achieve what I'm after?

I'm not certain what you are trying to do? Change content? Before importing to Calibre? (You can't) After importing (Convert from zip/html to your destination format and use S&R/regex. (I'm not sure why you want/need to change this content if you already have imported the ebook.) Change metadata? That's what Calibre is best at. Select the books and bulk edit, then S&R.

I have authors who have written over 40 works and I cannot edit them all. What am I doing wrong?

Explain the goal, then we can help. If I'm guessing, you should import them all, highlight them all and use bulk edit to fix the authors. It should only take a few minutes. If you need to change content, you'll have to do them individually (personally, I'd use Ultraedit to do it globally 0 Calibre isn't the best tool, even if it will do it.)

[citac]

I keep forgetting calibre makes a copy of everything it imports and puts it into its own folder; I kept looking at the original location and I could not see what you were talking about because of course it wasn't there.

I just saw what you were talking about: there's a META-INF folder, a mimetype file and content.opf, and there's also a metadata.opf in the ebook folder together with the zip. Everyone kept talking about how you should never look into calibre created folders. Since these folders are what drove me bonkers whenever I used calibre before, I was deliberately staying away from them.

I'm not really sure what the problem is other than that you don't like the way Calibre has to handle multiple file html ebooks? Is it interfering with how you use the html files?

It's not that I don't like the way it handles multiple file html ebooks, it's that I didn't understand why there would even be a need to zip up a plain html file when there's no additional content connected to it. I get that you'd need to zip up a html file and its style sheet, but a html file with an embedded stylesheet? It seemed redundant.

It sounds like you are trying to change content.

Yes, I'm trying to bulk change content, in calibre, after importing the fics, because the author information is wrong. It is wrong because the downloader program I had used before would insert the name of the program creator into the html coding itself (i.e. <meta name='author' content='Grzegorz Hordynski' />).

Once the fics get imported into calibre, it reads the <meta name='author' content='Grzegorz Hordynski' /> information and puts that as the author's name. How do I go about automatically changing <meta name='author' content='Grzegorz Hordynski' /> to <meta name='author' content='Author Name' /> ? Am I making myself clear? You say there's a way to do that on the Conversion screen, and that it's possible even though it's not meant for it, but I don't understand how. What expression do I need to enter to make this change because obviously pasting the above line does not work?

My ultimate goal is to eventually convert all my saved fics into epub, after which I plan to delete the imported books from the calibre library. I do not intend to use it for ebook management, just for converting.

(I just bet it's going to turn out to be something really simple and I'll be kicking myself for missing something painfully obvious!)

[Starson17]

Originally Posted by citac

Yes, I'm trying to bulk change content, in calibre, after importing the fics,

To bulk change content, you would select the ebooks, bulk convert them to the desired final format (it can be the same as the original format) and set the S&R regex to replace content.

because the author information is wrong. It is wrong because the downloader program I had used before would insert the name of the program creator into the html coding itself (i.e. <meta name='author' content='Grzegorz Hordynski' />).

Afte the file is in Calibre, this won't affect Calibre, even though it's wrong. Calibre will control its own metadata.

Once the fics get imported into calibre, it reads the <meta name='author' content='Grzegorz Hordynski' /> information and puts that as the author's name.

This will produce the wrong author in Calibre's metadata, and you need to bulk edit these records to fix the author metadata.

How do I go about automatically changing <meta name='author' content='Grzegorz Hordynski' /> to <meta name='author' content='Author Name' /> ?

As above. You can't do it before import (at least not with Calibre.) You'll get the wrong author name (unless you use filename to set the author and your filename is correct.) After import, you fix it in Calibre's metadata, and optionally fix the content if you want.

Am I making myself clear? You say there's a way to do that on the Conversion screen, and that it's possible even though it's not meant for it, but I don't understand how. What expression do I need to enter to make this change because obviously pasting the above line does not work?

That's a regex question, and I'd need to see your original content to tell you how to set the regex to match. However, if the content has:

<meta name='author' content='Grzegorz Hordynski' />

then a simple Search for "Grzegorz Hordynski" and Replace with "Author Name" should do it to create:

<meta name='author' content='Author Name' />

My ultimate goal is to eventually convert all my saved fics into epub, after which I plan to delete the imported books from the calibre library. I do not intend to use it for ebook management, just for converting.

You'll be missing a lot of nice features, but if that's what you want, that's fine. I started thinking that's all I needed Calibre for, too. Then I tried the Recipes, then my wife started using the Content Server for her smart phone, then we decided to track the books we'd read, then .... and here I am - unpaid volunteer tech support and occasional open source developer for features I want and no one else has time for

[citac]

OK, I am doing this as we speak.

I have imported six files from an author saved in Fandom > Fic > Individual authors > Author folder. The author tag inside the file is as above. The title tag is <title> Title by Author </title>. The files themselves are named

01 Title.htm
02 Title Coda Subtitle one.htm
03 Title Coda Subtitle two.htm
04 Title Coda Subtitle three.htm
05 Title Coda Subtitle four.htm
06 Title Coda Subtitle five.htm

These files have no author name in the title of the file because they are all saved in their author folder. I have another group of fics which go into Fandom > Fic > Various authors, and contain single stories out of several an author has written (so one story from author a, one from author b etc etc). I haven't imported any of those stories, but they are named Author name - Title.html.

After import, calibre reads the author and title tags and populates the metadata with what it finds there. It also doesn't read the order in which the fics should be read, and alphabetizes the fics, which gives me

Title.htm
Title Coda Subtitle three.htm
Title Coda Subtitle five.htm
Title Coda Subtitle four.htm
Title Coda Subtitle two.htm
Title Coda Subtitle one.htm

I will obviously have to do something about the files themselves (though it will be a massive undertaking - I have thousands of fics, the earliest dating from 2004) before any further importing into calibre, because what I'm trying to do now simply isn't working.

OK, so the files are imported now. I click Convert books, go to search and replace, enter Grzegorz Hordynski into the First expression window, the Search Regular Expression line, and enter the correct author name into Replacement text lin. I hit OK. calibre converts the files into epub. Author is still Grzegorz Hordynski. A simple Search for "Grzegorz Hordynski" and Replace with "Author Name" is not working.

What am I doing wrong???

This is what I have in the html coding of the file (details changed but you should get the picture):

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name='author' content='Grzegorz Hordynski' />
<title> Title by author </title>
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000FF" vlink="#800080">
<div> Title by author </div>
<div> Summary of the story. Disclamer. Author notes/thankyous. </div>
<div> Story word count: 33355 </div>
<div> <a href="http://www.site.com/">Original link to the story</a> </div>
<br />
<br />
<div>

<hr />

<div> Content of the story. </div>
</body>
</html>

So, first: what is the file naming scheme I will have to apply to the files in order for calibre to correctly interpret individual authors who may have up to 60 fics saved in their respective folders (Fandom > Fic > Individual authors > Author > saved fics.html), and authors for whom I have saved only a single story out of several they have written (Fandom > Fic > Various authors > saved fics.html)? Should I apply Author name - Title.html to all the fics I have saved? Will it correctly recognize files named Author name - Title 01.html, Author name - Title 02.html files?

What about those with a more complicated naming scheme, like those saved in

Fandom > Fic > Individual authors > Author folder > Series

Book 1 - Title - Part 1 - Subtitle.txt
Book 1 - Title - Part 2 - Subtitle.txt etc
Book 2 - Title.txt
Book 3 - Title - Part 1 - Subtitle.txt
Book 3 - Title - Part 2 - Subtitle.txt etc
Book 4 - Title - Part 1 - Subtitle.txt
Book 4 - Title - Part 2 - Subtitle.txt etc

Will I need to add author name to all these files, even those in author folders?

I also need to change the incorrect tags inside the fics themselves. Is there a program that will do this for me? Yes, this will not affect calibre, but like I said I don't intend to manage anything with it, just convert and transfer the result to my reader.

Frankly, at this point I don't even care about any additional features, I just want the conversion to work, and I want to find a way to change the content of the converted file, the way others are changing headers and footers left in their files after converting them from lit or whatever. I want the metadata to be saved inside the file, so that a reader I uploaded the fics to can read the metadata and present it correctly on the screen.

I'm going to try connecting calibre to my device and transferring the converted files to see how CoolReaer will read the files.

[citac]

OH MY GOD, it just deleted everything on my SD card!!!

I had three fandoms saved there in my fiction folder, and notes from my postgraduate lectures, word documents and powerpoint presentations - you've got to be kidding me!!

I clicked Save to disk, Save only epub format to disk, it recognized Pocketbook's internal card, but when I clicked on Removable disk (G it asked if I wanted to format the card. Of course I don't, I have important documents there. So I click Cancel, and I get

G:\ is not accessible.
Data error (cyclic redindancy check).

What the hell is that??? :headdesk: All those files gone. I will have to re-download them from our university's server. I really did not need this.

And it's not just those files; I had all my apps backed up to that card. I had information for my contacts stored there. I had appointments saved in my diary app. How the hell am I going to rebuild those?

[speakingtohe]

Cyclic redundancy has nothing to do with calibre AFAIK. It is an IOS error.
Usually a disk error caused by bad spot on disk. Electrical shock or scratches on media or removing media before file writing is complete.

Try removing card, reboot computer and try the card again.

BTW it is your OS wanting to reformat the disk because it could not read the system files to write a file. Nothing was deleted the card was not readable so calibre could not write anything or delete anything.

Helen

[citac]

I had to leave it all for a while because I thought there was something obvious I was missing, hence not being able to make the necessary changes.

I just removed the card from my device, inserted it into the card reader on my computer, and you're right, I didn't lose the files. I am sorry for losing my temper though.

But what am I supposed to do when I want to send files to my device? Or should I just forget about it and use the SD card reader on my computer and transfer the files manually? I thought calibre was supposed to be able to do that. Why am I having so much trouble with it?

[speakingtohe]

There is probably something wrong with the card reader or the card.
Maybe just dirty contacts.
Best thing to do back up your data from the card just in case.

Perhaps when you are 100% sure you have a good backup you could reformat the card.

Or get another card and start fresh.

There are lots of ways to send files, but try sending them again (after backup ).
Less confusing IMO.

For instance you could send them to the card when it is in the computer card reader.

To beat the horse to death, the errors you were getting are basic i/o errors. Calibre uses the computer operating system i/o functions to transfer data.

You were probably doing everything correctly but had bad luck (data read error). I am sure you will get it all done very soon.

Helen