Tool to automatically download and check epubs

[skreutzer]

For the EPUBs in Free eBooks-de/de, I started a tool to automatically download and epubcheck them, unfortunately their technical quality isn't quite as good as the proofreading is claimed to be. Now some users started to correct their uploads manually, maybe we might even try to automatically fix them. This makes me wonder if the English EPUBs could need a similar technical check, either by you guys or, if you have a public list of them like the Wiki list, via automated download and epubcheck. If you have any questions or need a translation of the thread I've linked, just feel free to ask.

In theory, more could be done with a library of valid EPUBs, but as they're not all in the Public Domain and the legal state per book is unknown, it's way too risky to work with them except in private (which I would consider as a waste of time, as other people would want or need such results as well).

Could you please prominently sticky post a link to the origins and history of the Patricia Clark Memorial Library? I was unable to find it, but maybe I have just overlooked it.

[Jellby]

We are not going to automatically fix them, but some uploaders (including myself) would appreciate reports and try to fix problems.

[DaleDe]

Originally Posted by skreutzer

For the EPUBs in Free eBooks-de/de, I started a tool to automatically download and epubcheck them, unfortunately their technical quality isn't quite as good as the proofreading is claimed to be. Now some users started to correct their uploads manually, maybe we might even try to automatically fix them. This makes me wonder if the English EPUBs could need a similar technical check, either by you guys or, if you have a public list of them like the Wiki list, via automated download and epubcheck. If you have any questions or need a translation of the thread I've linked, just feel free to ask.

In theory, more could be done with a library of valid EPUBs, but as they're not all in the Public Domain and the legal state per book is unknown, it's way too risky to work with them except in private (which I would consider as a waste of time, as other people would want or need such results as well).

Could you please prominently sticky post a link to the origins and history of the Patricia Clark Memorial Library? I was unable to find it, but maybe I have just overlooked it.

It would be fine to check the other books. They are all compliant with life+70 requirements. The entire library has been checked. This how we lost about 5,000 books. There is a history of MobileRead in the wiki but not specific to the library. The name of the library was changed when Patricia Clark had an untimely death. She was a major contributor and a MobilRead editor at the time of her death.

Dale

[skreutzer]

OK, I'll look into adjusting my tool [1] to take the ePub Books forum or the corresponding index as source for threads with EPUB attachments as soon as time permits.

Originally Posted by DaleDe

They are all compliant with life+70 requirements.

I naively assumed this, especially as I see no point in a restrictively licensed public library, but I was soon corrected that the copyright notice and upload policy allows cases where

the copyright holder has given specific permission for distribution

which doesn't require the work to be in Public Domain [2]. As neither the upload thread nor the Wiki list contains any machine readable information about whether the work is in Public Domain or only provided to mobileread with special permission, at least I'm all the time at risk because people might use my software to download those works and redistribute them, and depending on the law of certain countries, this might be considered aid to copyright infringement.

I just realized that the index has a nice notice regarding Patricia Clark, while the eBook forums might be already way too full for it.

[1] Note that with such enhancements, the idea of converting threads to EPUB and PDF for printing immediately pops into my mind...
[2] Seems the posting guidelines under item 8 make such uploads CC0, and (some German) uploaders are not aware of that. If so, the copyright notice containing "the copyright holder has given specific permission for distribution" is misleading, it should explicitly state the CC0 licensing.

[Jellby]

Originally Posted by DaleDe

They are all compliant with life+70 requirements.

Originally Posted by skreutzer

I naively assumed this, especially as I see no point in a restrictively licensed public library, but I was soon corrected...

Indeed, the library is basically life+70, but it also holds other books which are not public domain, including (but not limited to): copyrighted translations uploaded to Project Gutenberg, for which redistribution is allowed (probably with some conditions); Creative Commons works; works uploaded by the author (whether or not it is OK to redistribute them, is up to the author, we only know we can host them, because the author uploaded them).

I'm all the time at risk because people might use my software to download those works and redistribute them

Not being a lawyer in any country, my opinion is that if you download the whole library and host it elsewehere, you may be doing something wrong. If you provide a tool that someone can use to download books (which they can easily and legally download otherwise, anyway) and they decide to disregard any possible copyright/license and distribute them... well, I can't say you're to blame, not more than MR.

[skreutzer]

Originally Posted by Jellby

but it also holds other books which are not public domain, including (but not limited to): copyrighted translations uploaded to Project Gutenberg, for which redistribution is allowed (probably with some conditions); [...]; works uploaded by the author (whether or not it is OK to redistribute them, is up to the author, we only know we can host them, because the author uploaded them).

And this breaks it for me completely, because neither in the threads nor in the lists nor in the index I can learn about which is which. And maybe you can't too, as users become inactive. If users want to make use of the uploaded texts (redistribution, for instance), which a basically life+70 library suggests to be permitted, they have to re-evaluate the legal status individually for every work over and over again.

Originally Posted by Jellby

Not being a lawyer in any country, my opinion is that if you download the whole library and host it elsewehere, you may be doing something wrong. If you provide a tool that someone can use to download books (which they can easily and legally download otherwise, anyway) and they decide to disregard any possible copyright/license and distribute them... well, I can't say you're to blame, not more than MR.

I didn't do so and don't intend to do so, but that's what almost certainly happens. My validator startet out because a user asked for a way to download all EPUBs in the German Wiki list automatically. He has done it by hand, and that's quite a painful process. As I'm an publishing automation guy and was naively thinking the library is in Public Domain, I wrote a downloader, and the user immediately uploaded the works immediately at some other web space publicly [1]. Uploaders of restrictively licensed works complained, and the link was removed (I don't know, however, if the files are still available publicly at this or at other locations). After some time, I attached epubcheck to my already existing downloader, because obviously I can't epubcheck files without downloading them first. Additionally, without having a library of Public Domains only, people can't mirror those works on other places and everyone who wants to download the whole library, will do so separately using your resources exclusively (and my downloader just invites them to do so).

Originally Posted by Jellby

well, I can't say you're to blame, not more than MR.

I hope so, yes, because I only retrieved copies and never distributed them. The main difference here is that MR has permissions from the uploaders while I have none, and by providing a tool to obtain the whole library at once, users of it might be unaware that the library is in part restrictively licensed. I might have to notify them that they're obliged to use the library only privately except they can resolve the legal status of individual files. By using the downloader, they don't get to see any legal notices for instance.


[1] In some ways, that was very kind of him, because the download takes around 6+ hours, as there is a delay of 5 seconds (an eternity) between two downloads, and every download of the whole library from his server would have saved mobileread resources.

[DaleDe]

Yea I momentarily forgot Creative Common. The eBooks that are Creative Common must be marked as such. However, someone could declare their book public domain without the book being marked as such, I suppose.

Dale

[skreutzer]

DaleDe, I'm glad to hear that I'm not the only one who sometimes isn't all too aware of the conditions. Not that they're well understood by now, but at least I need a reminder from time to time.

CC0 and Public Domain are perfectly fine, and as I would consider them as "default" for most works in the lists, I don't worry about their legal status isn't explicitly stated in lists or threads. Instead, the minority of restrictively licensed works are a problem for me, as this information isn't machine-readable available somewhere, so they're indistinguishable from the free works, therefore the whole library looks like a minefield for me. When working with the entire library, there's absolutely no way to pass the field without stepping on a mine, and I have no method for finding paths around them.

[doubleshuffle]

Perhaps you should accept the fact that the library isn't an entity to be downloaded automatically, but a collection of individual books to be found, downloaded and enjoyed individually by human beings. You have created a problem that only exists because you have created it. I don't see any reason why we should put any effort into making the library compliant to your ideas, sorry.

[DaleDe]

As I have said before. The library is compliant. The moderators here at MobileRead have inspected every author in the library and compared it to life+70. That is why we now have an offline library to collect the out of compliance books. They will be added back in on or after Jan 1st of each year when they become eligible. No one else need worry about this. The Moderators are doing the job.

Dale

[doubleshuffle]

Originally Posted by DaleDe

As I have said before. The library is compliant. The moderators here at MobileRead have inspected every author in the library and compared it to life+70. That is why we now have an offline library to collect the out of compliance books. They will be added back in on or after Jan 1st of each year when they become eligible. No one else need worry about this. The Moderators are doing the job.

Dale

Yes, I know. That's why I said to skreutzer "compliant to your ideas." I was referring to his ideas, not to Life +70.

[skreutzer]

Originally Posted by DaleDe

As I have said before. The library is compliant. The moderators here at MobileRead have inspected every author in the library and compared it to life+70.

If this is the case, my “ideas” are no different than yours. The only problem I'm creating is that

Originally Posted by Jellby

Indeed, the library is basically life+70, but it also holds other books which are not public domain, including (but not limited to): copyrighted translations uploaded to Project Gutenberg, for which redistribution is allowed (probably with some conditions); Creative Commons works; works uploaded by the author (whether or not it is OK to redistribute them, is up to the author, we only know we can host them, because the author uploaded them).

contradicts what you've just stated. Could you please resolve what's actually true? If life+70/CC0 without exception is the policy, I then would notify the German uploaders who uploaded restrictively licensed works, so they take their uploads down (no difference if users get those works by manually downloading them or as part of the entire library). Or could it be the case that you've checked those uploads and that they're in the Public Domain, with only those uploaders thinking wrongfully that they're not? Then it wouldn't matter what the uploader thinks or wants to restrict.

After all, I'm just looking into what 8142 EPUBs actually means.

Update:

• In 2007, somebody posted mobileread ebooks on the Usenet, and it was made clear that there were restrictively licensed works among them. If the library is life+70/CC0, then all of those works have been removed from the library by now.
• In 2011, it became apparent that somebody sold the e-books from mobileread burned on a CD-ROM. Mobileread moderators made the guy discontinue the product, another user obtained a cease and desist letter from him. This would have happened without legal foundation if the works would be all life+70/CC0. Some users mentioned that trademark law could be a reason for restrictive licensing, and for German uploads the „Leistungsschutzrecht“ (protection of the effort somebody put into it) could also be a reason.
• Also in 2011, somebody redistributed ebooks from mobileread gratis, without mentioning mobileread as the source. One user suggested that mobileread is financed by advertising and therefore doesn't want other sources to distribute the same files. It seems no action was taken.
• In 2015, one or more individuals uploaded e-books to Kobo Writing Life and sold them, so some uploaders got their titles with original content removed. They interpreted the Posting Guidelines to permit any Creative Commons license, but not other licenses, and were in favor of CC BY-SA-NC (especially the NC clause to prevent such instances from happening in the future again).
• In 2015, as a user redistributed the library obtained from my downloader, one uploader stated that at least one of his uploads is not in the Public Domain and threatened with “serious consequences”. This uploader and a second one asked for removal of their uploads from the external public copy of the German EPUBs. A moderator removed links to this external mirror, maybe not because of legal reasons, but because of respecting the wishes of the authors. Via PM, another uploader told me that he wants to prohibit any commercial use, despite the fact that his works are in the Public Domain. He refused to tell me how he wanted to achieve this, especially after he has uploaded some of the files to a company which does use them commercially.

[DaleDe]

Yes, in the past there was a difference. When the download site was set up we were USA compliant, not life+70. The site owner is from Switzerland and began to be worried about his personal liability, so within the last year he has changed the license to be ok for his country. We do watch for takedown notices and will respond to legitimate notices, but there have been none recently. We do have some cc eBooks with the license in the book for non-commercial use. Over our history the licenses have been USA, life+50, and now life+70. These were accompanied with different server locations. Of course a downloader could be anywhere in the world so the downloader is responsible to download that which is legal in his/her country.

Dale

[beachwanderer]

... another user obtained a cease and desist letter from him. This would have happened without legal foundation if the works would be all life+70/CC0 ...

As DaleDe explained:

We do have some cc eBooks with the license in the book for non-commercial use.

And exactly this was the case. The letter (which, by the way, was not followed by a bill for a legal fee for the violator as it could have been) concerned one special title and the addressee of the letter admitted the violation without delay.

Concerning ebooks for which MR-members created own front pages there is (at least under german law) another aspect to keep in mind, that you are well aware of:

If the font page contains a design that qualifies as a new work of craft or art, copyright exists for the creator. If the front page contains any photography which was created by the person who did create the front page, copyright for the photographer exists. This is, of course, also true for any such works embedded in the text of the ebook. As a result there could be claims concerning the complete ebook-file.

I case of splitting front page (or illustrations) and text, the next question that arises would be that of the protection of a special layout, special fonts etc. etc.

So on the whole I tend to agree with doubleshuffles statement on the matter.

My expectation would be that if one tries (after loads and loads of interesting legal review of masses of individual titles) to make the whole library compliant to the project you have in mind, another library, or rather two or more of them would result and much be lost.

(And that's the point where you can picture the old guy on the bench in front of the house saying: "Naa, wouldn't be the Patricia Clark Memorial Library then. Need some new names too."

[DaleDe]

The cover is indeed an illustration but often the book itself also has illustrations which are covered by a second copyright. A translation also has a separate copyright. When known, all of these things were considered when the mod's reviewed the works for the latest pass at purging the books that were non-compliant. Of course we are only human. If you find any errors report them to the mods via the little ! symbol under the poster's name.