New Yorker too large to email

[mkgtu]

Don't know what, if anything, can be done about this. But today for instance (Feb 26) the downloaded New Yorker was 93mb. Way too large for emailing. Even after editing the epub by compressing images to the default (20 and 40) level the resulting file is still 83mb. Still too big.

So I converted the 83mb epub to mobi and the result was a 15mb file. I then the converted that mobi to epub and emailed it. Quite a difference! But at least it emailed ok to kindle and displays as it should.

The New Yorker does occasionally create overly large files. Just wondering if there is some way that the recipe can prevent that. I suppose one can create custom recipes that omit images or whatever. But those custom recipes begin to fail when the base recipe is updated. But in this case even severely compressing the images didn't shrink the file by much, so images may not have been the problem.

It's beyond my novice ability to fix.

[DNSB]

If you use the https://www.amazon.ca/sendtokindle site (change the .ca to the appropriate Amazon site for your country), you can send up to a 200MB file.

[unkn0wn]

https://github.com/unkn0w7n/calibre/...1c870c0fdb8334

[DNSB]

Thanks! An nice fix to the recipe to allow continuing to automate the procedure.

[mkgtu]

Sounds good because it preserves the automation. But I'm curious about this line in the new recipe:
self.log.warn('Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold')

Does this mean that in order for the necessary compression to happen the Common Options "Page Setup" has to be set to one of the "Kindle" options. Otherwise how does calibre know this is a "Kindle Output"?

As has long been the case, if you set the Page Setup as any "Kindle", the resulting filename only includes the Title itself, but none of the Date information. For that reason, I have always used Generic E-ink HD as the page setup because that allows the inclusion of the date. Kindle itself has no problem displaying the dates, but calibre still prevents the dates from appearing in any specifically "Kindle" outputs. And, at least to me, the dates are really important on my Kindle so I know which issue to open.

I would hope that this doesn't create an either/or situation: you can either have dates, or files that are small enough to email, but you can't have both?

One solution is to remove the restriction on including dates in the "Kindle" outputs if that's possible. I'm wondering if that's a leftover from long ago when maybe the Kindle mobi files weren't compatible with the dates. But mobi is dead. Amazon now accepts epubs.

And I'd also note that while the New Yorker is a frequent culprit in creating gigantic files, The New York Times, The Spectator, and The Economist also occasionally break the bank - especially for "double" issues.

I use Fastmail because it allows a total email size of 70mb, but with encoding etc that usually means an attachment of 50mb. GMX is also 50mb, GMail only 25mb.

[unkn0wn]

yes, output profile needs to be kindle and I think you wouldn't need compression after this update.

I also face the title-date problem, especially when calibre shows that the mobi is already present when i connect my device, but then its the yesterdays mobi i left there to read.

set profile as kindle and use this code.. the titile will include date too.

    def __init__(self, *args, **kwargs):
        BasicNewsRecipe.__init__(self, *args, **kwargs)
	from datetime import date
        if self.output_profile.short_name.startswith('kindle'):
            # Reduce image sizes to get file size below amazon's email
            # sending threshold
            self.web2disk_options.compress_news_images = True
            self.web2disk_options.compress_news_images_auto_size = 5
            self.log.warn('Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold')
            self.title = self.title + ' [' + date.today().strftime('%b %d, %Y') + ']'

we can add this code to all the recipes you mentioned.

[kovidgoyal]

There is no compression needed. And if you connect your Kindle with a USB and send downloaded news directly you will see why kindle profile causes date to be removed. That's not going not change. If you are sending EPUBs anyway, which will be *converted* by amazon's sub par conversion service, there is no reason to use Kindle profile.

[unkn0wn]

i know date shows up under the periodical in kindle but if i have past editions of the periodical, calibre thinks that the book is already present, then i get confused if the new edition in the library is actually synced to kindle or not.

With different titles (of date) I can check which one's of the same recipe are synced and still on my kindle.

thats why I actaully added a piece of code to change title + date if output profile is kindle for some of my recipes.

[kovidgoyal]

Unless you are sending them to the kindle using some means other than USB transfer via calibre, there is no reason for it to get confused. Matching is not done via title/author in such a case.

[mkgtu]

Kovid, if I read between the lines, you seem to feel that transferring via USB to a Kindle device is the norm.
At least in my case that is far from the norm. Automated emailing is for me a priority. I want/need these items saved (via email or "send to Kindle") to the Amazon cloud so I can download them to my phone or tablet Kindle apps. That can't be done via USB.

I read your comment to imply that if you're doing anything other than USB transfer, then you may be S--t out of luck.


Sent from my SM-G986U1 using Tapatalk

[mkgtu]

Originally Posted by unkn0wn

yes, output profile needs to be kindle and I think you wouldn't need compression after this update.



I also face the title-date problem, especially when calibre shows that the mobi is already present when i connect my device, but then its the yesterdays mobi i left there to read.



set profile as kindle and use this code.. the titile will include date too.

    def __init__(self, *args, **kwargs):

        BasicNewsRecipe.__init__(self, *args, **kwargs)

from datetime import date

        if self.output_profile.short_name.startswith('kindle'):

            # Reduce image sizes to get file size below amazon's email

            # sending threshold

            self.web2disk_options.compress_news_images = True

            self.web2disk_options.compress_news_images_auto_size = 5

            self.log.warn('Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold')

            self.title = self.title + ' [' + date.today().strftime('%b %d, %Y') + ']'

we can add this code to all the recipes you mentioned.

So has this code been added to the stock recipe, or do we have to create a custom recipe and add the code, which would mean the recipe might stop working if you update the stock recipe.

I used to, and sometimes still do, create custom recipes just to add code to "left align" text. But I keep getting frustrated by custom recipes no longer working after changes to the underlying stock recipe. Then I'd need to re-create a custom version of the stock recipe.

Some recipes left align text quite well, others don't. And justified text created by Calibre usually leaves huge gaps between words, making articles difficult to read or scan.

Sent from my SM-G986U1 using Tapatalk

[kovidgoyal]

Originally Posted by mkgtu

Kovid, if I read between the lines, you seem to feel that transferring via USB to a Kindle device is the norm.
At least in my case that is far from the norm. Automated emailing is for me a priority. I want/need these items saved (via email or "send to Kindle") to the Amazon cloud so I can download them to my phone or tablet Kindle apps. That can't be done via USB.

I read your comment to imply that if you're doing anything other than USB transfer, then you may be S--t out of luck.


Sent from my SM-G986U1 using Tapatalk

If you are emailing you dont need to use a Kindle profile. You arent emailing to a kindle. You are emailing to amazons conversion service.

[mkgtu]

True. And I never do use the Kindle profile because that profile omits the dates in the title. So my quandary, or question is, it looks like the suggested code above to be added to recipes only triggers image compression if the output profile "begins with 'Kindle'". And if you have to set the profile to Kindle you can shrink images but you lose the date in the title.

Don't know how it would work in practice, but it's the SIZE of the file that's the problem. In the New Yorker case I cited, a 93mb file attachment would be rejected by almost any email service, regardless of whether it was being emailed to Amazon or my grandmother.

And while the inclusion of dates in the Kindle output may have caused problems in years gone by, is that still a problem, given that Amazon has dramatically changed the way they do things? Is it still necessary to exclude dates in the titles of news in the Kindle output? If the dates were allowed then we could still use the code above to limit file sizes without losing the dates, as the dates are really the only way to distinguish among multiple issues of the same magazine on a Kindle.

Sent from my SM-G986U1 using Tapatalk

[mkgtu]

Originally Posted by unkn0wn

yes, output profile needs to be kindle and I think you wouldn't need compression after this update.



I also face the title-date problem, especially when calibre shows that the mobi is already present when i connect my device, but then its the yesterdays mobi i left there to read.



set profile as kindle and use this code.. the titile will include date too.

    def __init__(self, *args, **kwargs):

        BasicNewsRecipe.__init__(self, *args, **kwargs)

from datetime import date

        if self.output_profile.short_name.startswith('kindle'):

            # Reduce image sizes to get file size below amazon's email

            # sending threshold

            self.web2disk_options.compress_news_images = True

            self.web2disk_options.compress_news_images_auto_size = 5

            self.log.warn('Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold')

            self.title = self.title + ' [' + date.today().strftime('%b %d, %Y') + ']'

we can add this code to all the recipes you mentioned.

I still have concerns about the date inclusion. This code does add the date to periodicals when using Kindle output, but obviously only with recipes that include this code.
When selecting Kindle output, that is a global option only. Recipes that include this code will add the date, but other news downloads which don't contain this code will exclude the date because they are all forced to use the Kindle output profile. I schedule about 14 periodicals for download. Most of them don't have size problems, only the 2 or 3 I mentioned, and those only occasionally. So if I switch to using the Kindle output for the sake of those 2 or 3, I either lose dates on most downloads or I have to customize all my scheduled download recipes to include this code, or at least the part that forces the inclusion of the date in the title, which means those recipes will be frequently outdated.

Seems I'd be better off not using the Kindle output and if I get an occasional file that is too big to email I can just temporarily switch to Kindle output and re-download the offending issue.

Or, another thought, those who don't email publications but instead transfer using Calibre and a device connected by USB, must also be using the mobi or azw output format. (I don't think Kindle can display epubs natively) So maintain the practice of not including dates in the mobi format since it causes problems transferring to Kindle via USB, but allow dates to always be included in epub files, even those created created using the Kindle output profile, because dates don't create problems with emailed epubs or epubs uploaded using the "send to Kindle" app. And they don't cause problems with Amazon's conversion process.

If dates could be included in epub news files, there'd be no problem switching to the Kindle output profile globally.

Sent from my SM-G986U1 using Tapatalk

[kovidgoyal]

You dont need any special image compression. Just run the download again and see for yourslef.