Embedding audio/video in epub files
 

Has anyone done this? I have been searching around for a solution. Just wondering if anyone here does it? I posted in the Sigil forum as that is what I primarily use. Apologies if it should go somewhere else.

I read this: https://ebookflightdeck.com/handbook/media - Seems interesting.

Also I have never created an epub 3 file yet so it is a little daunting.

What is your process for this, if you have done it? Would be interesting to hear your thoughts. Thank you

Yes epub3 officially supports some specific audio and video file types. Adding media files is quite easy in Sigil using the Add Existing menu. You can also use the Insert Link menu item. Standard html5 audio and video tags work. There are sample epubs that have been posted to Sigil in the past. Search the Sigil and epub forums here.

The real question is what e-readers support it.

I just found that Video mp4 is very broken on the latest Sigil builds because our custom url handler does not properly handle byte ranges that only get used for audio and video when loaded via a custom scheme:.

Here is a related bug report:
I will look into adding some sort of byte range streaming for our sigil scheme. Silly really since we offer up the entire file in a buffer but obviously needed with later Qt versions given our change to a url scheme handler.

Okay that is a Qt QWebEngine/Chromium bug that impacts only audio and video that uses proprietary codecs and a custom scheme handler.

I have added a workaround our URLSchemeHandler to redirect them as file: scheme urls to prevent the partial byte range requests as there appears no way to handle them inside QtWebengine.

This has been pushed to master so mp4 videos now work again in Sigil's Preview in the next release. WebM videos should work already.

Yes

Be aware that it are the rules for HTML5, while EPUB format is not completely the same. You can use it to change it to the following code:

Adapt the location for the video and subtitling files to your situation. This will work in:

• Sigil preview screen;
• PageEdit;
• The free GitDen reader (tested on Android only)

So far, there are standard options. The big challenge arises when you want to embellish your video player with bells and whistles. After trying a dozen HTML5 examples, I can tell you that they all break in one way or another.

Let me know if this is enough or if you want more like me. I think I've found the difference between the HTML5 and EPUB standard and happen to be working on a solution today. Despite many searches on the Internet, I haven't actually found a working EPUB solution to adjust Video views in an EPUB.

Update: I forgot to mention that you can set the size of the video's width and height as a kind of maximum width and height on a large screen. Up to those dimensions, the frr class adjusts the width and height to the physical screen dimensions.

.mp4 is one of the official video resource types that any epub3 must/should support according to the spec. But of course notall readers do. At one point in time there was a multi-media epub3 test epub that was used to show epub3 compliance among readers.

Sigil should support all of those media types but that does not mean e-readers do.

The EPUB standard has many open ends. And even if the video/audio player is supported, the full screen support for a video is also a challenge. The free GitDen reader for Android and IOS loses most keys in full screen mode under Android OS.

Unfortunately, the free GitDen reader is not being developed further. As a result, it no longer meets the current Android requirements. As a result, it can no longer be loaded for people with the latest Android IOS version.

:help:Does anyone know of another EPUB reader for Android that can display videos under the current Android OS version?

I did not search too closely, but I checked the epub3 video in two applications available on Google Play:
com.flyersoft.moonreader
com.s2apps.reader

Of course, their imperfections can be annoying as there is no perfect application for reading epub files.

Libera claims to support Audio/Video in epub3 books. Do you have a test book I can try?

Sent from my Pixel 4a using Tapatalk

An app is usually a better solution as iOS, Mac, Android, Windows or Linux is needed anyway to have video + audio. There are free frameworks and once it's done once the next time is easy. Almost everyone would have to download an App even if epub3 did what what you want.

People have been making multimedia since 1980s and commonly from 1990s. The idea of epub3 competing with multimedia apps and HTML5 websites was a crazy design decision.

I disagree. Epub3 can easily handle video and audio and javascript. Bad reader implementations are the issue.

Yep. The willful ignoring of (and/or deviation from) epub specifications is the main issue. Otherwise, Audio/Video would just work. The same holds true for epub reading app devs: iOS, Mac, Android, Windows or Linux is needed for their reading app anyway, so why not just make your EPUB3 reading app work like it's supposed to to begin with?

I've not yet tried it, but Bluefire Reader for iOS is using the latest RMSDK. That might work for this multimedia ePub 3 eBook.

FYI, eBookflightDeck has been dead for what, 7 years now?

Hitch

The challenge is not so much displaying a video on an EPUB page, but in the fact that switching to full screen only leaves the standard play, pause, time scrollable and back to normal screen buttons. Or self-defined keys expire in full screen mode. So the button to turn subtitles on or off has also disappeared.
The EPUB readers I know don't allow debug mode. So I can't see if the EPUB reader uses a measure of shielding the HTML/CSS code in the shadow DOM or if the EPUB reader in full screen mode falls back to OS screen routines.

But then the user needs to install a compatible app. If you want a single download to work, then an App always beats any attempt to use epub3 with Audio & Video, i.e. Multimedia.

By not doing an app for multimedia and assuming the epub3 will work for the customer is worse than expecting someone to install fonts on a Kobo or Kindle (mostly they won't).

Unless you test every ereader and App you won't even know which ones work with epub3 used to do multimedia.

But then you'd need to create and maintain apps for different operating systems.

Many eInk readers have indeed problems with embedded multimedia content, but most epub3 apps for tablets and PCs support them just fine.

Problems? Hell, that's the kindest word I can come up with for multimedia on eInks. :-) Catastrophe is the next-best word.

Hitch

Most people already have ZERO programs or apps on PC, Mac, Linux, iOS or Android that support full multimedia epub3 and that is a huge more number of apps/programs to test and support than OSes.

It's possible to use ONE FW for Linux/Mac/Windows multimedia, one for iOS and one for Android. A lot would be in common and once it's made you can use same content for all three for updates and new titles only needs new content. People have been doing this for over 30 years on desktop and over 12 years for iOs and Android. There are many excellent free tools and frameworks.

Epub3 multimedia is limited compared to full HTML5 + code on a web server which is clunky and limited compared to a multimedia app. Even the Web Multimedia has to be tested with different browsers. The epub3 is much worse because by default no-one has a an epub3 that supports multimedia.

Huh?

All epub3 support multimedia. Only some epub3 readers are broken.

Instead of supporting you own apps (which are a pain) simply provide a list of known good epub3 readers. And that includes almost all modern browsers and any epub3 reader that uses those engines. Many are completely free to download and work on all major platforms including iOS and Android.

Make it clear that your epub3 uses multimedia so that broken e-ink users can self select away

Readium and Thorium are good examples of Browser based epub3 readers that work well on multiple platforms.

The only challenge is that objective comparison of what an EPUB reader can and cannot do is impossible to find. With the EPUB code below, I can see an HTML test score in an IFRAME.

And what's even more interesting, you can also see what is partially supported. When you click on the circle, you will see what is not supported!

These are different test results:
Firefox version 90: 513
Sigil preview window: 523
Gitden Reader: 506

PageEdit asks for permission to open the website in the browser. So I'm not sure whether the Gitden app itself displays the website or whether it also displays the website in a redirected browser view.

Since none of the EPUB readers have an inspection mode except for IOS, it is very difficult to investigate erroneous displays in an EPUB reader.

As for the Sigil preview window:thanks:, a big compliment from me for the high degree of support for HTML mode!:iloveyou:

Happy you're happy. But the browser score is sort of pointless in an EPUB. It's testing for things that aren't based on EPUB specifications. There are probably many things Sigil's WebEngine browser will support that EPUB doesn't.

Back to my multimedia EPUB Video challenge in the EPUB readers. It's actually a chicken and egg problem. If you don't use video, you won't run into any problems. If it appears that there are no problems, the problems are not solved.

If you, as one of the Epub readers market party, limit the size of an EPUB and the EPUB standard requires both an MP4 and a WEBM file, yes, then it will be a very short and therefore pointless video.

If none of the EPUB readers support debug facility, then how can you fix EPUB reader errors?

It is a misconception to think that an EPUB reader is a black box that does not reveal how that black box works. A valid combination of HTML, CSS and JavaScript in an EPUB file can be rendered cleanly and successfully. The EPUB reader is built around a standard rendering solution for HTML, CSS and JavaScript. On top of that render engine solution, the specific EPUB rules are implemented. As long as you respect the specific EPUB rules, you can just run HTML, CSS and JavaScript code in your EPUB.

The version of the render engine used in the EPUB reader determines what can and cannot be done within the specific EPUB rules. In addition, one EPUB reader adheres to or has implemented the EPUB rules better than the other EPUB reader. That is why you can successfully run a bootstrap program in an EPUB file with some adjustments.

The irony is that in the early days of HTML5, not everything worked either. Those were reported as bugs and those bugs have been resolved over the years. The knowledge to bypass these HTML5 bugs with all kinds of "hacks" and best practices is slowly disappearing from the Internet. That's a shame because sometimes they can also be used in an EPUB file.

In the EPUB market divided by dominant market parties, openness that can contribute to solving bugs is "hard to find". Bookworms who are addicted to a paper book with the turning of a paper page with page number, you don't get that until reading an EPUB book. Even digitizing a page turn doesn't help with that.
On the other hand, the question is whether the current generation that has grown up with interactive multimedia will buy an EPUB reader to read chunks of text supplemented with a still picture with maybe audio text.
The first market party to understand this has taken the first step to enable debugging in its EPUB reader.

I'm happy, well I can test the HTML, CSS, and JavaScript in Sigil and then hope the EPUB reader will do the same. A low score also makes it clear what is or is not supported. And that gives the opportunity to try whether it will work with an old HTML5 hack or whether it is better to forget about certain HTML code.

Actually the version of the render engine has very little to do with what EPUB rules they will support. An EPUB3 rendering engine should (and can) support all of the EPUB3 specs. Same for an EPUB2 rendering engine. Devs don't choose to defy EPUB specifications in their rendering engines because doing so would be hard to do. They defy EPUB specifications because they (or their employers) simply don't care about complying with all of it.

Just remember that Sigil is not an EPUB rendering engine. It's a browser that we use to approximate what an epub might look like on an ereading system. We try to support everything that the epub specifications say should be supported. But it's not to be trusted implicitly. It will also probably support many things that very few mainstream rendering engines will. The devices people will be reading your work on should always be what's used for final testing. It's better to fail gracefully than spectacularly where ebooks are concerned. Unsupported CSS that creates unflattering-looking text is still much, much better than unsupported CSS that creates something unreadable.

Not to mention that the various generations of the ePUB specs are strewn with MUST and SHOULD. Treating any SHOULD item as a null operation makes an ePUB renderer compliant. See 2.2 EPUB Reading Systems from the EPUB 3.2 standard documentation.

For instance:

I totally agree with DiapDealer, but when Philips decided not to release certain content on the superior VCC recorders, the market chose the lower quality brand that did release that content. Why would a simple digital copy of a paper book suffice without the additional digital options, giving the digital reader a completely different experience?

In a culture of sixes at school, it is enough to pass with only sixes instead of tens. As a buyer of a six culture-based dominant EPUB reader, there is no way of knowing beforehand how well or how badly that new model follows the EPUB standard.

If you want to create content for an EPUB, you have no idea what to avoid (well video is definitely, but probably not the only thing) which is asking for trouble, in today's sixes culture for EPUB readers.

There is simply no website that provides an objective comparison for EPUB readers and applications. The introduction date also says nothing about the (im)possibilities here. What I like is that the developers of Sigil are better and probably more likely to release a new bug fix release than the Firefox developer community.

Sorry, you're losing me with your various analogies. If you want to create epub content: simple is usually best. If you're not sure something will work: ask. Someone here (or elsewhere) will know. That it works in Sigil's Preview is not a very definitive measuring stick for experimental epub content.

Yes, I could not agree more. The MUST, SHOULD, and MAY verbiage of the spec really sets an extremely low bar (too low imho) to be called a "compliant" epub3 reader.

Given the eventual epub spec convergence with living html standard, my guess is that most future epub3 readers will use browser engines as their underlying engine which will make SHOULD level very easy to obtain and raise the bar for what a epub3 reader should handle. Both Readium and Thorium already do that as does EpubJS, and Bibi as well.

As for file size limits, hosted remote resources are one of those SHOULD items that really should have been a MUST.

It seems that the manufacturers have not tried very hard to raise the EPUB standard to a higher level. If no one is going to complain, the discipline of the market will not work either. In the longer term it is also not in the interest of the manufacturers if their customers turn away from EPUB.

My video problem is that there is actually very little information to be found. With some EPUB readers you can disable the standard video keys and also replace them with your own keys as long as you don't touch the key full screen. At that moment you lose your own keys and the video player reverts to hard-coded keys.

What is noticeable if not native English speaking is that the subtitle key shown initially is not shown in full screen mode. Great chance that at that moment the shadow DOM is used that the manufacturer uses to shield its incomplete default full screen video setting. In itself, with a correct implementation, there is, but with a faltering application, this is less to live with. Firstly, I don't believe that there are no deaf people in English-speaking countries and secondly, if there are, they probably want to be able to turn the subtitles on or off in full screen.

Does anyone have experience how to use the video player shadow DOM in an EPUB?

Why would you need to manipulate the shadow DOM? What do you mean by "keys"? If you want to add subtitles, that is already supported. The epub3 test epub shows them being used.

The word "keys" instead of button was an unfortunate choice of mine in the context of the question. :smack:

As a non-native speaker and writer, there are times when I like to turn the subtitles on or off. There is indeed a button for that automatically appears in non-full screen mode if a valid subtitle file is present.

But .... once you switch to full screen, that button for turning the subtitles on or off is unfortunately no longer displayed.
To add those, and other buttons, you must have access to the shadow DOM.