Turned out I was wrong.
The two books I was testing had embedded fonts. Quite a bad luck as these were the only ones in my library according to quality check.
I removed the fonts and it works now as expected.
Strangely enough, my touch works fine with the embedded fonts on these books...
Anyway, thanks for your help !