Nah, no worries - I'll probably just try the even smaller (and weaker) magnets
The ones I've got aren't strong enough for anything to actually "stick" through the cover and I can't even feel any pull when moving a small metal piece over the front of the cover but I am still able to operate the wake/sleep switch through the whole thickness of the leather.