I haven't read Silas Marner. I can see I'm going to have to read George Eliot's other books! The only other one I have read was The Mill on the Floss as a set text when I was doing English at University about a hundred years ago, and I don't remember much about it except the sad ending.
Fred grew on me as the story unfolded, I suppose as he started to take some responsibility for his actions - I was appalled by him to start with. But then he too was a product of his indulgent family, and he was decent, as you say, fantasyfan. And he had the sense to recognise Mary's goodness and strength, rather than taking off after a pretty face because she was severe with him.