I also live in Sweden (like Tompe) and my visa is also a debit card with no credit tied to it. Some banks here in Sweden will charge you an annual fee for your account, visa and intenet-bank, but my bank don`t.
They will however charge you a fee if you withdraw money in another currency or if you use your visa on your out of country vacation (this some people mean just shows that we need to join the Euro-club).
Some shops and resturants do charge you if you pay very small sums (less than 10 euro) although they are really not allowed to under contracts with visa. Those shops always place the blame on the banks that charge them, although they are also not allowed to do so...
