I've waited several months for books where there only 4 people ahead of me on the list. I imagine they each kept it the full 3 weeks. It's not unusual at all and these are older books from the 40s and 50s. These older books only have 1-4 copies max. Newer bestsellers can have as many as 150 copies though, and I imagine those wait lists go quicker.
Knowing about that waiting period makes me always return the book immediately when I'm done, last book I had out only 2 days... it was over the weekend and I read straight through it. I think a lot of people wait until the period has expired and just let it drop off their reader to return it instead of making the effort to return finished books early.