Yes, it can be a lot of work to create scenes whose only end use is an abstract feeling in the actual story, or - if you're lucky - a throw away line in a conversation somewhere. I think it's worth it, but then I write for much the same reason that I read, to immerse myself in another world, so I don't find it difficult or something I begrudge. You mention at the top of this page that you hate writing the first draft - and these external scenes are pretty much all first drafts - so that is going to colour your perspective.
You might also console yourself that you need not necessarily do it for every book. Do it well for a couple of books and you have assembled an entourage that can accompany you on future journeys. Witness Agatha Christie: not only did she get to keep Poirot (and Marple and Battle and so on) going for book after book, but most of her cast of suspects get pretty familiar too - with the names changed to protect the guilty. There are many more examples.
As for feeling that a plot is silly, most fiction can be viewed that way. I suspect it's why some people don't read much fiction: all they can see are the flaws (all fiction has flaws) so they can't let themselves go enough to enjoy the ride. And "the ride", for me, is what fiction (reading and writing) is all about. It's why I'm happy to enjoy kids books, romance, mysteries, sci-fi, fantasy ... pretty much anything, and why I re-read a lot of books; even when I already know the ending, if I enjoy the ride I don't care that there's no surprise. This reminds me of a line from a Harry Chapin song (
Greyhound): "It's got to be the going, not the getting there, that's good."
Speaking of Harry Chapin, he also wrote a song called
Mr Tanner. I sometimes think of this song when it comes to trying to publish. The song tells of a cleaner who sang while cleaning clothes, and his friends convince him to give a concert, which eventually leads to the lines:
But the concert was a blur to him, spatters of applause
He did not know how well he sang, he only heard the flaws
For all the enjoyment I find in writing, there is a certain amount of loss when it comes to making it ready for the public and you are forced to see the flaws. It's one reason I'm happy not to publish everything that I write.
(In case you can't tell, I'm supposed to be working - computer stuff - but I'm not in the mood. So getting carried away here seems like a much better use of my time.
)