World building matters even in works where it's not the main focus
...in so much as if it's not done well it risks ruining whatever immersion was holding the story up.
In some sense this is a skill issue more for the audience (me) than it is the authors. But it's hard to climb back on board a story when you encounter something where a detail pulls you out, whether by feeling contrived or just not sitting right with what the story has established up until this point. Usually the author wants something to happen and breaks the rules or comes up with some excuse for why it must happen, and it's frustrating if you have an idiot brain like me which goes "no you're not allowed to have fun here!"
With that said here is a quick view of where I think this comes into play as well as some approaches that I think work even if they lead to pretty bullshit places.
NO FUN: Authors changing their mind
Star wars by virtue of a lot of different authors, a ton of quasi canon/legends material is full of this shit, but the biggest one that I will not accept is Darth Maul.
Darth Maul... no I don't care about whatever story about him being too mad to die, we saw him get split in half, do not tell me he survived that shit. I don't care how good his bit in Rebels is or however good the new show is, dude is dead in my mind. And it's frustrating because the dude was just a cool design in one movie with no depth, just make him have a brother/twin!
We see some pretty wild stuff in the series (extended universe) but if you are just watching the main movies I feel like you should be forgiven for thinking that someone getting split the fuck in half is dead-dead.
NO FUN: Convenient-but abstract-excuses to drive the plot forward
Frieren is a beloved series that I absolutely don't give a shit about.
My partner loves it so I've been watching it, and while I loved the initial premise, the mage exam started to derail things for me. The rationale for the arc is contrived. "We need a first class mage to go to the northern plateau," no you don't! We do not need a certification arc in any story unless that certificate is integral to the themes and plot itself.
Given what we know about the characters so far it would just be as reasonable for Frieren to tell Fern "oh they are doing a mage exam in this town, and I hear you get a free grimoire if you pass, let's do it!" The authors control the story, make the reason for something salient to the plot.
I'LL ALLOW IT: Building off of established rules
I think in the animanga realm there are a few series that do a pretty decent job of laying a foundation and then following things logically from that foundation.
Witch Hat Atelier: Magic is drawing circles and people develop their skill by creative application of different elements of the circle. Welcome back Magical Circle Guru Guru!
Full Metal Alchemist: Does a good job of laying groundwork and logically building to a conclusion.
Dungeon Meshi: The first half of the story slowly trickles information about the ecosystem and world, the second half of the series cashes all of that in culminating in a some crazy stuff, but I buy it because it's done such a good job of building things in line with it.
JJK: Man as much as this series is "fine" or whatever, somehow by overwhelming the reader with lore about the mechanics of how each individual cursed technique works, you just sort of give in and go along with it, until you get completely to broken shit Hakari. I don't like it but i'll allow it.
I'LL ALLOW IT: Nobody knows what the fuck is happening
There are a lot of stories where the breadth of what is going on in the world is hidden from the audience. Things are left up to interpretation. Nothing is really spelled out and that makes it a lot easier to embrace what semblance of logic guiding the world exists. That isn't to say there isn't a canonical explanation for things, but that the story gives enough leeway for you to make it make sense to yourself, to fill in the gaps
Angel Egg, my beloved.
Twin Peaks, my beloved.
Tsutomu Nihei shit... Abara, Biomega: Story starting 15 minutes before a world ending catastrophe means you don't have time to process why everything is falling apart.
Blame!: Story starting thousands of years after a world ending catastrophe means you just gotta accept that shits freaky.
I'LL ALLOW IT: Episodic storytelling
Star Trek (original and TNG), X-files, etc. most episodes are stand alone, there may be some contuing themes etc. but by the premise you sort of assess each episode by itself. Some episodes are goofy (mirror universe shit in Star Trek) but it doesn't matter because the next episode things are back to "normal".
What are people's worldbuilding gripes? What caused you to bounce off a "good" series or something that prevented you from suspending your disbelief?