I feel like stories tend to fall somewhere between "worldbuilding is a critical component of the story" and "the story could be moved to any world and it would work fine"
Its the difference between LOTR where the world is so critical to the story, that you could not take the story and put it in any other world without having to just re-creating a huge chunk of Tolkien's world all together, and the Princess Bride which I think you could put into the middle of almost any world and it would function nearly the same. And then, in the middle you have something like Hunger Games, which I think you could put the story of into a lot of worlds with pretty manageable adjustments.
For my own WIP I think the story would be easily moved to other worlds without much problem. It boils down to a young woman escaping her old life which is constricting her options in life, and a young man following her because he loves her and wants to help her find that freedom. While I do a lot of world building within the story, that's mostly worldbuilding that helps connect this story to other stories of mine instead of worldbuilding that's critical to the story itself.
I'd be interested in hearing about how much worldbuilding is needed in your story, and how well you think your underlying story would behave if it moved to a different world.
This all might also have to do with how well a story boils down in the first place. I can't seem to boil down LOTR very much before I get stuck on something, but others might be able to boil it down to something less attached to the world it's built in.