Nemesis (hypothetical star) was postulated to orbit at a distance of 1.5 light years about a third of the way to the nearest stars. At that distance, passing stars would sometimes pass between it and our sun. Scholz’s star passed within 0.4 light years of our sun 70,000 years ago. See A Star Passed Through the Solar System Just 70,000 Years Ago - Universe Today

Yet Nemesis was thought to be a stable orbit for a second star. It’s been pretty much ruled out by the WISE infrared survey as an orbit for a star but there could still be a very cold brown dwarf star at that distance - if so it would have to be one of the coldest brown dwarfs you can get, so not very likely.

So anyway - at that distance of course it would be vulnerable to being detached from our solar system if a star happened to pass close by. But that’s very unlikely, as unlikely as a star doing a close flyby of our own solar system.

So, I don’t think there is a firm limit. And the Oort cloud is thought to extend pretty much to the nearest stars. As stars pass each other then comets may get shifted from one to another or become rogue comets / planets that are not attached to any star. So we could even have comets out there that originally orbited other stars in the past.

Unless the planet is very heavy then its mass won’t make much difference - its trajectory will be the same whatever its mass unless it is able to deflect the passing star in some way - and I’m not sure whether that would be a plus or a minus for it to be massive enough to deflect passing stars significantly as far as stability is concerned.