It's a bit complicated. But you can start with the Hill Sphere. A satellite would have to orbit inside the planet's Hill sphere , otherwise it would go into a separate orbit around the sun. It also has to be outside the planet's Roche limit or it would get torn to pieces by tidal forces from the planet.

Or - the moon can be closer than the Roche limit if the planet and its moon are both tidally locked to each other. But you can only have one moon like that, because in any other orbit the planet can't be tidally locked to the moon so the moon gets torn apart.

Then as well as that you have to consider drag forces on the planet from tidal forces. If the moon orbits the planet faster than its rotation period, it's going to be dragged inwards and will eventually hit the planet. That will be the fate of Phobos for instance, innermost moon of Pluto.

If it orbits the planet slower than its rotation period, it's safe from that, but will slowly spiral outwards. So long as the Hill sphere is reasonably large it can stay bound to the planet for a long time as happened to our Moon. And Mars's tiny moon Deimos is far enough away to spiral outwards. The further it spirals out, the less the tidal forces, so the slower the spiral. So that can last for billions of years. But in the opposite direction, the closer it spirals, the faster it spirals inwards, so generally it's not a very stable situation to orbit a planet faster than its rotation period.

That's why Venus doesn't have any moons. It spins so slowly, that any moon of any size would have hit it long ago. It could have tiny moons of a few meters across however as the moons spiral inwards more quickly the more massive they are, because the spiraling inwards is due to the tidal effects of the moon on the planet.

real-colour image of Venus taken by Mariner 10 processed from two filters.

Venus rotates retrograde spinning once every 243.025 days (relative to the stars). It orbits the sun once every 224.701 days. So its day is longer than its year. Is also retrograde.

It's impossible for a moon to orbit Venus as slowly as it spins. If it orbited that slowly it would be well outside its Hill sphere. So any moon of Venus would spiral inwards and eventually hit it - due to the tiny tides it raises in Venus. In detail - the tug of the moon raises a tide in Venus. But the moon gets ahead of the tide it just raised (because there's a slight delay in the planet responding to its gravitational tug), and so the tide tugs it towards Venus. Though the tide would be tiny for a distant small moon of Venus, still that's enough. Over millions of years it would spiral in faster and faster - one of those counterintuitive things that a tug backwards on a satellite causes it to go into a lower orbit and orbit more quickly - and it would soon hit the planet.

It would spiral in quickly if it is massive. A tiny moon of a few meters diameter would raise hardly any tide in Venus and could survive all the way to the present from the early solar system. There have been searches for tiny moons of Venus but none found yet - is still possible we might find a really really tiny one though.

Then as well as that you have the effects of the moons on each other. If you have several of them, then they will eventually fall into resonant orbits if close to the planet. So you can expect something like the 1 2 4 of the Jupiter planets for the resonances - so that means there isn't really any space for other moons of any size in between those because they wouldn't fit in nice resonances like that and would be distorted into other orbits, hit Jupiter or one of the other moons or be ejected.

But if you are further away from the planet, then there's plenty of space, so the moons have less gravitational tug on each other so can continue to orbit for a long time with no worry about them needing to fall into resonances, at least for billions of years.

And depends on the size of the moon. If they are really tiny ones of a few meters across then you can have thousands of them. And the tiniest ones aren't bothered by the Roche limit because they can hold together under the cohesive forces of rock, even ice, against the tidal disruption of the planet. You could think of Saturn's rings as consisting of numerous tiny satellites :). For such tiny ones, even Venus could in principle have many of them though it doesn't seem that it does.

I think those are the main things you'd need to look into.

So it depends most of all on the planet's mass. If the mass is large like Jupiter it has a huge hill sphere so plenty of space for extra moons.

Also depends on its rotation period. If it rotates reasonably quickly, then it is easy for its moons to orbit faster than the planet spins. But if it spins very slowly like Venus, then it is not likely to have moons of any size for long - maybe for a while in the first few million years of the solar system.

Then once you have those two conditions sorted, it's a matter of detailed modeling and looking to see what works by way of orbital resonances, and whether you can fit in any extra moons, for the close up moons. And this also depends a lot on the size of the moons and the planet, small moons around big planets are much easier than large moons around small planets.

If it's got a big hill sphere like Jupiter, you can put in lots of distant small moons no problem. If you have any really heavy moons it becomes a multi-body problem, e..g add a Jupiter sized moon to Jupiter or Earth sized to Earth -  they would probably quickly be tidally locked to each other, no problem. But add extra moons to this system and it becomes complicated. The Pluto system orbits are very complex due to the need to orbit a double planet, Pluto and Charon.