I'm not sure wormholes are the right technology for this - as you have to create the wormhole in the first place. Maybe you are thinking in terms of space warp instead?
If so - well 4 hours is way way faster than even that - which is a far future idea at present.
But - supposing it is possible - say as a science fiction scenario.
Then what matters is how long it takes to make a new spaceship when you get there. Visiting all 100 billion stars in the galaxy one after another, every four hours, with no time to stop at each one - that's 45 million years.
But - if each time your spaceship visits a new star it then makes new spaceships - with this advanced technology that's not likely to take long, say another 4 hours to make a new spaceship when you get there from raw materials using nanoscale 3d printers etc - and say you make ten of them simultaneously - and have some way of coordinating this vast fleet so you never send two spaceships to the same star - then you can send spaceships to all of the 100 billion stars in our galaxy in four days.
That's the basis of the Fermi paradox - except - that you don't need to make it nearly as fast as that. Make the time to get to a nearby star and then create ten copies of your spaceship a million years even - and you can then explore the entire galaxy in a few million years. Even if it takes ten million years, you are still talking about just hundreds of millions of years to send a ship - or a seed colony (with incubators to bring the seeds and embryos to maturity) - or whatever it is you want to send - to every single star in the galaxy.
With technology not far advanced over what we have now - an ET that evolved as recently as a hundred million years ago could easily have explored the entire galaxy and colonized it - or done whatever they wanted with it.
So why haven't they done that?
Many ideas and solutions for this.
My own best solution to this is - that it's a risky thing to do to colonize the galaxy. Because of those trillions of descendants of ours - then some may well make some mistake - e.g. the proverbial "paper clip event horizon" - a paper clip making self reproducing machine that turns entire galaxy into paper clips - or else - create artificial life forms - or create alternative biology life forms - or just change of culture so they want to do things to the galaxy that we find unacceptable. And with that level of power - it is feasible to reshape the entire galaxy - once you have self reproducing machines + exponential growth.
So - to set out to fill the galaxy with beings each individually with the power (through self replicating machines or artificial intelligent biological lifeforms or cyborgs) to unleash technology that could reshape the entire galaxy to their desires - or by mistake to reshape the galaxy into something they don't want - that doesn't seem an especially wise thing to do. Surely we'll have evolved beyond such wishes by then - or else - will have destroyed ourselves already.
So - maybe all sensible ETs make the decision not to colonize the galaxy - they can explore it - even send self reproducing probes one per star (say) to survey the galaxy - but not set out to colonize it. Instead just limited colonization whatever is needed for survival of their species - and exploration of the galaxy.
If all sensible ETs make that decision - and all reckless ETs destroy themselves - that might by itself explain the paradox.
That's just one idea, but there are many solutions offered to the Fermi paradox.
For that reason - I look forward to exploration of our solar system and galaxy. But before colonization - would hope that we have thought through these issues and found a way forward that won't unleash galaxy transforming colonization that could harm us and other ETs in the galaxy. If there are other ETs already there - then the one thing that might get them concerned about us is if we show signs of wanting to colonize the galaxy ourselves in an uncontrolled way like that without thinking through the repercussions and taking steps to make sure it is beneficial and not harmful to us and to other ETs.
If such ETs do exist at a technological level not that much advanced over our own - then they probably have mechanisms to prevent that happening (e.g. "guardian self replicating machines" antibodies around every star to prevent infection of the galaxy) - which doubtless would only need to operate on centuries and millennial timescales so would be a while before we notice anything.
Or perhaps that's not necessary - maybe the natural mechanism is all that's needed - that any sensible ET self regulates - and reckless ones destroy themselves. If far seeing enough to not destroy ourselves - then we are far seeing enough to think through the long term consequences of interstellar colonization before we do it.
If they are not there and we are the first ET with this technological capability - then there are no guarantees that anyone else will step in and prevent harm - or that we will automatically do so ourselves.
If so, we need to take especial care to think through possible consequences of colonizing the galaxy rather than just set out and do it because we can.