Historically, I think mainly because they turned up so many problems staying for any length of time in LEO. When the Russians were slowly pushing records, with Valeri Polyakov finally setting the record of 437 days in 1995 - a record that still holds to this day - but many health problems in zero g - body slowly deteriorating - that's not too encouraging for a long space flight that would last several years.
When they first launched humans into space, at first they didn't know if a human body could withstand even an hour in zero g. That's why they had those early spaceflights with monkeys.
If things had turned out differently, if the human body just hadn't been able to handle an hour in zero g easily - it might have taken a lot longer to get to the Moon.
As it turned out, we can handle several days in zero g fairly easily (with some changes in the way the body behave).
But months and years were another matter and needed careful study and care. So Moon wasn't too hard for humans, Mars though was much much more of a risk and quite possibly lethal.
There's also the communications delay issue. With the Moon was only a couple of seconds, easy to have conversations with ground control. Even on the Moon - Houston was in direct communication all the time and they asked each other questions and had continuous conversations between the astronauts and ground control.
At distances of Mars you'd be very much on your own with up to 40 minutes delay time between when you first start to report your problem and when you first start to hear responses from Earth. With also a lot more potential for confusion and error in a crisis.
So that's surely a factor also.
Fuel is not so much of an issue, I think, as Apollo 11 had about enough fuel to get a lunar module to Mars orbit (Mars capture orbit) and back - though of course astronauts in a lunar module would be long dead before they could travel the six months to Mars and back. Still it's not the main issue.
Closed system habitats are an issue. Our space stations need continual supply and when the only way you can keep someone in space for more than a few months is to resupply them from Earth, then again - a multi- year journey to Mars is kind of hard to imagine succeeding.
If you wanted to land on Mars - that's another issue, it's the hardest place probably to land in the inner solar system. Almost vacuum of an atmosphere - so can't use conventional parachutes all the way to the surface - yet - can't just "hover" and can't go back into orbit because you get captured by the atmosphere.
So, landing on the Mars surfaces is technically harder than Venus, Mercury, the Moon, and of course Earth, and any of the asteroids or comets the moons of Mars (where I'm assuming you have enough delta v to get to the places and just talking about the difficulty of the final descent).
So you get these complicated descents, all the missions so far anyway - first aerobraking, then parachutes, then cut parachute and fire a rocket, or you use bouncing ballooons, and then hit the surface.
Many things have to go right in a few seconds, and all automated, so plenty of room for errors, and half of the missions to the Mars surface have crashed or failed soon after landing.
As it happens I'm glad we didn't do it back then.
Because - back then they thought Mars was totally sterile. Now we know that it was at least habitable for life in the past, and may quite possibly (perhaps 50/50 who knows) still have life there today.
Microbial life only, but still would be amazingly interesting if it is based on a different biochemistry.
All the myriad of lifeforms we have today on Earth are based on DNA - but not only that - entire mechanism it uses to transcribe the DNA into proteins, error correction, cell wall, metabolism, it's a huge complex structure that probably took as many evolutionary steps to develop as it took to get from the last common ancestor of all Earth life to ourselves.
So - on Mars we might have the chance to find life that took a different direction completely - either early on not even DNA or broke off before the last common ancestor, or broke off more recently but evolved new traits in the extreme Mars environment.
Whatever it is - if there is present day life there especially, I think is just as well we didn't send humans there with 1970s technology. And we don't really have the technology quite yet to send humans to the surface either without introducing Earth life. Quite possibly never will.
So need to step slowly and lightly in this case.
Just for Mars, Europa and Encladus particularly, some other places in the solar system also of some importance to keep clear of Earth microbes for now, but those are generally recognized as the top three for Planetary Potection.