Originally Answered: Why are all planets round and not square or rectangular?
Anyway first to explain why most planets are round - to first approximation they are liquid, so imagine you had a huge mountain of water in the middle of an ocean. It would spread out of course, to form an "equipotential" surface.
So for a planet that doesn't spin at all, any irregularities in the shape of a planet spread out like that until you get a spherical planet.
As they rotate faster and faster, planets become flattened at their poles. The Earth is slightly flattened in this way and as a result, the equator is further away from the center.
Because of this, the point furthest away from the Earth's center is not Mount Everest, which is quite a bit North of the equator. It is a volcano called Chimborazo in Equador
In Hal Clement's Mission of Gravity he explores the idea of a planet spinning very rapidly so it looks like this:
However since then we've found rapidly rotating dwarf planets in our solar system. And the ones found so far are a bit different, they are triaxial spheroids.
If a planet rotates very quickly, it can look this - this is an artist's impression of a dwarf planet in the Kuiper belt:
Here is an artist's rendition of it rotating showing its distinctive red spot:
Haumea: Rugby Ball Planet
Another example
They may look similar to Hal Clemens planet, but are actually a little different. These are triaxial ellipsoids
They rotate around their minor axis, the shortest of the three axes.
It turns out that there are two possible solutions, as the spin rate increases. You can get an oblate spheroid, or a triaxial ellipsoid - the solution "bifurcates". But the triaxial ellipsoid is the most stable of the two as Jacobi found out in his paper published in 1834 Figures of Equilibrium - Historical Account* - Chandrasekar
Now if that seems bizarre, it's not the most unusual planet shape possible.
Planets can actually be triangular, or square, or even pentagonal in shape in theory. And in principle they can also be donut shaped too, with a hole in the middle - a stable configuration though one that is easily disturbed so there probably aren't that many donut shaped planets in the universe :).
For that to happen they have to be spinning so fast they are on the verge of breaking apart.
There are cool videos of triangular and square droplets in simulated zero g here.
Here the droplets are held together by surface tension. But apparently theoretically droplets held together by surface tension in zero g have the same shape as a planet or star held together by gravity. So this is experimental work that also confirms theoretical predictions for possible shapes of planets too.
So these are also possible shapes for rapidly rotating planets. PPaper here: Nonaxisymmetric Shapes of a Magnetically Levitated and Spinning Water Droplet
Then as it spins faster, it can turn into two spheres joined together. This is the scenario of a binary planet.
GRAVITATIONALLY LOCKED BINARY PLANET
If two planets are gravitationally locked to each other, in the same way as Pluto and Charon, then tidal effects no longer matter.
They are just permanently distorted. You could in principle even have an Earth sized planet almost touching our Earth - even with a shared atmosphere and ocean.
This is explored fictionally in Robert Forward's "Rocheworld" which is an "overcontact binary"
Though we don't know any Rocheworld planets yet, there are many contact binary asteroids. These are too small to be rounded under their own gravity, but are approximately the shape of an overcontact binary.
This is 216 Kleopatra
And comets also, this is 67P/Churyumov–Gerasimenko which is also approximately the shape of an overcontact binary
Also many contact binary stars are known, again including "overcontact binaries" also like these asteroids and comets, so close together that their atmospheres overlap - the first discovered W Ursae Majoris.
As it spins even faster then in theory anyway, a planet or star can turn into three or even four spheres joined together to make an "overcontact ternary" or "overcontact quarternary" planet or star, or you could have a donut shaped planet too, though so far we haven't found anything like this:
This is an animation someone did of the last possibility the "hoopworld"
But thin hoops like this are unstable and easily break up into droplets.
Thicker hoops like this visualization from io9 though can be stable.
I shouldn't think there are many of these in our universe :). Needs very special conditions to form. But - not impossible apparently.
Here is a youtube video of a Moon of a donuut planet:
You can also get donut shaped solutions for end state of a collapsing gas cloud, which are stable, in principle. So, you could have young toroidal stars as the author suggests in this paper : General Relativisitic Sturcture of Star Toroidal Systems (1992).
Anders Sandberg's fun article explores many aspects of life on a donut shaped planet - and the orbits of satellites and moons around such a planet.
What would the Earth be like if it was the shape of a donut?
TIDALLY SQUISHED PLANETS
This is another way that planets may vary in shape from the usual round or oblate spheroid.
Again we don't know of any planets like this yet. But we do have moons. They are too small to be in gravitational stasis, but the innermost moons of Jupiter are seriously tidally squished as you can see from these photographs of them taken by Galileo:
Here are more images of the largest of these moons, Amalthea
In a recent paper, last year stretched-out solid exoplanets (the paper itself with the mathematical details is here: The observational effects and signatures of tidally distorted solid exoplanets) the authors suggest we may soon discover ellipsoidal habitable rocky planets tidally locked to a red dwarf star, and these may be in the habitable zone.
Red dwarf stars are much smaller and cooler than our sun. So planets have to "huddle" close to the star to keep warm. The habitable zone is so close to the star that a planet like Earth orbiting a red dwarf star would be tidally locked, and these authors suggest, also ellipsoidal in shape. They could be squished out towards its sun, by as much as 3 : 2.
I've just posted this answer to my Science20 blog here: After Rugby Ball Shaped Planets, What Next? The Numerous Possible Shapes Of A Rapidly Spinning Planet Or Star
About the Author
Robert Walker
Writer of articles on Mars and Space issues - Software Developer of Tune Smithy, Bounce Metronome etc.Studied at Wolfson College, Oxford
Lives in Isle of Mull
4.8m answer views110.4k this month
Top Writer2017, 2016, and 2015
Published WriterHuffPost, Slate, and 4 moreHuffPost, Slate, Forbes, Newsweek, Business Insider, and Science 2.0