Yes there is a formal protocol and legal basis. The legal basis is this article of the Outer Space Treaty
All space faring countries, and aspiring space farers, even N. Korea have signed and ratified this treaty (green in this map = signed and ratified, yellow = ratified only)
The article just talks about "harmful contamination" but that's been interpreted as including contamination that harms the experiments of other parties to the treaty.
So, if you introduce reproducing life to some other planet in the solar system, you are harming the experiments of other nations that may want to explore that planet and look at its native life - or whatever is there, even maybe its non life chemistry.
The details are worked out in a series of workshops by COSPAR. They decide what protocols are needed for which missions.
This is done on basis of categories. For instance the Sun, Mercury, Io and the smaller rocky asteroids are thought to be uninhabitable by Earth microbes, and also, to have no potential as places to learn about the chemical origins for life. So no precautions have to be taken at all.
In the case of the Moon, Venus and comets, it's one step up, category II, you have to document what you do, at present. But no other special precautions.
Then for category III, flyby, e.g. of Mars, or Europa, you have to do more involved protection, depending on the mission and whether there is some risk of impact on the body you are doing a flyby around.
Category IV is the category for locations that life could potentially inhabit. For this then you need to take stringent measures to protect them from Earth life. These also are subdivided into different sub categories for Mars.
Category V is a sample return. For sample return from a comet or meteorite or the Moon no special precautions are needed. For a a body that could have life, then very careful measures are needed. So far we haven't yet had a restricted Category V mission.
The enforcement is largely done by planetary protection officers. That's Cassie Conley for the US, NASA Office of Planetary Protection and we also have a planetary protection officer for Europe, Gerhard Kminek, from DLR Germany.
When anyone plans a mission, they get in touch with the Planetary Protection office and between them work out what needs to be done. They do this on the basis of the guidelines previously drawn up by COSPAR.
I'm not sure how it works in detail for Russia, China, India, Japan etc, but presumably they have a similar system. Any space faring country has these obligations under international law and they need to find a way to enforce it.
Also these obligations extend to all their citizens. For instance, Elon Musk is planning his own "space port" in the US to launch his own vehicles. But this doesn't of course free him from these international obligations. It is the duty of the US government to make sure all their citizens abide by them. No matter how they get into space. If you launch from a sea based platform or another country, if you are a US citizen, you still have this obligation (just as you have an obligation under quarantine regulations not to carry rabies, or some other disease of animals or plants or invasive creature to a country that doesn't have it, and it makes no difference to your obligation how you get there).
For details see Planetary protection
"Article IX: ... States Parties to the Treaty shall pursue studies of outer space, including the Moon and other celestial bodies, and conduct exploration of them so as to avoid their harmful contamination and also adverse changes in the environment of the Earth resulting from the introduction of extraterrestrial matter and, where necessary, shall adopt appropriate measures for this purpose...
All space faring countries, and aspiring space farers, even N. Korea have signed and ratified this treaty (green in this map = signed and ratified, yellow = ratified only)
The article just talks about "harmful contamination" but that's been interpreted as including contamination that harms the experiments of other parties to the treaty.
So, if you introduce reproducing life to some other planet in the solar system, you are harming the experiments of other nations that may want to explore that planet and look at its native life - or whatever is there, even maybe its non life chemistry.
The details are worked out in a series of workshops by COSPAR. They decide what protocols are needed for which missions.
This is done on basis of categories. For instance the Sun, Mercury, Io and the smaller rocky asteroids are thought to be uninhabitable by Earth microbes, and also, to have no potential as places to learn about the chemical origins for life. So no precautions have to be taken at all.
In the case of the Moon, Venus and comets, it's one step up, category II, you have to document what you do, at present. But no other special precautions.
Then for category III, flyby, e.g. of Mars, or Europa, you have to do more involved protection, depending on the mission and whether there is some risk of impact on the body you are doing a flyby around.
Category IV is the category for locations that life could potentially inhabit. For this then you need to take stringent measures to protect them from Earth life. These also are subdivided into different sub categories for Mars.
Category V is a sample return. For sample return from a comet or meteorite or the Moon no special precautions are needed. For a a body that could have life, then very careful measures are needed. So far we haven't yet had a restricted Category V mission.
The enforcement is largely done by planetary protection officers. That's Cassie Conley for the US, NASA Office of Planetary Protection and we also have a planetary protection officer for Europe, Gerhard Kminek, from DLR Germany.
When anyone plans a mission, they get in touch with the Planetary Protection office and between them work out what needs to be done. They do this on the basis of the guidelines previously drawn up by COSPAR.
I'm not sure how it works in detail for Russia, China, India, Japan etc, but presumably they have a similar system. Any space faring country has these obligations under international law and they need to find a way to enforce it.
Also these obligations extend to all their citizens. For instance, Elon Musk is planning his own "space port" in the US to launch his own vehicles. But this doesn't of course free him from these international obligations. It is the duty of the US government to make sure all their citizens abide by them. No matter how they get into space. If you launch from a sea based platform or another country, if you are a US citizen, you still have this obligation (just as you have an obligation under quarantine regulations not to carry rabies, or some other disease of animals or plants or invasive creature to a country that doesn't have it, and it makes no difference to your obligation how you get there).
For details see Planetary protection
About the Author
Robert Walker
Writer of articles on Mars and Space issues - Software Developer of Tune Smithy, Bounce Metronome etc.Studied at Wolfson College, Oxford
Lives in Isle of Mull
4.8m answer views110.4k this month
Top Writer2017, 2016, and 2015
Published WriterHuffPost, Slate, and 4 moreHuffPost, Slate, Forbes, Newsweek, Business Insider, and Science 2.0