It is important to understand that a molecule that contains an element is not the same as that element alone. The element might be deadly and the molecule harmless, or vice versa, the molecule deadly and the element harmless.
Pure mercury will mostly pass through your body unabsorbed.
Mercury in fish is absorable, and gathers in human nerve cells. A small amount can cause problems in brain development. A large amount can even cause problems in adults.
Thermisol contains mercury that is part of a chemical compound which is not absorbed, nor does it cause any harm to humans; there was only one study ever to suggest it does. The author pulled it and admitted to falsifying the study, he is also being investigated due to receiving large sums of money from a law firm which brought a lot of suits with his study as evidence. Any other study ever done (and many were done) have shown that Thermisol is harmless.
However, despite all that, the thermisol scare has resulted in it being removed from vaccines more then a decade ago, the vaccines given today do not contain any thermisol (and mercury).