Selfish Mining in short is theoretical concept in which a malicious pool of miners could gain a better income by deliberately forking the blockchain. If a mining pool were to not immediately broadcast blocks, but instead add them to their own private chain, when the private chain becomes longer than the current chain; the malicious pool can publish their longer chain directly after a legitimate pool finds a block, causing the network to orphan the legitimate pool’s block and give the reward to the malicious pool. By operating in such a way: every time the malicious pool orphans blocks mined by legitimate pools, the effort of the legitimate miners is wasted and they get no reward. In theory this could cause legitimate miners to jump ship into the malicious pool, giving the malicious pool more power, thus giving them a better ability to orphan other pools blocks. Eventually the malicious pool will be able to combine orphaning other pool’s blocks + hashing power, in order to execute the 51% attack
with less than 51% of the bitcoin network’s hashing power. Full explanation here
As most people know, bitcoin has no intrinsic value and is entirely based on supply an demand. Traders make up the majority of bitcoin holder and as seen in the April 2013 crash, they are willing to liquidate their coins at the first sight of bad news. As soon as the selfish mining attack was noticed (easily noticed by the same pool repeatedly orphaning other pool’s blocks), traders would begin to liquidate their coins, causing the value to drop. For the same reason the selfish mining attack could work (greed) other traders would see the value drop and also dash to cash out. Before long there’s a gigantic snowball of people liquidating their coins to the point where bitcoin is nearly worthless.
Most miners are also bitcoin investors. Due to the ever rising price per a coin, miners prefer to keep their coins as they increase in value with age. In the long run it is far more profitable to mine normally and keep hold of coins, than to mine maliciously and risk diminishing their value. Although this does not prevent the selfish mining attack, it makes it far less likely for non malicious miners to join a malicious pool, thus it would likely require the selfish pool to have a much larger number of malicious miners, before they could cause enough losses to bring in legitimate miners. It’s also likely the malicious pool would be identified and miners warned not to join it, I would also guess it would lead to distributed denial of service attacks against the malicious pool.
I don’t doubt that a pool could selfishly mine, however, I do doubt it would ever grow large enough to cause noticeable problems for other miners. I’m also sure the pool would never get big enough to preform any of the attacks outlined in the 51% attack explanation. These are some of my theoretical problems with this theoretical attack, though i do think the attack is an incredibly cool concept, I don’t believe it could be as damaging as stated.