Do long term investors have to diversify when investing in stocks?

I understand don't put all your eggs in one basket, and always diversify throughout different investments and sectors when holding for long term. But i was watching this Buffett Interview with Evan Davis. And he was talking about you don't have to diversify, if you own 50 stocks, would you understand the 50th stock better than the 1st one? The common thing to do is to diversify and be safe.