Carson does appear to have a problem with being factually accurate though, he's ill informed on a great many subject the voters don't particularly care about. The question is will he be ill informed on something that matters.
Our current moron in chief suffers from that particular problem in a particularly bad way, and the result is that we're working hard to overthrow petty dictators with moderate views, and put Islamic radicals in their place, and being nicer to people that want to kill us than we are to to our neighbors to the north, who we just shafted for supposed environmental concerns that don't exist. I may oppose his ideology in every way possible, but it's his poor grasp of reality that is in danger of resulting in WW3 a decade or two from now, not his socialist views on the economy.
If Carson can't get his shit straight, I consider it a valid problem. Which is a problem, because he's one of the three that have a prayer that I don't think would be worse than ending up with Hillary for 4 years so she can give us a super majority in both houses before losing her next election. Politifact may be a questionable rag that seems to do a lot more negatives on the conservatives while loading progressives up with positives(Hillary's profile is particularly egregious), but it says something when he's got mostly negatives even compared to other conservatives. Cruz has the same problem, nice views on issues, poor grasp of the facts surrounding them.
Accuracy requires care, researching further than the headline on whatever study happens to support your views. Carson doesn't appear to do this rather frequently both on extraneous and contemporary issues, so the big question will be does he treat his tasks as president like he did learning neurosurgery, or like he has studying Egyptology? A hard working genius he may be, but he opens his mouth when he knoweth not, frequently on political matters as well.