Beyond the DNA evidence that points to him as the most likely of the Jefferson-Wayles family to have fathered the children in question, other, circumstantial evidence points the same way.
Under that link, Wikipedia lists the most notable circumstantial evidence:
Jefferson freed all of Hemings' children: Beverly, Harriet, Madison, and Eston, as they came of age (they were the only slave family freed by Jefferson). They were seven-eighths European in ancestry, and three of the four entered white society as adults. Their descendants identified as white. Hemings was "given her time", and lived her last nine years freely with her two younger sons in Charlottesville, and saw a grandchild born in the house her sons owned.
Those are exactly my starting points. Had he freed other slaves, we might be talking differently. But, Hemings, and these particular children, were "special" for some reason.
Jefferson's sexual relationship is the most obvious answer, and the most simple one. Under the Jefferson-Hemings controversy, there's more.
So why did most white historians not want to believe this?
Maybe they thought it made Jefferson himself look too "low" or "base."
Maybe they didn't want to credit Hemings with negotiating freedom for future children when with him in Paris before they came back to America.
Maybe they, while acknowledging even more forcible sexual relationships between masters and slaves, didn't want to acknowledge one that involved emotions — but arguably was still forcible in some way, even if negotiated force.