It's no surprise that John Boehner got re-elected Speaker of the House, and with only nine GOP defections (and three "presents" and one not voting).
Per the 1998 push against Gingrich, which had its roots in a summer 1997 coup attempt, I think Cantor could have had the position, had he wanted to fight for it.
That said, Boehner's inability to manage his party in general aside, he probably could have kept a majority of Republicans for two or three ballots. The question ultimately would have been what Democrats chose to do.
And, I'm sure Nancy Pelosi would have told her troops to keep voting for her until Boehner came calling with a deal.
Boehner would have conceded to Cantor before doing that, though.
But, Cantor doesn't want the hassles of being in the tea party gunsights or taking its arrows in his backside.
So, no coup was ever going to happen.
Now, could he try one after debt ceiling negotiations? Push Boehner to then step aside, if he thinks that's "needed"? I'd still offer 3-1 odds against that, unless Boehner publicly kisses Obama on the House floor.