The 18-year-old from Northern Ireland was hoping to become the youngest winner in European Tour history in only his third event as a professional.
McIlroy started the final round four behind Argentina’s Daniel Vancsik.
But he could not match the surge from Dane Vibe-Hastrup as five bogeys undermined an eagle and five birdies and he finished on 11 under par.
English trio Ben Barham, Sam Little and Sam Walker were seventh on eight under.
But it was another eye-catching display from McIlroy which provided one of the main highlights.
He only turned professional after the Walker Cup last month but finished 42nd on his tour debut in the British Masters and third in the Dunhill Links Championship last Sunday – a performance which all but secured his card for next season.
McIlroy can still become the youngest winner when he moves on to the Portugal Masters.
“I would have taken fourth when I came here. I think I’ve done very well after all that has gone on over the last few weeks and I hope to follow it on next week,” he said.
“I am playing very well at the moment, I am hitting it really well. If I could get a couple more putts to drop it would make the difference.”
Vibe-Hastrup, meanwhile, claimed his first European Tour title with a near-flawless final round, carding six birdies and just one bogey in his 67 to claim the £103,000 first prize and two-year tour exemption.
Spain’s Alejandro Canizares was three behind while Vancsik, who led for the first three rounds, was a further stroke adrift in third.
Vancsik’s challenge was badly hit on the fifth when he tangled with heavy rough to take a quintuple-bogey nine.
And Vibe-Hastrup, two behind him overnight, took a firm hold with three birdies in four holes from the eighth.
Canizares stayed in touch with him until he missed a short putt to bogey the short 17th, shortly after Vibe-Hastrup produced a fine approach to three feet to birdie the 16th and effectively end the contest.
