Watch: Liverpool 2-1 Leicester in 90 seconds
The Reds took the lead five minutes before half-time through Sadio Mane’s 50th top-flight goal for the club, but James Maddison’s 80th-minute equaliser appeared to have snatched a draw for the visitors.
However, Milner kept his cool from 12 yards after Marc Albrighton had felled Mane in the box deep into injury time to ensure Jürgen Klopp’s team head into the international break with eight wins from eight Premier League outings this term.
Team news
Dejan Lovren came in for his first Premier League start of the season as one of two changes made by Klopp.
Milner started in place of Jordan Henderson, who dropped down to the bench along with Joe Gomez, while Harvey Elliott was also named among Liverpool’s substitutes.
First half
A strategic tweak by Klopp saw Mohamed Salah start the game at centre-forward, with Mane on the right and Roberto Firmino on the left of Liverpool’s front three.
And the No.11 had the game’s first sight of goal just four minutes in, rolling Caglar Soyuncu and attempting to steer a low shot into the far corner that was saved by Kasper Schmeichel.
Brilliance from Trent Alexander-Arnold created a more presentable chance for the Reds in the 13th minute.
The right-back nutmegged Ben Chilwell on the byline before standing up an inviting cross for an unmarked Milner at the back post, only for the captain to miscue his half-volley into the Kop.
Leicester enjoyed their best spell of the half thereafter, although the visitors could not convert improved spells of possession into goalscoring opportunities and Liverpool remained the more threatening team.
Another fine delivery from Alexander-Arnold found a stretching Firmino, but the No.9’s volley cleared the crossbar. However, five minutes prior to the break, the Reds had their lead.
A bold and patient sequence of passing in the left-back area ended with Milner injecting verticality into the move with a superb left-footed ball down the line.
Mane, temporarily back in his usual position down Liverpool’s left, surged clear of Jonny Evans and casually slotted home his 50th Premier League goal for the club on the occasion of his 100th top-flight appearance.
The Senegalese could have added to that impressive tally quickly too, but his connection on Firmino’s cut-back was not clean enough to beat Schmeichel as the hosts ended the half strongly.
Second half
Schmeichel was forced into action inside the opening minute of the second half, the Leicester goalkeeper parrying away Salah’s shot on the slide from Alexander-Arnold’s cross.
A double one-two between Salah and Mane then appeared to have afforded the latter the task of scoring from close range, but Evans’ last-ditch tackle denied him.
Liverpool’s search for a second was gathering momentum and Firmino’s wonderful first touch bought him space to shoot after collecting Alexander-Arnold’s punched pass forward, though the No.9 dragged his attempt wide.
The Foxes, however, kept themselves in the game and Liverpool needed Adrian’s alertness, the ‘keeper gathering at Jamie Vardy’s feet as the striker bore down on goal, to keep them in front midway through the half.
Dennis Praet drove an effort wide from 20 yards after Andy Robertson had blasted straight at Schmeichel at the other end and, 10 minutes from time, Leicester drew level.
Ayoze Perez slipped a through-ball into Maddison’s path and the midfielder’s low shot beat Adrian.
There was to be late, late drama, though, as Milner sent Schmeichel the wrong way from the spot to spark delirious scenes of celebration inside Anfield.