I meet Sir Peter Blake in Mr Chow, the institution of a Chinese restaurant opposite One Hyde Park, where flats sell for £75m. Blake knew the restaurant’s eponymous owner in leaner times. He first met Michael Chow, he explains as we sit down, when the restaurateur was living on a camp bed in the garage of the painter Victor Pasmore by the Thames at Chiswick. Blake and his friend Richard Lin went to supper in the garage, a glorious meal that Chow cooked on a single paraffin burner on the dirt floor. Both Lin and Chow were artists and political refugees from Mao’s China, where Michael’s father had been a leading performer in the Peking Opera. Ten years after that meal, in 1967, Blake was a guest of honour at the opening of this restaurant; Chow later opened in Beverly Hills and New York.

To decorate the restaurant back then, Chow invited his artist friends to donate paintings for the walls in exchange for food. Blake was then at the height of his fame as a pop artist in the year of Sgt Pepper; among other things he made a portrait of Chow as part of his “wrestlers series” (Frisco and Lorenzo Wong and Wildman Mr Chow). The deal proved a good one on both sides. Blake’s paintings have only increased in value, and his tab has never run out. (The same cannot be said for another, nameless, artist. “He ate his share in two weeks,” Blake says. “He brought 20 people a night and drank the best champagne and that was that. With me, it has gone on for ever.”)

Blake is a gruffly animated 84. His life is intimately woven into British popular culture in the last six decades, and Mr Chow has been a backdrop for some of that. He used to bring the students he taught at the Royal College here in order for them to get to know each other. On one occasion, a lunch ended at five in the morning with them all plunging naked into the sea at Brighton. He brought a former student, Ian Dury, for lunch with Talking Heads, who were on their first British tour supporting the Ramones; Blake was acting as their minder. Then there was the time he snubbed Andy Warhol at an upstairs party.

“I had met Andy a few times,” Blake says, first in the early 1960s before Warhol’s long 15 minutes had begun, when they had met as collectors of fairground ephemera in New York. At the party at Mr Chow’s some years later, Blake recalls, Warhol had become very grand. “Michael comes over and says Andy would like you to come and talk to him,” he says. “But I didn’t fancy playing the courtier so I refused.”

We study the menu. One advantage of coming to the same restaurant for 50 years is that you know your way around what’s on offer – the Mr Chow formula has not changed in five decades. Blake normally comes here with his second wife, Chrissie. In her absence he plans to take advantage of having beef, which she doesn’t eat. He rattles off eight dishes to share – including some ribs. “A feast,” he says, satisfied. “But not a huge feast.” He’s not drinking; he gave up a couple of years ago. He’s got dodgy knees and is on medication, and he doesn’t want to be falling over.

Officially, Blake retired at 65, but he still works most days either at home or in his studio at Chiswick. The retirement was an act; really, he explains, he gave up on that part of the art world that was about envy and greed and feuding, and just kept going with the work. In his ninth decade, his life-loving eye for colour and design remains a very British pleasure – witness his 60s-inspired graphics for the cover of William Sitwell’s The Really Quite Good British Cookbook.

Three plates of starters arrive. Blake makes a beeline for his favourite, “squab with lettuce” and then sets to with the ribs. You don’t talk to him for long to realise that, over the course of his life, he has known pretty much everyone. Is there anyone he hasn’t met?

“It’s funny,” he says. “I’m a very reluctant name dropper but I was on a trip not long ago with the writer Mark Ellen, and we played that game.He kept coming up with names. He thought he had got me when he said ‘Gorbachev’. But in fact, my wife and I had dinner with Gorby the previous week, when he was a guest at Westminster Abbey …”

Talk of Gorby leads to talk of politics. Blake has never voted at any election. Why?

“It’s really because I have never heard a politician I liked very much,” he says. “To me all a country really has is its culture. The rest is all infrastructure. Lawyers and doctors and shopkeepers and so on are, in my view, necessary to back up the culture, the things we can create, the things that will last. Music and art and design and writing, the things we are good at.”

Blake developed this belief in his student years, when he started painting “kids with badges and all that at the Royal College” and he has stuck with it. He was schooled in music hall (“the only two clubs I am a member of are the Max Miller and Max Wall societies”) and Soho jazz clubs and by early rock and roll, and he retains a fascination with tribal identities. In the last few years he has been painting a series of people with tattoos (“it seems rare not to have one now – two of my three daughters do”). He is gratified by the return of beards.

His signature goatee came about, he says, after a cycling accident at 17 left him with a scar on his face. He has only shaved it off twice. Once for national service. And once in his 20s when he was travelling in Holland “and kids formed groups when they saw me in the street and chased me around making goat noises … which became a little disconcerting”.

Blake’s passion for cultural tribes found its most famous expression in his cover for Sgt Pepper. We are well into our shared main course of green prawns and crispy duck by the time we talk about some of the issues around it, which have been brought into focus by the album’s 50th anniversary.

It clearly remains a source of some bafflement and hurt for Blake that he was only ever paid £200 for the album and that no one at Apple Corps, the Beatles management company, has ever sought to rectify that apparent injustice, saying to him on a couple of occasions in subsequent years that they believed it remained a fair fee.

He doesn’t want to talk about it any more, he says, and only reluctantly goes through the details of how his agent Robert Fraser signed over his copyright for the artwork without reading the contract (and pocketed £1,500 himself). It’s not just about the money, though the tiniest of royalties would have been useful over the years, it’s more the attitude. Apple contacted him recently saying they wanted to recreate the cover for an anniversary event and asked to borrow any elements and cut-outs from it that he still had. No fee was offered. Blake politely declined.

He’s learned since his “retirement” not to be bitter, but always to celebrate, he says. He remains friends with Paul McCartney, who attended his 80th birthday party at the Albert Hall. Madness were the party band. It was a bit like a pop collage come to life. “There was a point where you could see Paul, Jimmy Page, Eric Clapton, Tony Bennett all together in frame,” he recalls, with a grin.

There’s a part of Blake that will always feel he has never been quite given his due, the Dartford-born son of an electrician who brought pop art to Britain, but he is amused by it as much as anything. He traces it back to his days at the Royal College where he arrived only with experience as a commercial artist.

He thinks because of that “they never took me seriously in the way that they took Lucian Freud or Frank Auerbach seriously”.

I suggest, before we go, that there is a prejudice toward earnestness, that we want our artists to suffer.

“Yes,” he says slightly glumly, “I was maybe too light. But, you know, Frank [Auerbach] has his light side too. He was a great Ken Dodd fan.” He pauses, “My problem,” he says, “was that I was neither a fine artist or a commercial artist. I was a sort of bastard …”

“You mean a hybrid?”

He laughs, “Yes, I’ll go with hybrid…”

The Really Quite Good British Cookbook (Nourish Books, £25)