Will I be having lunch with Brian Sewell or will he be having me for lunch instead? It's a necessary question. At 82 years old, the eminent art critic is famed for his waspish asides and biting tongue. His acerbic reviews of gallery shows have been known to reduce curators to tears and his distaste for the contemporary art world is well-known. His weekly reviews in the Evening Standard have made him the master of the vituperative barb. In the past, Sewell's targets have included the street artist Banksy ("should have been put down at birth"), Tony Blair ("a man of extraordinary affectation") and female artists in general ("there has never been a first-rank woman artist").

So I'm nervous waiting for him to arrive at Bibendum Oyster Bar in South Kensington. Our table has been booked for midday. The clock ticks on to quarter past, then half past. I wait. I read my notes. I eat an entire plate of parmesan biscuits. After an hour, I begin to realise I have been stood up. Confirmation eventually comes via a publicist over email (Sewell doesn't own a mobile phone). He's having to file last-minute copy and can't make it.

We reschedule for a few weeks later. Same place, same time, same parmesan biscuits. Happily, at the allotted hour, Sewell shakily hops through the door on crutches, disdaining offers of help. The crutches are a consequence of a chronic back condition that leaves him in near-constant pain and makes sleeping difficult. A solicitous waiter pulls out a chair and is shooed away.

"I'm not used to help," he says. "At home I have to get by on my own."

He is mortified that he forgot our initial appointment and launches into a complicated apology that involves telling me about his "fail‑safe" diary system. This consists of having a diary but not actually writing in it, instead relying on typed sheets put in plastic wallets and slotted into the relevant week, which doesn't sound all that fail-safe to me, but I'm not about to disagree.

He is here to promote a DVD he has made about portraiture.

"Am I?" he says, shocked.

Yes, I say, I think you are. We talk about the portrait of the Duchess of Cambridge by Paul Emsley, which Sewell describes with typical aplomb as "a greasy photograph". So who are the best portrait artists, in his opinion?

"If you want a really good Ordnance Survey [of a person] go to Raphael but if you want something that really speaks to you, go to Titian."

We order a dozen oysters after deciding it would be rude not to. I'm not particularly concerned about the aphrodisiac side-effects: Sewell's first volume of his memoirs, Outsider, recounted his claim to have slept with 1,000 men. Outsider became a critical and commercial success. The second volume, published last year, begins in 1967 just after Sewell left his job at Christie's auction house and detailed both his friendship with the former Soviet spy Anthony Blunt and his evolution into an art critic. Again, the reviews were warm. Was he pleased?

"I was surprised," he says. "I'm not sure that pleasure came into it."

Did he hear from any of his conquests after the book was published? Sewell emits a high-pitched giggle. "No! I think they're probably all dead by now."

He digresses into an entertaining anecdote about Francis Bacon whom he used to meet in the mornings at the juice bar in Harrods. "There was a good greengrocer's near Francis's mews house and he would quite often turn up at Harrods with a brown paper bag. I remember one day he brought a kohlrabi, which he'd never seen before. And he'd just hand over the things and say 'Juice that, ducky'. So they were quite amusing mornings because sometimes the mixture was absolutely foul."

Bacon was a good gossip – "He was quite revealing about people, quite mean" – and confided to Sewell that he "loathed" the art critic David Sylvester with whom he did a famous series of interviews for the BBC.

Sewell's crab arrives in its dusty pink shell and is pronounced "too beautiful to eat" but eaten nonetheless. Despite his regal enunciation, Sewell remembers his childhood as "poor". He was raised by a single mother in Kensington until the age of 10 when she married Sewell's stepfather. His earliest food memories consist of "tinned sardines on toast" and "going to the fishmonger and being able to buy vast quantities of roe for virtually nothing."

For decades, Sewell didn't know who his real father was. When he came out to his mother in his 30s "a look of total, absolute scorn came over her face and she said: 'Just like your father' and walked out of the room leaving me completely mystified. I thought 'My God, what does this mean?'"

He eventually discovered his father was a sexually voracious minor composer who went by the name of Peter Warlock and who committed suicide in 1930. Was it difficult to be open about his sexuality, I wonder?

"There wasn't really any alternative," Sewell says, quietly. "There is a point at which you have to reconcile yourself to things and get on with life. It had reached a point where I couldn't go on dissembling."

It is, I suspect, this fear of hiding secrets that makes Sewell so honest in his opinions. During the course of our lunch, he calls Tracey Emin's sketches "trivial" and Grayson Perry's pottery "vulgar… Most of his pottery could be intriguingly subversive if converted into table lamps."

Does he care about hurting people's feelings? "No, I can't. That's why I have no friends," he replies, expertly cracking a crab leg in half. "But I only review major exhibitions so the people who really suffer are not the working artists, they're the curators."

I can't help but think Sewell's insouciance towards other people's sensitivities is a bit of an act. His own feelings have clearly been hurt, again and again. He remains distraught to have left Christie's but says his boss there made life "intolerable". He wishes he could have had "a proper job" as head of a gallery. He is upset that Blunt's executor, John Golding, never gave him credit for being Blunt's loyal friend. He keeps writing for the Evening Standard despite his wages being cut "because one has to swallow one's pride".

But when I ask him about this vulnerability, of course he denies it.

"There's no point in letting it hurt," he says. "Once you realise your role is court jester, you don't allow it to hurt."

The only true friends he seems to have are his dogs – Lottie and Gretel – of whom he talks with inordinate fondness. Sewell likes the Bibendum mayonnaise so much he takes a pot home to share with them.

We finish the meal with Bailey's ice-cream. Sewell drinks his coffee quickly then hoists himself up off the chair, shrugs himself into his coat and balances precariously on his two crutches, tying the string from his bag around one of the armholes.

He is making his way back to Wimbledon via the number 14 bus and the District line. What about a taxi?

"Taxis are not very comfortable," he replies. "There's this sort of undignified scramble to get in, literally condensing yourself and if you're on crutches, you go flying and end up with two long strips of aluminium on the ground and the meter says £2.60 and you've just said 'Good evening.'"

On the way out, Sewell slips a crisp £20 note into the waiter's hand. Underneath his hard shell, he can't help being nice. But don't tell anyone: it would ruin his reputation.

The Essential Guide to Painting Portraits (Ideal Life, £14.99) is out now