I am eating oysters with shallot cream and a delicate pear sauce at Mirazur, a two-Michelin-starred restaurant on the French Riviera. The glass-fronted restaurant perches on a hillside with breathtaking views over the beach, harbour and town. It isn't surprising to find such a high-end restaurant on the Côte d'Azur. But this isn't the celebrity haunt of Saint-Tropez, Cannes, Nice, or Monte Carlo – it is Menton, the little-known last stop on France's famous coast before you cross into Italy.

Menton is an unlikely location for a restaurant that regularly makes the annual list of the world's 50 best (it is currently number 28). It's a sleepy little town, beloved by genteel visitors who come for its microclimate – said to be 3C warmer than the rest of France – and exotic gardens. But Mirazur, which was taken over by Argentinian chef Mauro Colagreco in 2006 and has been growing in reputation ever since, has put it on the culinary map. The restaurant's dishes are deceptively simple, based around two or three ingredients from the fish market or the its own gardens: sea cucumber with spring onions, say, or Cevennes onions with taleggio sauce and black truffles. On my 11-course tasting menu (€105pp, plus wine), just one dish, crispy pork with pickled pears, contained meat. Dishes are delicately garnished with flavour-packed herb snippets and bright pink or purple edible flowers.

This attention-grabbing restaurant has brought Menton a glimmer of the limelight that the area's more famous resorts bask in, but it still feels like an ordinary small town, albeit one with an exquisite medieval heart and an extraordinary setting between mountains and sea. The beaches – a mix of stony and sandy, public and private – are packed in high summer, but out of season they were empty. To me, it felt warm enough in mid-March for sundresses and flip-flops, but locals were still in coats and scarves – one even stopped me in the street to ask if I was cold. It didn't help that it was pouring with rain – so much for the "316 sunny days a year" trumpeted by the tourist board. Luckily I was in town to eat, not to sunbathe.

The cuisine has a notably Italian influence – being so close to the border. It did not become part of France until 1860, having belonged to the Republic of Genoa in the 13th century, to Monaco from the 14th, and been a protectorate of Sardinia in the 19th.

At Le Bouquet Garni (+33 4 93 86 20 71, 1 rue Palmaro, about £30pp), run by a family from Rome, the standout dish was a simple tomato and mozzarella salad made with the freshest, creamiest burrata. Le Cirke (+33 4 89 74 20 54, restaurantlecirke.com, about £25pp) on the seafront specialises in seafood – the signature grande friture is a delicious mix of crispy fried squid, prawns, scallops and white fish, served on a sheet of brown paper with nothing but half a lemon and some bread (ask for aïoli and a side salad). One of the chefs at Mirazur told me he likes to eat at Le Petit Port (+33 4 93 35 82 62, 4 rue du Jonquier, about £60pp) on his days off: it's a more upmarket (read pricey) bistro with local specialities.

Regional dishes to look out for include barbajuan, a sort of ravioli; socca, chickpea pancakes; and pichade, a tomatoey version of the pissaladière onion pizza/tart speciality. There's an excellent covered market each morning at quai de Monleon, but the one at Ventimiglia, a short train ride across the border, is even better, with the added excitement of going shopping in a different country. Mirazur chefs buy from this market, which has a huge mix of stalls, from commercial vendors to old women selling a few vegetables from their garden.

Menton is proud of its citrus fruit, and there are lemons everywhere. Shops are crammed with lemon products: try the jams and liqueurs from Maison Herbin (2 rue du Vieux Collège), lemon-infused olive oil from Oliviers & Co , and lemon biscuits from La Cure Gourmande, both on rue Saint-Michel.

But when a town plays up its fruit-growing prowess, you can be pretty sure it's not a place to party. There are plenty of cafes on the seafront and at the foot of the old town for pre-dinner drinks, but not many buzzier sports. So when a bar caled Time's (facebook.com/menton.times) opened last year on rue de la République, it was welcomed by the town's young people, who go for cocktails and live music until 12.30am – a late night in early-to-bed Menton.

However, there are daytime charms aplenty. A new museum dedicated to Jean Cocteau (museecocteaumenton.fr), Menton's adopted artist, writer and film-maker son, opened in a prime position on the quay in 2011. A blockbuster exhibition of works by Cocteau, Matisse and Picasso is running until 3 November. Included in the €8 admission price is entry to the Bastion museum, a 17th-century fort in the harbour wall that was restored and decorated by Cocteau. Another stop on the Cocteau trail is the town hall's register office (Place Ardoïno), which Cocteau adorned with wall and ceiling murals. Menton couples still get married in his Salle des Mariages, which can be visited for €2 – when there isn't a wedding taking place.

You don't stroll around the gorgeous old town; you hike up it. The steep cobbled medieval steps and alleyways are dark, narrow and winding. Shafts of sunlight reveal jumbled dwellings in muted shades of ochre and pink and gold. On rue Saint-Michel is the basilica of the same name, a baroque church with a landmark belltower. This is one of the venues for the Menton music festival each August (festival-musique-menton.fr). A little further on is the ornate Chapelle des Pénitents Blancs, and right at the top, you come to the cemetery on rue du Vieux Chateau. A peaceful spot with incredible views, it is a place of pilgrimage for rugby fans: the sport's inventor, William Webb Ellis, is buried there.

Staying in Menton is certainly more affordable than in its glamorous neighbours. I opted for the reasonably priced Hotel Napoléon, a bright, modern hotel with decor inspired by Cocteau and British artist Graham Sutherland. Rooms have balconies with views of the Mediterranean or the Alps; there is a small pool in front and a garden at the back (vice versa might have been better). The hotel is in a great location on Garavan bay, to the east of town, 100m from the beach and a 10-minute walk, past Menton's two pretty ports, into the centre.

Menton may not have Saint-Tropez's party people, Cannes' film stars or Monte Carlo's high rollers, but that's what makes the town so appealing. It has buckets of old-fashioned charm – and one very modern restaurant.

• Flights were provided by British Airways which flies from Gatwick, Heathrow and London City to Nice from £69 one way. Train travel from Nice to Menton costs €13.60 return (ter-sncf.com). Accommodation was provided by Hotel Napoleon (+33 4 93 35 89 50, napoleon-menton.com), doubles from €89 room only. Dinner was provided by Mirazur (+33 4 92 41 86 86, mirazur.fr), three-course dinner from €47