A few hours before I arrived at Kit Dean's dairy farm on the edge of the Yorkshire Dales National Park, near where North Yorkshire smudges into Cumbria, he took a call from his animal feed supplier. "I had to put them off again," he says. The bill, running into thousands of pounds, will have to remain unpaid. "They've put a stop on the account," says his wife, Jane, bluntly. "I don't know how we're going to afford to buy the feed we need through the winter."

We are at the table in their handsome, flagstoned kitchen. There are mugs of tea and homemade flapjacks, and a heart-stopping view of the northern hills. The financial outlook, however, is anything but beautiful. The Deans have a milking herd of around 90 cows, and nearly 300 sheep for lambing on their 250-acre farm. Every part of their business is under pressure. At the heart of the challenges they face is what can, by turns, be both farmers' friend and foe: the Great British weather. For 12 months it has been only a foe. It's a narrative of rain and gloom and more rain. "You might get a bad summer," Kit says. "But it wouldn't coincide with a bad winter and spring. I've never seen it as bad as this."

He lists his problems, starting with silage, the green fodder harvested from fields during the summer and stored as animal feed for the harsh winter months. "The first crop of silage was fine," he says. "The second crop was very late and half what it should be. Plus I couldn't graze the cows. They had to be in from July, it was that wet. And obviously you have to feed them properly because if you don't feed them they don't produce milk."

So they had to turn to higher volumes of concentrates than usual, where costs have shot up too, because of grain price rises on the international markets. "Feed has gone up by £50 or £60 a tonne." With each animal consuming a couple of tonnes of concentrate a year, that's a major cost. "As a result, the milk price is still below the cost of production because of the cost of that feed."

Meanwhile, lamb prices have collapsed, and there is the threat of disease in the flock, both Schmallenberg, which leads to birth defects, and the growth-inhibiting parasite liver fluke. Kit is a British farmer to the very soles of his muddy boots: solidly built, determined, proud. At first he asked not to be named in this article; admitted that he felt that in some way he might have failed. But he's come to realise it's not the case; that instead he is simply a victim of a brutal set of circumstances far beyond his control. "Consumers don't realise how tough everything is," he says.

British farming is in the midst of a very deep crisis. After a drought in its first months, 2012 went on to become England's wettest year on record, and the second wettest in the UK as a whole. Early summer was a deluge. High summer was sunless, resulting in silage that was as much as 40% lower in all-important sugars. The rains came again for the heart of the growing season. "This crisis compares to both BSE and foot and mouth," says the farmer and agriculture expert, Donald Curry, who chaired the government inquiry into food and farming after the foot-and-mouth outbreak in 2001, and who now sits in the Lords as a crossbench peer. "The weather has affected the entire country and some have had the double whammy of bovine TB and Schmallenberg in sheep." There are also price pressures on both inputs – feed and fuel – and the amount retailers are prepared to pay. "We know it's going to affect farmers for this year and next year. For the farming help charities this has become a very serious issue."

Just before Christmas, responding to dire reports from charities, the Prince of Wales convened an emergency meeting at Clarence House. In 2010 he had founded the Prince's Countryside Fund (PCF), to support people working in agriculture, using money gathered from corporate donors. "One of the reasons the prince wanted to establish the charity was to create an emergency fund," says Tor Harris, director of the PCF. At that meeting the entire £150,000 emergency fund was earmarked for distribution to farmers in crisis. It was matched by another £150,000 from the Duke of Westminster, one of Britain's biggest landowners. Donations from corporate partners, including Waitrose, Asda, HSBC and even McDonald's, brought the emergency fund to around half a million pounds, which is being distributed by four charities.

The Royal Agricultural Benevolent Institution (RABI), founded 150 years ago, is one of them. It used to be a standard chronic welfare charity providing help to the elderly and disabled with connections to agriculture. Now a lot of its work involves people in acute distress. "In the past year we've seen a dramatic rise in calls to our helpline and those are coming from working farmers," says Philippa Spackman of RABI. It is, she says, the small family farms and, in particular, the tenant farmers who are being hit hardest. "The narrative is never just one thing. It's two or three coming together, a perfect storm of need. And the situations can be drastic. Our welfare officer in Cornwall was even handing out sandwiches from her car to people who had nothing to feed themselves. We are talking about farmers being pushed on to the breadline."

What's more, with changes to state benefits being introduced in April, they expect to see even more hardship. "These are people who might have a freezer full of beef, but not enough money for washing powder. We're paying for school uniforms, basics like that." The issue now is just how long the situation will continue for. Spackman says: "We are concerned that if the number of calls continues to rise at the same rate we simply won't be able to meet demand." The PCF acknowledges the problem. "It's why we're calling on the public to help get the emergency fund to £1m," Harris says.

Kit and Jane were one of the farming couples who turned to RABI for help and were grateful for the welfare grant and support, though it's clear they hated having to ask for it. I ask Kit if he ever thinks of finding something else to do. After all, while he's worked on farms all his life, he's not part of a multi-generation farming family. "What else would I do?" he says. "When it's all you've ever known, where else do you go?"

A journey across the landscape of British agriculture can be a dispiriting business right now. On the eastern side of North Yorkshire at Northallerton I meet 67-year-old Edward Dennison, a fourth-generation farmer who is currently passing on the 830 acres he farms to his sons. "Until yesterday we were dairy, sheep and arable," he says, when we sit down for coffee. "Now we're just sheep and arable." The day before he had sold most of his cows. After a century in dairy farming the family is getting out of that side of the business.

"Dairy hasn't been that clever for 10 or 15 years, but now…" his voice trails off. "What's crippled us is the input costs, the feed. We got enough silage by volume but it's not a good enough quality so the cows are not milking."

We go out for a walk to have a look at the cow sheds, where what remains of the herd is still housed prior to going off to new owners. "I've never experienced bad weather like we have had, not in 60 years," Dennison says. We stop and look out at the fields in front of the farmhouse. "That's meant to be a field of winter wheat," he says. "It's meant to be green." There is a sludgy expanse of brown in front of us. "There are three fields we haven't even attempted to drill. I'm sick of looking at my waterlogged fields." British wheat harvests are down nearly 15% and much of it is of very poor quality. In 2011, 90% of the harvest was of a high enough grade – rich in protein and gluten – to be milled for flour. In 2012 only 10% was. Millers had to import the rest.

Edward says part of the reason the cows are going is because neither of his sons likes that aspect of the business. "Though if it made economic sense for us to stay in dairy it might be different. It's a global market now and we are a high cost country. Only 6% of dairy production is actually traded on the world markets but that defines the global price."

Dennison and his family are managing to keep their heads above water. But he knows through his work as a volunteer for the Farm Crisis Network that the mood in the industry is very bleak. He has been there at the end of the phone, offering advice to other farmers who are not managing quite so well. "One farmer I know had to sell his herd simply because he couldn't afford the feed." And then he says: "Farming can be very lonely. You're more often than not working by yourself, battling the weather. You're constantly trying to play catch-up. To appreciate the loneliness and isolation is very difficult."

The usual reply to these stories of woe is that farmers should diversify, become entrepreneurs, but that isn't always the solution either. Over in Cumbria, Caroline Watson has done everything she can and is still failing to make ends meet. She and her husband, John, have the tenancy of Yew Tree farm, a National Trust property once occupied by Beatrix Potter and which even featured in Miss Potter, the film starring Renée Zellweger. Although there are 700 acres, more than 250 of them are the stunning crags and fells of the Lake District National Park.

It is land suited only to livestock, which was why it appealed to the couple when they came here 10 years ago. They have around 90 Galloways and a few hundred head of Herdwick sheep, the stout-legged Lake District breed. "We decided to try to take out all the input costs by swapping to Galloways which can forage for their own food." Likewise, they chose the Herdwicks even though they don't breed until they are three years old and generally have only one lamb (rather than breeding at one year and having two lambs as with more standard breeds).

It was, she says, a move into a premium product, with a marketable back story: the landscape, the short supply chain, the Beatrix Potter associations. They set up their own meat business. They offered B&B accommodation, and even clambered on to the street food craze with a high-end burger van to trade at a local beauty spot. But they have been hit by both the weather and the tightening economy. "In 2011 we made 3,000 bales of hay," she says. "Last year we couldn't make any hay at all. Plus the quality of the silage was shocking. It started rotting because of the damp." Like so many others in the industry, they will be having to supplement the feed. "This last year other things have started to hit us. B&B bookings dropped off at the same time as the caterers on the meat side of the business were tightening their belts. They all wanted more for less."

Talking to these farmers, one theme comes up time and again: a failure by consumers to recognise the importance of the food supply. "There's a lot of support for farmers out there," says Edward Dennison. "Shoppers express a lot of sympathy. But then they go and shop on price alone." Kit Dean agrees. "There were protests when the milk price was cut last year and that opened a few people's eyes, but it can't stop there." Or as Caroline Watson puts it: "We should support British farming, buy quality food. Pass up the second holiday if necessary."

One clear example they point to is lamb. First there was the poor feed situation, due to the weather, which led to more breeding sheep than usual being barren and therefore fewer animals to sell. Those that were available fattened more slowly because of that poor grazing. And then farmers were hit by the double whammy of a sudden glut of New Zealand lamb imports and a weakening in demand from Europe due to the eurozone crisis, which resulted in over-supply. Over the past few months, farm-gate prices for whole lambs have dropped by 25% and legs of lamb, by 17%. And yet, the National Farmers Union says, the price for lamb in the supermarkets has fallen only by 2%. On average, farmers are losing £29 for every lamb they sell. While some retailers, including Sainsbury's, have promised to increase the prices they are paying, many of the big retailers are being criticised for not passing on revenue to farmers.

The irony is that, in global terms, this should be a boom time for farming. With the emerging middle classes in India and China fuelling demand for high quality food, and the sustainable intensification of agriculture on the international agenda, landowners and those who know how to farm it should all be well placed. "The medium to long term is very healthy," says Allan Wilkinson, head of agriculture at HSBC, and himself from a farming family. "The short term is much more challenging." The fear is, however, that with so many people leaving the industry, and fewer in-family farm successions, there won't be enough farmers left in Britain when we finally reach those sunlit uplands.

But there is hope. At a tea room in Borrowdale, near Keswick, I meet a group of apprentices on the Hill Farming Succession Scheme, which is also funded by the PCF. A group of sturdy Cumbrian men aged between 17 and 24, sit about with mugs of tea, bantering as only a bunch of students who know each other too well can. "The statistics on succession are hard to come by," says Veronica Waller of the Farmer Network, which runs the two-year course of work placements, training and day release for agricultural college. "But we set up because farmers themselves were talking about how tough it can be to get their sons and daughters to take over. It is a hard physical career and not vastly profitable."

Very few of the men in the room come from farming families, though many of them worked on farms from a young age during school holidays. "My friends call me stupid for doing it," says Matthew Aleixo, 21. "But I know I can make a living doing this." The apprentices talk about the toughness of it. They don't mind getting up stupidly early even on the darkest winter morning. They don't mind the long hours. They all agree that school hadn't worked for them. "Not being good at school is practically a qualification for being on this course," says Veronica Waller, dryly. Or as another lad called Bobsy puts it: "It's a gang. Once you're in you can't get out."

These are dark times for British farming; the challenges are real and serious. That perfect storm of atrocious weather, global economic pressures and livestock disease has come together to create some of the harshest working conditions in living memory. And yet a vibrant, thriving agriculture sector is not some optional luxury. It's not something we can take for granted. The availability of affordable, quality food – the robustness of British food security in the 21st century – depends upon it. Right now we need the enthusiasm of the likes of Matthew, Bobsy and their friends. Fragile as it sounds, they are our hope for the future.

Why the farming crisis means higher food prices

Food price rises are inevitable. At the heart of the problem is the way the British supermarkets work and how they interact with a global food market.

The brutal facts are these: harvests in the UK are down. The 14% shortfall in wheat is only part of the story. The poor quality of what has been harvested has forced manufacturers to look abroad for much more than that shortfall. In January, for example, Hovis dropped its commitment to use 100% British wheat. Likewise, in the same month, McCain's broke its pledge to use 100% British potatoes, with British potato stocks down nearly 20% year on year. Vining pea yields are down 40%. Some apple varieties are down between 30% and 50%. And so it goes on.

The impact is two-fold. A smaller crop means higher prices. Inevitably there will be an even bigger shortfall from within the UK than usual and so retailers will have to look abroad. The problem for the mass retailers is that, with the rise of new food markets such as China, Brazil and India, there is also much more demand for the food produced internationally.

The British supermarket buyers have been used to dictating prices. They now have fierce global competition and increasingly they will find that prices will be dictated back to them. They will have to pass that cost on to the consumer.

Meanwhile, their desperate attempts to maintain the era of artificially cheap food are likely to push more farmers in the UK out of farming. Which in turn will force supermarkets to source even more from abroad. The role of the mass retailers in the farming crisis and, their failure to act responsibly, cannot be underestimated. It's a failure which consumers will soon be able to measure in pounds and pence.

Jay Rayner's book on the challenges of food security in the 21st century, A Greedy Man in a Hungry World, will be published in May.