Spare a thought for the garden girls. The team of secretaries who keep No 10 running smoothly do not have many perks for their long hours and loyal service. They do, however, get to go on holiday with the prime minister of the day.

For years, while Tony Blair was in charge, the garden girls sunbathed in Barbados at Sir Cliff Richard's mansion, swam in Florida in the Bee Gees singer Robin Gibb's pool, and toured the churches of Tuscany thanks to Geoffrey Robinson's San Gimignano home.

This year they will be going to Southwold, Suffolk. And there may not be much time off to go bird-watching at Minsmere, eat fish and chips in Aldeburgh or taste the local Adnam's. “When Tony was on holiday you could never drag him away from his sun lounger by the swimming pool,” one senior civil servant says. “With Gordon it's the opposite problem - you can't get him off the phone.”

It is impossible for politicians to win when it comes to holidays. If they go to the Caribbean - like the Blairs - they are too flashy. If they stay in a caravan - like the Becketts - they are naff. Take a month and you are lazy, take a week and you are sad.

Somehow, the summer vacations politicians take come to symbolise their character. Harold Wilson burnished his “man of the people” image by sucking on his pipe on the Isles of Scilly. Margaret Thatcher looked patriotic and hard-working when she was photographed on a windswept English beach during her annual short break.

As Chancellor, Gordon Brown often spent summers in Cape Cod where, between tennis and iced tea, he would plough through books by the American neo-Conservative philosopher Gertrude Himmelfarb. The message of his holidays was clear: he was hard-working, intellectual and pro-American. With two children under 5, the Browns now prefer to holiday in Britain - particularly because their youngest son, Fraser, has cystic fibrosis, which makes it difficult to fly.

The Prime Minister's holiday plans this year are being seen as a barometer of his psychological state. David Blunkett says that Brown needs to take a “really long holiday” and switch off completely over the summer after a difficult few months. The Downing Street officials are unsure how long Brown will be able to stay away. Last year, he rushed back from a family holiday in Dorset within days to deal with the foot-and-mouth outbreak. This year it looks as if, even before he has departed, he has opted to lose a few hours of his holiday so that he can have a meeting with Barack Obama, whose visit to London has been delayed.

The other thing about politicians' holidays is that something always goes wrong over the summer. Economic crises, sex scandals, financial shenanigans seem to erupt when the world is waiting to get through security at Heathrow.

It is for this reason that Alistair Darling has decided to go no farther afield than Scotland next month. Last year, he went abroad - and discovered that Northern Rock was about to go under while talking on his mobile phone by a faraway swimming pool that he never actually got into. This year he wants to be within striking distance of London. He is planning to have a couple of weeks in his house in the Western Isles. His wife, Maggie, will be sucking up the Highland midges with the Dyson.

Five things to do in Southwold

1 Walk across the common, tracing the route of the old Southwold railway, past the harbour and over the footbridge to Walberswick.

2 Stroll along the beach, past the beach huts and out along the pier and giggle at the rude water clock.

3 Climb up the 118-year-old, 31-metre lighthouse on opening days and gaze out to sea.

4 Enjoy a pint of Adnam's bitter in the Lord Nelson, the Red Lion or the Sole Bay Inn in the heart of the town.

5 Book a seat at Southwold's summer theatre, where Jill Freud's repertory company is this year performing Salad Days, Mistaken Murder, See How They Run, The Lady Vanishes and Bedroom Farce.

Times recommends

• The Joy of Sex: why you need help in bed
• Don't move - improve
• Adoption: finding my birth mother