When I was still a stand-up comic in the distant past, I was once at a comedy party in East London, at a club I hardly ever played. One of my friends asked the promoter why he didn’t book more female comics, and the promoter explained to him that they only had one spesh act on the bill at a time. A speciality act, in case you aren’t familiar with the term, is someone who deviates from the comedy norm of a bloke with a microphone. So, musical acts, magicians, ventriloquists and women never appeared with one another at this club (and many others) because that would freak out the audience, unaccustomed as they were to seeing a bloke with a guitar and a woman on the same stage (no Blondie fans, presumably).

By the time I left comedy in 2008, it was still the case that if there was more than one woman on the bill, it was probably International Women’s Day. Things were no better for black comics than for women: when Stephen K Amos quipped that he was waiting for Lenny Henry to die before he got more TV work, he was only half joking.

And now Chuka Umunna has spoken out against the same problem in British television: there are so few good roles for black actors that they’re leaving for the US to avoid ‘lazy stereotypes’. On first glance, he has a point. David Harewood – who starred in Homeland - said last year that he’d had to move to the US to make his name. Umunna surely had Harewood and Luther’s Idris Elba in mind when he said, ‘It’s often only after they've made it big in the States that black British actors get more — and more varied — roles here. That is unacceptable and has got to change’.

It’s worth pointing out, however, that Elba was only one of two British actors to have their careers rejuvenated in the UK after appearing in the mesmerizing US show, The Wire. The other was white, Eton-educated Dominic West. Both actors managed to slip through the net in the UK until their talent had been acknowledged by American television. So perhaps another question that needs asking is why British television is not confident about spotting and using homegrown talent.

In part, the problem is surely that we make television in a very different way from the Americans. In the US, a season of a drama show is about 23 episodes, or 12 if it’s on HBO. That’s a lot of episodes which need to be filled with a lot of story arcs. There’s scope for a large, diverse cast in Homeland, Game of Thrones, Elementary or CSI: Wherever because there’s time for each of the main characters to have a few episodes which focus on them. That’s just not possible with a show like Sherlock, where each series runs to a meager 3 episodes. Having a major cast of 5 or 10 characters means you’re much more likely to have interesting storylines for more actors.

And that’s before you factor in the conservatism of some viewers. When the BBC announced a present-day version of Sherlock Holmes, we may have been sceptical but the casting of Cumberbatch and Freeman filled us with hope. When the Americans announced the US version, Elementary, with Lucy Liu as Dr. Joan Watson – not just a woman, but Asian – British critics were appalled (which is a pity, because it’s a great show).

And it’s not just about time, it’s about period. Our big budget television is often set in the past: obviously Downton doesn’t offer many roles to black actors because it is set in a time before there were many black people living in the UK. Even Midsomer Murders, which is famously white, is set in a version of the present which looks like the 1930s.

So Umunna is right to suggest there’s a problem in our television schedules, though for my money he doesn’t go far enough. I don’t just want to see David Harewood playing a great lead role in a drama, I want to see more of all the actors that TV is too often blind to, in programmes made on both sides of the Atlantic, instead of an endless parade of interchangeably pretty people who I couldn’t pick out of a line-up.

When I was growing up, I could watch The Golden Girls on endless repeat on Channel 4. Then there was Sex and the City, and now there’s Girls. Keep this up, and the next show with four female characters talking to each other will be set in a kindergarten. Older women used to be allowed to be funny: can we let them be in sitcoms again? And if an idea isn’t presenting itself: Frances De La Tour and Miriam Margolyes, please, as retired teachers with a stash of medicinal brandy under the sink.

We’re so used to being told that diversity is something that must be grudgingly achieved that it’s easy to forget how many brilliant programmes are made with diverse writers and performers. The glorious, anarchic US sitcom, Community (I’m begging you to watch it) is a case in point: a racially diverse cast, and a 50:50 male/female writing staff.

The problem isn’t just that actors miss out on work, it’s that we — the viewers — miss out on them. Benedict Wong gave a tremendous performance in Lucy Kirkwood’s play about the US and China, Chimerica: where’s the TV drama about Britain and China, starring Wong? Isn’t it about time we had a great drama which deals with our growing dependence on Chinese investment in British firms? I’ll swap you two footmen and a duchess for a contemporary thriller.