My first sight of this year’s Arthur C. Clarke Award shortlist left me with the same feeling of queasy disbelief experienced the morning after a general election when you wake to find that the British electorate – most of whom did actually live through the Thatcher era – have somehow conspired to vote in another Conservative government.

We had to rush off to Oxford first thing on Monday morning, so this has been the first opportunity I’ve had to try and put my feelings into words. Catching up with the news and comment of the past two days, I found amongst the various rumination and reaction a post on the Guardian books blog by Damien Walter entitled Should Science Fiction and Fantasy Do More than Entertain? I have to say that when applied to the judges’ decision it seems a fair question.

I’m not having a go at Damien Walter – I follow his ‘weird things’ series with interest and applaud his fascination with the fantastic in general – but I had (perhaps stupidly) been hoping that we were moving however gradually towards a state and time where such a question (such an overworked, ignorant question) need no longer be asked. Apparently not. In the grudge match against misconception, it would appear that this year’s Clarke team have succeeded in scoring the most spectacular own goal of the decade.

The central criticism of SF by the literary mainstream persists in the idea that even when SF is cool, groundbreaking, thought provoking and awe inspiring, when it comes to style, form, language, psychological verisimilitude, characterisation, sense of place – all those markers by which mainstream literary fiction is customarily judged – it is slipshod, second rate and often embarrassingly lacking in finesse. In other words, SF is great at ideas but cannot be taken seriously as literature. As a writer and reader who is committed to speculative fiction heart and soul, who believes that SF has given us some of world literature’s most enduring classics and that literature achieves its highest potential precisely where the quotidian and the fantastic intersect, I cannot stress how bored and tired I have become with that pronouncement. I’ve said before that I’m evangelical about SF. I stated here last month that I believe the Clarke Award is there – more than the prize money, more than the kudos of winning – to showcase the best that speculative fiction has to offer, as a platform for us to shout about what we’ve got.

Not, then, as an affirmation of literary parochialism, a placard on the ghetto gate proudly proclaiming our critical indifference.

When I look at this year’s shortlist, what I see is not an honest selection of the best SF novels of 2011, but a political decision to promote what is known as core or heartland SF at any cost, regardless of literary quality, regardless of how far the work goes to promoting speculative fiction as a credible artistic movement. That cost is not only to the authors of the five or six genuine works of literature that have been wilfully excluded from the sbortlist, but to SF in general and those who love it and care passionately about it.

Let me stress that this is not a diatribe against so-called core SF. I write literary speculative fiction, and so my own personal tastes are bound to err in that direction. But when I find and read SF with a hard science edge that is as achieved as art as it is as SF – Ian McDonald, Neal Stephenson, Simon Ings – then I am exhilarated, excited, passionate in its defence. It is not about the what, but the how.

Writing earlier this month about Greg Egan’s The Clockwork Rocket (not one of the shortlisted novels but I sense it very easily might have been), Adam Roberts gets it spot on:

We ought to hold out for the highest standards in our SF content—but we ought simultaneously to hold out for the highest standards in our SF style and form too. Why can’t be have both?…… SF is a metaphorical literature, one that aims to reproduce the world without representing it. It is more akin to poetry than it is to science.

The answer is of course that we can have both, do have both, and that a generous handful of the novels submitted contain both. It can only be a source of shame that this year’s Clarke shortlist does not offer a full and frank representation of that fact. (AR’s post, structured as an argument, is brilliant from beginning to end and I would recommend it to anyone but particularly to members of this year’s ACCA judging panel.)

Some readers of this post might express bafflement at this point. The current shortlist does contain two of my own guesses, after all, two-and-a-half if you count the Miéville, which has things wrong with it, sure, but could not be counted as anything less than a hard-fought and honest attempt to write serious fiction. So what am I getting so worked up about?

What I am getting worked up about is that we now have – in 2011 Booker Prize parlance – a Julian Barnes Situation. In other words, the shortlist as a whole is so weak that the reasonable stuff that is on it is forced to assume an importance that is wholly out of keeping with its genuine stature. Julian Barnes almost had to win last year’s Booker, because any other result would have been ludicrous, a situation as unfortunate for Barnes himself as for everyone else.

I love Jane Rogers’s book. I love its neatly crafted sentences, its clear descriptive language, its all round competence. I think the voice of Jessie Lamb, the story’s young narrator, is beautifully and convincingly rendered, the story itself provocative and compelling. The novel possesses a level of literary achievement that sets it way ahead of the other five books on the shortlist, and I think it will rightfully become a minor classic. I would recommend it without hesitation to anyone, regardless of whether they were regular readers of SF, which surely should be the acid test applied to any novel that hopes to find itself in contention for the Clarke Award. However, in SF terms The Testament of Jessie Lamb is fairly conservative, what you’d call a Kazuo Ishiguro kind of book rather than an Ian McDonald kind of book, and although it should absolutely be on the shortlist, in an ideal world and measured against other books that should more rightfully have been in contention with it, it probably shouldn’t win.

Drew Magary’s The End Specialist is great fun, with some degree of flair on display in the writing. The book plays around with form a bit and I like that. It’s a neat effort by a first-time novelist. I’m pleased to see it on the shortlist, I hope Magary intends to stick with speculative fiction and that we’ll see more of him in the future. But light on its feet as it is, this book simply does not have sufficient power to propel it to the top. It should have been a wild card, an outside bet, not a major contender.

Then there’s the vexed question of China Miéville. China is a serious artist with serious concerns. He’s a talented writer who cares, clearly committed to what he does and with the intellect and passion to make a unique and important contribution to the field. I like him and I admire what he’s doing. But he is still an evolving writer, a fact that gets overlooked far too often and more or less ignored by pundits who seem determined to crown him the once and future king of contemporary speculative fiction. My reading time these past few days (and fortunately the train journey to and from Oxford meant I had more of it than usual) has been spent in finally catching up with The City and the City, widely considered to be CM’s most achieved work to date. The core concept is glorious, and Mieville handles his faux Eastern Europeanisms beautifully, but as with Embassytown I found the final quarter of the book disappointing. Indeed, TCATC and Embassytown – the book on this year’s Clarke shortlist – in spite of their differing premises are actually very similar, almost templates for one another. Characterisation ends up taking second place to the too-bald exposition of an idea; sense of place – which in the case of both novels has astounding potential – is finally relegated to the status of backdrop for a revolution that fails to ignite our enthusiasm, largely because the characters caught up in it never get around to revealing to us their inner lives. Consequently very little seems at stake. We could easily care more about Mieville’s protagonists – if only we knew more about them.

In sum, I have no problem with Embassytown being on the shortlist – it probably should be. Where I do have a problem is when I see the main discussion around the ACCA coming down to whether Miéville should win the prize for a fourth time. In my opinion he should not, on this occasion – not because it somehow looks greedy, but because there were other books published last year that were ultimately more satisfying than Embassytown. To sanctify the idea that CM is the perennial best that SF has to offer, the ‘thinking man’s science fiction writer’ if you will, does none of us a favour and least of all Mieville.

I do not have much to say about the other three books on the shortlist. The Egan I would probably have enjoyed when I was fifteen and indeed it could easily have been written that long ago. I can see what Stross is trying to do – I can see merit in what he’s trying to do – but the problem is he pays too little attention to the nuts and bolts craft of writing good fiction. What aspires to be hip, slick and cool too often comes across as rushed, written on the run. Personally I prefer Coupland and Gibson. Sheri Tepper’s novel is a retrograde, overwritten fantasy that contains nominal SF elements but to my mind entirely lacks a SFnal sensibility. Could any of the judges honestly say that any of these are better books than Osama, Dead Water or The Islanders? I seriously doubt it.

Whoever wins now, this year’s Clarke, like last year’s Booker, is basically a write-off, an opportunity wasted. I’m sad and I’m angry. Most of all though I feel let down.