Archive for August, 2010

big society

A4E and Miss £5 Million will get you a job….pretending to get other people jobs. I love it when a plan comes together.

An unusual case. (Horrible right-wing) Blogger convicted of threatening to kill three judges. One judge is himself a (not all that horrible) right-wing blogger, and another is a horrible right-wing blogger and journalist’s brother. The prosecutor in the case is the former idol of Firedoglake, Patrick Fitzgerald.

Here’s something interesting.

We must also consider the alternative that many of the most prominent and powerful Afghans are in fact motivated by greed and opportunism. [harrowell: ya think?] It is therefore in their interest to maintain the status quo of massive US and international spending that fuels the Afghan “rentier state” economy.

This isn’t just recreational cynicism; they argue that the latest announcement of a clampdown on private security companies in Afghanistan is to be taken more seriously than the last six, and that this actually represents an effort to integrate them into the Afghan government’s forces or at least its allies. Importantly, and very differently from Iraq, the main players are local rather than foreign – like the 24,000-strong Watan Group. (Check out their Corporate Social Responsibility page.) Rather than just being part of the ISAF baggage train, they’re a significant nonstate actor in Afghan politics.

If you were feeling optimistic, you might consider this as being similar to the various political fixes the Soviets arranged in 1988 to keep the roads open for the Afghan government post-withdrawal. If you’ve been reading this since at least 2007, you’ll know that I think the absolute best that could happen in Afghanistan would be to get back to something like the 1989-1992 period, just without the continuing US/Pakistan/Saudi destabilisation and the cut-off of Russian aid that kicked off the civil war (and the destruction of Kabul, the invention of the Taliban, and so on). I agree this is pessimistic, but then, well, I wouldn’t start from here.

In Iraq, understanding the business/organised crime environment may have played a bigger role than is publicly acknowledged in getting the US Army out of town. For example, here’s a Joel Wing piece on the history of oil-smuggling (you’ll note that the Baiji refinery comes up. party like it’s 2005!). Interestingly, the initial Awakening Council leader Sheikh Abu Risha was an important oil smuggler, and you can bet those networks were of use.

Leaving aside the obvious Afghan export, the analogous business is probably selling stuff to ISAF. Bagram now has its own cement plant, inside the perimeter, but that’s a Turkish construction firm.

A major philosophical difference between the UK and USA halves of the SIGINT tribe, and between the tribe and the military, was who the intended customer for intelligence was. The Americans were traditionally very keen on bringing everything back to Fort Meade for processing and analysis, and then feeding intelligence reports to the top level of government. As very often, the British followed suite, but only up to a point. GCHQ as an institution was traditionally very concerned with its status as a direct contributor of intelligence to the core executive, co-equal with MI6, the diplomats, and the armed forces’ Defence Intelligence Staff. In fact, as we saw in part one, in some ways it had greater independence and status – as well as its own private diplomacy with the Americans, it also has the unique privilege of sending the prime minister intelligence outside the formal processes of the Joint Intelligence Committee machinery.

In practice, though, it was often more interested than the Americans in pushing information forward to the military in the field or to diplomatic posts. This was influenced by the British specialisation in ELINT, which tended to be more interesting to the military and more dependent on collection from their ships or aircraft, and also by the Bletchley heritage. ULTRA’s triumphs weren’t just about Alan Turing or about computers; a huge problem that had to be solved to make it useful was the distribution of highly secret information to the army in the field in near real time. (A key motivation was that GCHQ was well aware that the Germans were in the habit of breaking Allied cyphers, and then transmitting the results over their ENIGMA and FISH radio networks – allied traffic turned up in the take all the time.)

It’s probable that a major reason why GCHQ wasn’t more like that, rather than less, was that the American approach was useful politically. Supplying the Cabinet directly obviously helps to win the budget wars. Similarly, too much emphasis on tactical work might give the impression that the agency was a support service to the armed forces, rather than something like a fourth service in its own right. Horrors.

But this didn’t stop some important projects from being designed to fill the gap. GCHQ had been called in to investigate whether the Territorial SAS’s stay-behind reconnaissance teams, intended to target the Red Army’s rear areas for air (and specifically nuclear) attack, were likely to avoid getting caught for long enough to be useful. They demonstrated that, even using burst transmissions, the Soviet electronic-warfare units would very likely triangulate on them within 24 hours. This obviously wasn’t good enough, and one of the results was the Nimrod R1, the RAF’s airborne electronic intelligence system. System is the right word; as well as the planes, the project included a special RAF intelligence centre at Wyton, communications links forward to the army, and the capability to have intelligence analysts, Army liaison officers, or linguists actually fly on the plane with the radio operators. (As well as the R-1s, the Nimrod MR2s have done a lot of this in Afghanistan, and paid the price.)

That was then; the RAF is now leasing three RC135 aircraft from the Americans, actually older than the R1 airframes and designed for the model then considered inappropriate.

This may be a serious problem; one of the big questions facing GCHQ is the age of fibre-optics and open-source cryptography. With less and less telecoms traffic going by satellite or microwave, and less of that going in the clear, what to do? Further, the questions aren’t the same ones as they were in the cold war.

An example of why this is relevant is this piece by Spencer Ackerman on the US Air Force’s MC-12 aircraft and its role detecting improvised explosive devices in Afghanistan. In fact, as he points out elsewhere, the MC-12 (roughly, a Beech King Air stuffed with sensors, extra fuel, and spooks) does a lot of other things too, although they’re mostly classified. It’s an example of a current trend – rather than UAVs, there’s increasing interest in cheap light aircraft carrying the latest sensor packages. This has the advantage that they can take up intelligence agents and work more closely with the troops, as well as being cheap.

There’s much more detail here, which makes the interesting point that the role of Task Force ODIN, set up to kill insurgent bombmakers in Iraq, is now a broader one in support of the counter-insurgency strategy. This changes their relevance from being purely tactical and military to being political and strategic. They haven’t been inactive on this – from Aldrich’s site, here’s a fascinating data sheet on their backpack SIGINT kit, the ideal gift for the geek who has everything and a death wish and very similar to some Rohde & Schwarz mobile network testing gear.

Speaking of mobile networks, Aldrich also confirms that a capability to listen to cellular networks exists, mounted on the British Army’s three Islander aircraft – it’s not clear from his discussion whether this means the access side or microwave-backhaul, or whether this relies on the old A5/0 and A5/1 cyphers still being in use.

A major claim of the recent group of “intelligence historians” is that the study of the secret world is the “missing element” in contemporary history – that, just as the history of the second world war needed revising after the British government finally let on about ULTRA, history (especially of the Cold War) is missing the perspective provided by intelligence. Richard Aldrich’s GCHQ is certainly part of this project, just as his The Hidden Hand was one of the better works in it for covert action, propaganda, and human intelligence.

But do we know that much more about the main line of history from it? There are, of course, a couple of serious documentary and methodological problems with this. Even where we do have good sources on the history of secret intelligence, it’s typical for the actual intelligence product to remain secret. We have a reasonable idea of what all those antenna farms were after – we don’t have much, post-ULTRA, of what the prime minister actually got delivered to his desk in the blue-jacketed files. Writing my own Master’s thesis, I remember that the literature was rather better on the contribution of Soviet intelligence to the 1973 crisis than the US kind, but even that was because various individuals had been forthcoming. The Soviets tried to persuade Sadat to end the war by producing MiG-25R imagery showing the Israeli counter-offensive building up; he wasn’t apparently convinced. We don’t know, however, if the Americans did anything similar with the Israelis, although we do know that the Israelis weren’t sharing their own information with the Americans. (And we know now that Ted Heath turned off their SR-71 operation out of Lakenheath, so how much did they know?)

There’s another problem, though, which is understanding what contribution intelligence actually makes to decisions. Cynically, you might say that giving politicians more data is pointless; they’ll either ignore it or pick the bits that suit their preconceptions. John Keegan argued that across history, intelligence was more often misused, ignored, or just irrelevant to the balance of forces on the ground than not. Obviously, having regular deliveries of ULTRA decrypts didn’t prevent Dunkirk, although it may have helped bring off the evacuation. Even more obviously, whatever intelligence sources Tony Blair was using in 2002 didn’t bring him very much enlightenment. That raises another question – was the intelligence valid even before the upsexers got at it? Why did all the European countries with their own overhead imagery choose to stay out?

These problems are less serious when the events in question were motivated by intelligence interests, rather than by the content of intelligence. Aldrich is good on this – the times when “the SIGINT tail started to wag the policy dog”. Notably, this seems to have been a major motivation in the whole sorry story of Diego Garcia, intended as a replacement for the abandoned sites on Mauritius and Ceylon and for the NSA’s intelligence-gathering ships after the attack on USS Liberty. Around this time, GCHQ also considered building an enormous, nuclear-powered ship intended to contain a complete overseas station of the size of HMS Anderson on Ceylon or Little Sai Wan in Hong Kong, plus a BBC World Service transmitter site – Harland & Wolff’s was commissioned to carry out a design study.

The plan was to have it flagged as a merchantman, but it would have been an enormous and expensive sitting duck.As plans go, at least it didn’t involve ethnic cleansing.

Later, when the third Wilson government decided to pull out of the remaining overseas bases in 1976, it was the GCHQ interest, backed up by the NSA, that led them to keep the presence on Cyprus – as well as huge British intelligence facilities, the Americans had transferred numerous organisations there from Turkey when the Turks asked them to leave, which had then moved into the British bases for security after the 1974 invasion.

So we’ve discussed GCHQ and broad politics and GCHQ and technology. Now, what about a case study? Following a link from Richard Aldrich’s Warwick University homepage, here’s a nice article on FISH, the project to break the German high-grade cypher network codenamed TUNNY. You may not be surprised to know that key links in the net were named OCTOPUS (Berlin to Army Group D in the Crimea and Caucasus) and SQUID (Berlin to Army Group South). Everyone always remembers the Enigma break, but FISH is historically important because it was the one for which Bletchley Park invented the COLOSSUS computers, and also because of the extremely sensitive nature of the traffic. The Lorenz cyphersystem was intended to provide secure automated teleprinter links between strategic-level headquarters – essentially, the German army group HQs, OKW and OKH, the U-boat command deployed to France, and key civilian proconsuls in occupied Europe. The article includes a sample decrypt – nothing less than AG South commander von Weichs’ strategic appreciation for the battle of Kursk, as sent to OKH, in its entirety.

Some key points, though. It was actually surprisingly late in the day that the full power of FISH became available – it wasn’t enough to build COLOSSUS, it was also necessary to get enough of them working to fully industrialise the exploit and break everything that was coming in. This was available in time for Normandy, but a major driver of the project must have been as a form of leverage on the Americans (and the Russians). The fate of the two Colossi that the reorganised postwar GCHQ saved from the parts dump is telling – one of them was used to demonstrate that a NSA project wouldn’t work.

Also, COLOSSUS represented a turning point in the nature of British cryptanalysis. It wasn’t just a question of automating an existing exploit; the computers were there to implement a qualitatively new attack on FISH, replacing an analytical method invented by Alan Turing and John Tiltman with a statistical method invented by William Tutte. Arguably, this lost something in terms of scientific elegance – “Turingismus” could work on an intercept of any length, Tutte’s Statistical Method required masses of data to crunch and machines to crunch it on any practical timescale. But that wasn’t the point. The original exploit relied on an common security breach to work – you began by looking for two messages of similar length that began with the same key-indicator group.

Typically, this happened if the message got corrupted by radio interference or the job was interrupted and the German operators were under pressure – the temptation was just to wind back the tape and restart, rather than set up the machine all over again. In mid-1943, though, the Germans patched the system so that the key indicator group was no longer required, being replaced by a codebook distributed by couriers. The statistical attack was now the only viable one, as it depended on the fundamental architecture of FISH. Only a new cypher machine would fix it.

The symbolic figure here is Tommy Flowers, the project chief engineer, a telecoms engineer borrowed from the Post Office research centre who later designed the first all-electronic telephone exchange. Max Newman, Alan Turing’s old tutor and the head of the FISH project, had shown Flowers a copy of On Computable Numbers, which Flowers read but didn’t understand – he was a hacker rather than a logician, after all. He was responsible for the shift from electromechanical technology to electronics at Bletchley, which set both Newman and Turing off towards their rival postwar stored-program computing projects.

Another key point from the book is the unity of cryptography and cryptanalysis, and the related tension between spreading good technology to allies and hoping to retain an advantage over them. Again, the fate of the machines is telling – not only did the FISH project run on, trying to break Soviet cypher networks set up using captured machines, but it seems that GCHQ encouraged some other countries to use the ex-German technology, in the knowledge that this would make their traffic very secure against everyone but the elect. Also, a major use of the surviving computers was to check British crypto material, specifically by evaluating the randomness of the keystreams involved, a task quite similar to the statistical attack on FISH.

Finally, FISH is exhibit A for the debate as to whether the whole thing has been worthwhile. What could have been achieved had the rest of the Colossi been released from the secret world, fanning out to the universities, like the scientists from Bletchley did themselves? Max Newman took racks of top-quality valves away from Bletchley when he moved to Manchester University, and used them in the very first stored-program, digital, Turing-complete computer; Alan Turing tried to do the same thing, but with a human asset, recruiting Tommy Flowers to work on the Pilot-ACE at NPL. (Flowers couldn’t make it – he had to fix the creaking UK telephone network first.) Instead, the machines were broken up and the very existence of the whole project concealed.

On the other hand, though, would either Newman or Turing have considered trying to implement their theories in hardware without the experience, to say nothing of the budget? The fact that Turing’s paper was incomprehensible to one of the most brilliant engineers of a brilliant generation doesn’t inspire confidence, and of course one of the divides that had to be crossed between Cambridge and GPO Research in Dollis Hill was one of class.

news

Well, here‘s a piece of news. Al-Jazeera seems to think the Russians are hoping to exchange him.

OK, so some more on Aldrich’s GCHQ. Obviously, technology is at the centre of this story. I’ve said that the signals intelligence world is special among spooks because it guarantees results – they may not be the right results, they may not be helpful, but you can usually depend on it producing something to whack on the PM’s desk, that he or she can spring on cabinet ministers later. One of the things that makes it special is its industrial nature; unlike most forms of intelligence, it needs machines, great buildings, thousands of technical staff working shifts, and its performance is heavily dependent on engineering.

From a budget-politics point of view, there’s a symbiosis here. Back in 1941, the permanent secretary of the Foreign Office got into the habit of bringing top officials from London to be dazzled by the brilliance on display at Bletchley and terrorised by its security officers. It worked. On the other hand, getting the resources necessary to build the crypto industry required the direct intervention of a group of top scientists around Turing and Gordon Welchman with Churchill. Of course, as someone regularly dosed with their product, he didn’t find it hard to give them what they needed, which was money and lots of it. By mid-1942 and the introduction of the third rotor on the Enigma machine, it became very obvious indeed that signals intelligence was now an industrial enterprise. This led directly to the decision to let the US Navy build its own Ultra capability, and hence to the founding treaties of the special relationship.

As soon as the Holden agreement let the Americans get hold of the Ultra secret, however, Bletchley was frantically building up new technology that would maintain a bargaining edge. The huge effort to crack the German on-line cipher known as FISH, for example, which led to the COLOSSUS computers, has to be seen partly in this light. This combination of a sort of fatalism – the Americans would eventually triumph – and a hunt for an edge would colour GCHQ’s role in the history of technology from then onwards. Despite its founding achievements in computing, and those of the post-war diaspora of scientists, they were always suspicious of British technology. Post-COLOSSUS, GCHQ joined the long, long queue for IBM 360s and then, oddly enough, veered off to get all its computers from Honeywell into the 1980s.

On the other hand, a number of key research projects were pressed ahead, notably a range of exotic over-the-horizon radars, agent equipment, the Nimrod R-1, and the never-completed Zircon satellite. This combination of cringe and competition was mirrored by the SIGINT tribe’s attitude to technology in general; starting in the 1960s, they were both keen to spread good cryptography among NATO and other friends, but also to prevent the development of independent crypto. On the one hand, “free licensing” was meant to let second- and third-tier agencies and Western non-governmental systems get access to effective security; on the other hand, rather like the bundling of MS Internet Explorer, it was meant to secure a monopoly. This put the UK in a difficult position – it strongly intended to develop its own crypto, thanks, and export it, but the companies involved very much wanted to claim royalties on their patents.

This eventually ended up with the incredible effort to subvert Crypto AG of Switzerland’s high-end cipher machines (CAG, by the way, owned the intellectual property of Hagelin, the makers of what became the Enigma…), under which the NSA and GCHQ persuaded them to fix certain cryptographic problems, but to leave other security bugs unfixed in order that they could continue to spy on their users. The exploit in question referred to TEMPEST, the now-well known problem where some electronic devices leak information in the clear as radio interference, which strongly suggests that the point was to protect some of the many embassy spying operations.

This couldn’t, and didn’t, last – by the 1980s, as with general policy, the monopoly of security technology was crumbling as the Europeans (mostly) got better at it. There were efforts to change this – GCHQ was given a special responsibility to keep an eye on Nokia, while other allied agencies got tasked with Ericsson, Siemens, Olivetti, etc (but notably not Alcatel). Another important factor, eventually decisive, was that it was moving from hardware to software. In the light of this, the 1990s crypto wars seem a lot more radical than a bunch of geeks playing at spies; something very important did change back there. On a critical note, I did think Aldrich’s book could have done with a good technical reader on software, Internetworking, and related issues – the focus is a bit off here, and he seems to depend more heavily on the civil servants.

Did GCHQ hold back or promote technical progress in the UK? There are various views on this. One is that it’s part of a huge cluster of PhDs in the Severn valley that must be having some sort of spin-off benefit to the country – even if it’s only that when Thatcher offended them to the extent everyone in the computer division of HEO rank or above quit, a lot of other tech companies filled their boots. Another is that it’s a sort of shadow of the British Google that didn’t happen, because the potential founders were wasting their time sucking up to the intelligence-administrative complex.

Of course, it’s true that they invented public-key cryptography in 1971 and didn’t tell anyone for 35 years. But this was largely because nobody could think of a use for it back then. (Apparently, they thought of using it to authenticate nuclear launch orders, until it was pointed out that they didn’t have to be sent in real time any more because the nukes were submarine-launched.) On the other hand, much of its purpose in life is to provide a source of clue for the wider government (a sort of infosec Shi’ism, a marja e-taqlid for system administrators and government ministers), and who can say British governments have suffered from too much competence?

So, Richard Aldrich’s book on GCHQ. This looks like it’s going to be another in our occasional series of multi-part book reviews that nobody reads, as the book is nothing if not comprehensive. (It’s a mere Laundry-esque 666 pages in paperback.) Apart from being packed with good things, like paper and words, as Spike Milligan said about his autobiography, I think it’s undeniable that this is the best factual account of British signals intelligence you’re likely to get. It practically bursts with detail and is clearly the fruit of an enormous effort of primary research, and a fair bit of the secondary kind too. If you want to know about the continuation of the First World War crypto effort into the inter-war era, the construction of the Hong Kong over-the-horizon radar site on top of a sheer cliff thousands of feet high and the number of Land Rovers the RAF Regiment lost over the edge, or exactly how many index cards Special Branch found in Geoffrey Prime’s private database of young girls, it’s here. This is in itself quite an achievement, given how much of this stuff remains classified.

Of course, what everyone wants to know about is the intelligence special relationship with the US and the other Commonwealth nations. You will not be disappointed. Aldrich argues that we’re unlikely ever to find a smoking document, even after the release of what was described as the UKUSA agreement earlier this year – the terms of the alliance were repeatedly renegotiated, and its content is spread over many different documents. In fact, it might be more interesting to think in terms of the technical documents. He makes the excellent point that the alliance consists, in practice, of a set of shared operating procedures and technical standards, rather like the Internet, with the distinction that here everything is secret. Rather than gaining access to the IETF by making your work public, you gain access to the tribe of SIGINT by submitting to ever greater secrecy, in a sort of masonic career of increasingly complex rites. Crucially, wherever the documentation goes, the internationally agreed security requirements go with it. This, of course, has an impact on parallel technological decisions, but I’ll come to those later.

This tribal nature – and in many ways it is tribal, with different agencies’ membership in the relationship stemming from their alliance with the founding couple of Bletchley Park and US Naval intelligence – has important and counterintuitive effects on the politics of SIGINT. For example, the tribal leaders have frequently been keen to help their kin succeed in developing new technologies, extracting more funds from national budgets, and securing their secrets from their common enemies. On the other hand, they have also been very keen to prevent them from developing relationships that bypass the central alliance, and to restrict the degree to which they can secure their own traffic against the “level one agencies”, GCHQ and NSA. All tribes, however, are in part mythical, and the status of the leader derives in part from the consent of the led.

In the early 1970s, for example, Henry Kissinger ordered the NSA and the National Reconnaissance Office to cut off intelligence sharing with Edward Heath’s government (Heath’s GCHQ director was, among other things, in the process of negotiating a special link between the Joint Intelligence Committee and the French equivalent). The British were horrified, but it’s telling that the NSA itself was very suspicious of the move and took steps to undermine it – it seems that information kept reaching Britain via sharing with Canada and Australia. When the Yom Kippur war broke out, Heath retaliated by refusing to let SR71 reconnaissance flights land in the UK or at Akrotiri, and imposing conditions on U-2 operations from the UK, specifically that the imagery from them could not be shared with Israel.

In the 1980s, the Reagan government imposed a similar “cut-off” on New Zealand to protest their refusal to let US warships call without saying if they were carrying nuclear weapons. The New Zealanders were unexpectedly unimpressed, which was at least in part explained by the fact that the other alliance partners continued to pass information to them, and also by the fact that the New Zealand GCSB was a major analysis centre for traffic from Asia. Notably, GCHQ was collecting French traffic on their behalf as part of the Rainbow Warrior inquiry.

Over the years, the power-relationships within the alliance shifted with the varying scarcity of different resources. To begin with, in the heroic days of Bletchley Park, the UK had a strategic advantage based in its extremely scarce knowledge of cryptanalysis and computing. As the importance of computing and bulk data processing in general grew, this shifted towards the US; they had more money, and their own technology was improving fast. The result was that the Commonwealth partners essentially traded collection for analysis – we had territory, relationships, and collection platforms that the Americans didn’t. That included some hideously dangerous overflights, submarine missions, and covert actions around the edge of the Soviet sphere of influence. Again, if you want to know what it was like sailing an old submarine into Polyarnyy harbour in 1959 without asking, it’s here.

This oversimplifies; in fact, however much money the Americans threw at the problem, they didn’t break the Soviet high-level ciphers between Black Friday in 1948, when the USSR carried out a forklift upgrade of their whole crypto network to end the VENONA codebreak, and the late 1970s. Information had to come, instead, from new forms of collection, targeting networks that weren’t encrypted because they were thought to be secure, and by studying the electronic signatures of new weapons. As a result, the inter-allied playing field had a structural skew towards the British, who specialised in forward collection and in ELINT, building up an enormous library of Soviet radars and emplacing microwave listening stations in unlikely places. However, it’s unlikely that this was realised at the time – it was all too obvious that Fort Meade was filling up with more and more computers, and it’s not clear how honest they were about their successes or failures. There was a sort of technical cultural cringe on the British side.

The other new field was of course space. Starting in the late 1960s, the US began to collect much more of its signals intelligence from satellites, invulnerable to the political turmoil down below. However, this brought about another twist in the political relationship. The Americans had ELINT and COMINT satellites, the allies didn’t. But when the RHYOLITE satellites, originally intended to spy on missile telemetry, started to pull in more and more data from the new microwave telecomms backbones, the NSA was forced to rely on its allies to deal with the mountains of data. That meant, among other things, a momentous step – intelligence sharing now included readout, letting the allied agencies point their dishes at the satellites and receive the stuff directly. (Incidentally, this is the purpose of Menwith Hill – it slurps intercept material from satellites and passes it to Cheltenham.)

At the beginning of the 1980s, then, the alliance was undergoing the sort of integration process that the founders of the European Union hoped to see. Rather than painful negotiations in high politics, technical interworking would result in a natural binding together. The system was evolving from the original hierarchical structure into a flatter network, with much greater interdependence. The Americans seem to have been aware that control was slipping away, and made efforts to assert traditional rights, for example by trying to impose the lie detector as part of the common security rules, which even Margaret Thatcher considered illiberal and unscientific. Some tribal practices didn’t translate. The New Zealand cut-off was part of this, as was its failure – among other things, what was to happen about the New Zealanders seconded to Canada and the UK, and the Canadians and Brits in New Zealand? What would the US customers for Korean traffic processed at GCSB say when it ceased to arrive?

Interestingly, the US seems to have found continental Europe more interesting as a result. They made efforts to cooperate more closely with West Germany, while the Germans for their part were organising a new European alliance, and the UK was developing close links with the Mitterrand government’s intelligence chief (while also helping the New Zealanders get information on his agents).

Yet another shift, possibly even more important than the end of the Cold War for the tribe, was now approaching – the end of the microwave network era and the dawn of widely available strong cryptography. Arguably, what is now scarce is code-breaking of any kind, again, and intelligence analysis; computer power has never been cheaper, while mass collection is much less practical outside one’s own borders. In fact, pharaonic proposals like the Intercept Modernisation Programme may be better understood as a sort of atavistic harking back to the microwave era or even to the high Cold War’s tunnels under Berlin and Vienna.

However, it’s certain that they ain’t going away. One thing that SIGINT has which other forms of intelligence don’t is that it works, it produces physical output, yer actual primary-source documents – every day, as well as the formal, all-source intelligence reports on particular topics, the prime minister is also sent a wedge of selected quotes from the raw traffic. It’s the world’s most classified blog! Thatcher’s civil servants referred to it as Comic Cuts (in the 1950s and 1960s, similar files were known as Blue Jackets or BJs – another way to make the president feel special, I suppose…), but she lapped it up, like they all do.

Daniel Davies once remarked that secret information is a drug – it alters your perception of reality and makes you feel superior to other people – and that it isn’t usually considered wise to make important decisions on drugs. Here’s the problem; whether or not the raw matter is actually useful, whether it’s typical or misleading, whether GCHQ is breaking a lot of the target’s traffic or none of the circuits that matter at all, it’s incontrovertibly present. They will produce something rather than nothing.

In our next thrilling instalments: GCHQ and technology, overseas outposts, internal surveillance, and the future…

I’ve finally got around to answering my own question here. The scraper is work in progress at the moment; the original pdf is rendered by pdftohtml into a tiresomely semi-structured (i.e. worse than no structure) tagpile. I was trying to tackle this through recursion, but I might either try using Python’s continue keyword or perhaps trying to pre-tokenise the document based on the number of blank lines between blocks, and then deal with the blocks.

This all depends on the thing actually having any underlying structure, of course – it may be assembled by copy-and-paste, so anything I do will blow up every month. The things I do for England…





Follow

Get every new post delivered to your Inbox.