Archive for the ‘programming’ Category

Progress update on fixing the Vfeed.

Dubai Airport has done something awful to their Web site; where once flights were organised in table rows with class names like “data-row2”, now, exactly half the flights are like that, they’ve been split between separate arrival, departure, and cargo-only pages, they only show the latest dozen or so movements each, and the rows that aren’t “data-row2” don’t have any class attributes but random HTML colours.

And the airline names have disappeared, replaced by their logos as GIFs. Unhelpful, but then, why should they want to help me?

Anyway, I’ve solved the parsing issue with following horrible hack.
output = [[td.string or td.img["src"] for td in tr.findAll(True) if td.string or td.img] for tr in soup.findAll('tr', bgcolor=lambda(value): value == 'White' or value == '#F7F7DE')]

As it happened, I later realised I didn’t need to bother grabbing the logo filenames in order to extract airline identifiers from them, so the td.img[“src”] bit can be dropped.

But it looks like I’m going to need to do the lookup from ICAO or IATA identifiers to airline names, which is necessary to avoid having to remake the whitelist and the database and the stats script, myself. Fortunately, there’s a list on wikipedia. The good news is that I’ve come up with a way of differentiating the ICAO and the IATA names in the flight numbers. ICAOs are always three alphabetical characters; IATAs are two alphanumeric characters, which aren’t necessarily globally unique. In a flight number, they can be followed by a number of variable length.

But if the third character in the flight number is a digit, the first two must be an IATA identifier; if a string, it must be an ICAO identifier.

Advertisements

There should be a special term for the phase in the adoption of an idea between the point at which everyone accepts its desirability, and the point at which it wins over other ideas politically. This isn’t the same as the point of implementation; it’s quite possible for your idea to go into practice, but still to be in the queue elsewhere. So here we are; from E-Health Insider, it looks like the NHS NPfIT is looking at throwing away the disastrous Cerner and iSoft systems and issuing new tenders. In fact, some trusts in the South East have been permitted to sort their own problems out.

However, David Nicholson (the very model of a modern managerialist) is in charge and he for some reason won’t let all the other trusts do this. Even though it is clearly sensible, and is being done, it’s still in the special gap of political unacceptability. I thought this was interesting:

Nicholson said a key problem that the NPfIT programme had faced throughout was the unique requirements of the NHS and what it is trying to achieve. “There is no system off the shelf we could go for.”

Yet the programme was set up so that the NHS IT community, to say nothing of the NHS clinicians, and even less of the patients, had absolutely no input to it. Both Cerner and iSoft are trying to adapt off-the-shelf products from the US. And the attempts to save by outsourcing were disastrous.

“The Lorenzo product is being developed at Morecambe Bay, so we’re really optimistic that something will come out of that, but its not inevitable,” he went on. “And I think we’ll know over the next few months whether these products will actually be able to deliver the things they promised to do.”

That might have been an idea before you bought them, eh. Further, note that he thinks Lorenzo still might get somewhere because of in-house development work

The other issue he said that was being focused on is how to deliver products more quickly and to give trusts more flexibility. Answering questions on the Summary Care Record, the NHS boss said it was possible to de-couple the Summary Care Record from the wider CRS development and simplify it.

This is damning to the entire project. If the record formats can be standardised without the rest of the system, there is no reason for “the system” as sold to Tony Blair to exist. Every trust could have its own system as long as they used the standard.

Remember, the only way to kill a zombie is to aim for the head. By the way, it’s not as if the Americans don’t have Bad Medical IT as well.

What with things like this, the LibDems’ drive to collect members of the scientific-technical intelligentsia at a secret location in Kazakhstan (surely it must be a….), it’s surely time to shake up my ORGANISE project – a messaging and task scheduling system for organisations of all kinds that implements a Stafford Beer viable system approach. (Unofficially, “like a bulletin board that actually encourages people to do something”.)

Before Christmas, we’d gone through a couple of iterations of the spec, I’d vaguely decided to implement it as a Django web application, and I’d written a first version of the models.py (the file that defines the database schema in Django) that (I hope) embodies Duane Griffin’s design of the data model. I also managed to discuss it at some length with Chris “Chris” Williams…

Now, it’s unlikely to get anywhere by the 17th of March, so Rewired State is out as far as our methodology of conference-driven development goes. But I want to shake the thing up, and I’m increasingly interested in alternative routes; specifically, should it perhaps be more explicitly about messaging/communications rather than being a read-heavy Web site?

I’ve been reading up on XMPP messaging, and specifically on its publish-subscribe protocol. Each group in an ORGANISE instance would look like a virtual user. This would make sense – the defining characteristic of membership is the right to send to that group, after all. Percolation rules would determine whether a message from a group would be extended into the next biggest group; thematic groups would work in a similar way. In fact, the XMPP Standards Foundation has the notion of “collection nodes” in XEP 0248 which look very much like ORGANISE groups.

Among other advantages – federation with a lot of IM communities and social networks, the ability to interact with the ORGANISE server via other XMPP networks (which could be handy), a generally more real-time system, and the possibility of interconnecting different ORGANISations. And, y’know, it *feels* more appropriate for various reasons summarised here. Unfortunately it looks like 0248 isn’t well supported yet…but what say you?

OK, so I was feeling sufficiently foolish to try and install the all-new version of Python for Symbian S60 phones. Not least because of rumours that things like the Location API (i.e. “all the interesting or useful stuff”) have been liberated from the finger-waggy signing process…

Unfortunately, Nokia has shipped it without completing the same finger-waggy signing process it imposes on everyone else, so it fails on install with “Certificate error – contact the application supplier”. Nokia FAIL. So I have to explicitly disable the fancy security system in order to install software supplied by the system’s manufacturer. Not just that, but software which is open-source, so I can read the damn thing myself. Why can’t they get it right already? GAH.

So, turn off the certificate check, and it installs. Great. Time for a quick hello world from the interactive interpreter. But no…”Python runtime missing!” We’ve just installed the sodding thing.

So no, until someone gets a grip I won’t be the one to do the S60 version of FixMyStreet:-)

OK. So we looked into voice stress analysis and the world telecoms infrastructure. And we concluded that proper VSA – the sort with the peer-reviewed scientific papers an stuff – was technically impossible. Recap; the original VSA research is based on a change in a signal in your voice between 8 and 12Hz, but even the highest-quality voice codecs used for public telephony filter out everything below 50Hz, so a VSA system based on – well – science couldn’t possibly work.

But there was always the possibility that “Nemesysco” had hit on some kind of roaring king-hell breakthrough. Minitrue couldn’t find a copy of the patent that covers their product; you might wonder why there wasn’t a US patent if it’s so great, or why every call-centre workflow system and high-end mobile phone in the world doesn’t have it as a much-valued standard feature, or why Amir Liberman, the CEO of Nemesysco, isn’t incredibly rich.

After all, he’s been hawking it since at least 1998. His company was formed in early 2000, just a tad late for the joy of the .com boom; at the time they were marketing towards consumers and businesses. But, as the venture capital dried up, the stock exchange cursed everything to do with computers, and it looked like a whole world of vaguely technical young sheisters would have to get a job…something happened, and suddenly his product became “Israeli intelligence service technology” that would save you from terrorists.

There is no evidence that Tsahal or the intelligence services ever made use of it, but as reader Chris “Chris” Williams points, there is a certain mana attached to the Israeli military – link your product to them, and it gets just that bit badder. I tell you, it’s the sunglasses.

So, let’s cut to the chase. The patent is here, thanks to the Canadian government. The “claims” section described how it is meant to work – there’s even an example implementation in Microsoft Visual Basic (you bastards). Here’s how: it takes samples of speech and identifies “plateaus” – flat bits – and “thorns”. Thorns are defined as:

A thorn is a notch-shaped feature. For example the term thorn may be defined as:
a) a sequence of 3 adjacent samples in which the first and third samples are both higher than the middle samples
b) a sequence of 3 adjacent samples in which the first and third are both lower than the middle samples

Now, all speech is roughly speaking a succession of sine waves; by definition it’s going to fit this. Anyway, they take a control sample of speech, count the plateaus and thorns and compute the standard errors, then they ask the questions they want to test, and do the same thing. They then look at the difference between the values and compare them to reference values to tell if you’re lying.

Where do these reference values come from? It is appreciated that all of the numerical values are merely examples and are typically application-dependent. So basically, the all-crucial message on the screen depends entirely on the sensitivity values you punch in to the thing; perhaps great if you’re trying to bully some random Palestinian, but not so good if you need real information.

Hey, if they only knew Visual Basic and were willing to commit Software Crime, Harrow council could crank the reference values down to zero and deny EVERYBODY their housing benefit.

From this, he reckons he can determine:

Excitement Level: Each of us becomes excited (or depressed) from time to time. SENSE compares the presence of the Micro-High-frequencies of each sample to the basic profile to measure the excitement level in each vocal segment.

Confusion Level: Is your subject sure about what he or she is saying? SENSE technology measures and compares the tiny delays in your subject’s voice to assess how certain he or she is.

Stress Level: Stress is physiologically defined as the body’s reaction to a threat, either by fighting the threat, or by fleeing. However, during a spoken conversation neither option may be available. The conflict caused by this dissonance affects the micro-low-frequencies in the voice during speech.

Thinking Level: How much is your subject trying to find answers? Might he or she be “inventing” stories?

S.O.S: (Say Or Stop) – Is your subject hesitating to tell you something?

Concentration Level: Extreme concentration might indicate deception.

Anticipation Level: Is your subject anticipating your responses according to what he or she is telling you?

Embarrassment Level: Is your subject feeling comfortable, or does he feel some level of embarrassment regarding what he or she is saying?

Arousal Level: What triggers arousal in the subject? Is he or she interested in you? Aroused by certain visuals? This new detection can be used both for personal use for issues of romance, or professionally for therapy relating to sex-offenders.

Deep Emotions: What long-standing emotions does your subject experience? Is he or she “excited” or “uncertain” in general?

SENSE’s “Deep” Technology: Is your subject thinking about a single topic when speaking, or are there several layers (i.e., background issues, something that may be bothering him or her, planning, etc.) SENSE technology can detect brain activity operating at a pre-conscious level.

He can apparently detect that all from a total of two measurements. Note also that there is no mention of Micro-High Frequencies in his patent claims; if they were particularly high, they would probably vanish in the band-pass filters above 3.4kHz….

I have collected these claims across his Web site; I wonder if Harrow council is aware that exactly the same technology is being marketed as a “Love Detector“? Or that another company has ripped off the patent, and he warns buyers that theirs won’t produce the advertised 85% accuracy, even though it’s the same patent? This is scienciness, not science. But then, the point is to scare the poor.

Update: See here.

Well I didn’t expect that – it looks like the Canadians have found a rather serious exploit in Westminster 2. And as far as I can tell, it probably affects Westminster 1 through 3 as well. Yes, we’ve got a class break on our hands!

Now, to understand this we need to realise that a very important part of the constitution exists only as a letter to the editor of The Times. Seriously. An anonymous letter to the editor of The Times. I am not joking. The procedure I described in the last post rests on the so-called Lascelles principles, which were laid down in the early 1950s by the King’s private secretary, Sir Alan “Tommy” Lascelles. Here is the text.

To the Editor of The Times

Sir,—It is surely indisputable (and common sense) that a Prime Minister may ask—not demand—that his Sovereign will grant him a dissolution of Parliament; and that the Sovereign, if he so chooses, may refuse to grant this request. The problem of such a choice is entirely personal to the Sovereign, though he is, of course, free to seek informal advice from anybody whom he thinks fit to consult.

In so far as this matter can be publicly discussed, it can be properly assumed that no wise Sovereign—that is, one who has at heart the true interest of the country, the constitution, and the Monarchy—would deny a dissolution to his Prime Minister unless he were satisfied that: (1) the existing Parliament was still vital, viable, and capable of doing its job; (2) a General Election would be detrimental to the national economy; (3) he could rely on finding another Prime Minister who could carry on his Government, for a reasonable period, with a working majority in the House of Commons. When Sir Patrick Duncan refused a dissolution to his Prime Minister in South Africa in 1939, all these conditions were satisfied: when Lord Byng did the same in Canada in 1926, they appeared to be, but in the event the third proved illusory.

I am, &c.,

SENEX.

April 29.

It should be clear enough that a Prime Minister who loses his or her majority and can’t immediately restore it doesn’t automatically get another chance at the polls. This is necessary, in order to observe the principle that the will of the people is expressed in a Parliament they elect.

Now, it’s also clear that this is a weird kind of constitutional text. Some of its features are obviously bound by context; the bit about the national economy would seem to give the monarch a bizarrely large role in Treasury policy, but this is very much a document of its time, a time of menacing war debts, a fixed and overvalued exchange rate, the Sterling balances, and whatnot. According to Peter Hennessy, this principle has been dropped from the Cabinet Office file some 30 or more years ago.

And of course, it’s weird that Lascelles should choose to express himself in this way, rather than issuing a statement, getting a legal opinion, or asking the Government to put the matter before Parliament. Perhaps he found it so obvious that it didn’t need a more formal statement? Or so controversial that he didn’t want to go on the record? But then, why did he go public at all?

I don’t see any reason to think that the Canadian parliament is unable to do its job, or that it no longer represents the electorate. Further, it evidently has another candidate, and written assurances to that effect. And although the Lascelles principles are a British document, they are based on Canadian precedent, so you can hardly deny they have standing. It seems there has been a grave LDQN error – can they really have allowed a prime minister just to get rid of parliament because he doesn’t want to lose?

But ha. Harper didn’t ask for a dissolution, but only prorogation, and Lascelles doesn’t make any mention of prorogation. Here’s the bug. Is there any way to stop a PM from proroguing again, and again? Is there, in effect, a way of getting root access to the executive?

This is especially interesting for a number of reasons. There have been various Acts of Parliament recently that strengthen the ability of the executive to govern by itself, notably the Civil Contingencies Act, which contains powers which almost amount to rule by decree. If it’s possible to kill confidence votes by proroguing for any or no reason, a malicious PM (or actually almost any other cabinet minister, having first invoked the CCA) could declare an indefinite state of emergency with the help of a weak LDQN. And, at some point in the near future, we’re going to replace ours. I’m fairly confident in the current one, but the likely replacement is both flaky and given to statements a lot of people consider unsuitably partisan.

It’s high time to legislate for the whole mess.

Alternatively, if a letter to The Times can be part of the constitution, surely so can a blog post? Perhaps I’d better get in there first – it’s the only way to be safe. Let’s just say that prorogation exists to do two or maybe three things. The first is to send the MPs on their summer holiday. The second is to start the dissolution process. The third is to stop Parliament if for some reason it’s utterly impossible for it to meet.

The first we can surely leave to the Speaker. The second we could simply roll up into dissolution – it doesn’t do anything useful. The third, well. In 1941 the German air force wrecked the Commons chamber, but the Commons reconvened over the road in Church House. Obviously we need some contingency planning, but we don’t need a way to get rid of Parliament altogether.

We’ve blogged before about the NHS’s computer project. So I’m not at all happy about this remarkably silly post at Timmeh’s. He takes issue with a post of Richard Murphy’s about bank nationalisation:

Yup, the people who brought you the NHS Spine are to be put in charge of developing all banking software in Britain.

Well, this is a strawman to begin with. Is Murphy the Chancellor now? But let that pass. Really? A group of mostly American healthcare computing specialists? Several of which no longer exist? Or does he mean the big IT consulting firms involved – like IBM, BT Global Services, and Accenture? Because I’m pretty sure they do a hell of a lot of financial work as it stands; in fact, everyone was worrying last week about IBM’s third quarter results precisely because banks are big customers. (They turned out to be OK, in that mysterious IBM way.*)

But perhaps he thinks the NHS NPfIT was developed by teh government bureaucrats? Or at least, he’s willing to pretend it was to suit ideology? The whole problem with NPfIT, as we’ve said before, is that the system was developed completely in isolation from NHS bureaucrats or indeed anyone else who would have to use it. The NHS trust IT departments were kept well out of it. The upshot was that the developers knew literally nothing of the NHS’s requirements, its business processes, or the data the system was meant to handle.

No wonder it was a disaster. In fact, when a group of US hospital bureaucrats had a go at designing a medical IT system, they came up with a beauty – there’s even a satisfied customer in the comments. Why? Because they knew what it was meant to do and how. Compare this comment:

I met a guy who works for this company. I cannot repeat what he said, since he has a family to feed. But suffice to say he was deeply worried about the implications for safety of life. That was a few months ago.

The whole thing is rotten to the core, and desperately needs to be scrapped. Now.

The good news is that the thing still doesn’t work well enough to turn it on even as a pilot project, so we’re safe for a while yet. But what did happen the last time the Government took on a really challenging in-house IT project? You ask Daniel Davies.

(* probably something to do with asking the fucking users – that or the staple Nazi market, or wearing a lot of pale blue shirts.)

Just to say that the Viktorfeed/Google Maps mashup is now working, and it’s here. Relatedly, the URL for the GeoRSS feed is changing; replace seagrass.goatchurch.org.uk/viktorfeed.xml with www.harrowell.org.uk/viktorfeed.xml.

Met up with regular reader and Linux hacker Duane Griffin to discuss the ORGANISE project . As a result, here’s a new and much amended specification that supersedes the first one.

So I took my stupid damn idea off to the stupid ideas club. When we got there, guess who? Spyblog was waiting at the rendezvous with some Dutchmen and an Argentine documentarist and half the No2ID members not currently in hospital. And after we made our way through Jock McZanu’s EU Maddie monsoon (GOOD HERE ISN’T IT???) to the pub, who shows up but Rat; carrying a total of 30GB of mass storage on his person in an array of USB drives, a fob GPS, and God knows what in his piercings.

Anyway, we talked over the thing, and many other things besides; what should happen if secret police become members? wouldn’t it be easier to do an open-source clone of a BMC helpdesk ticketing app? (why? why? I thought my brain would concrete) how would you sterilise an airport fingerprint reader in less than 10 seconds? So I promised to revise the proposals, and well, here they are.

Or would be, but nobody likes a 2,000 word blog post. So instead it’s here on Google Documents, which probably means something badological. Read. Mark. Learn. Inwardly digest. Comment. Here at first, but if you want to take part just tell me and I’ll give you write privileges. If anyone cares very much I’ll get it set up on Sourceforge and set about preparing a list of functions and tables. I still think Django is the way to go, in which case the mapping of the org model into Python classes into db tables should be as straightforward as these things ever are.