mcnalu's musings

Inequality part 1 - Cointoss Kingdom

2018-08-01T19:34:00+01:00

A talented person can always improve their lot in society. Of course it helps if you start with a lot of wealth in the first place. As the late, great Denholm Reynholm said "When I started Reynholm Industries, I had just two things in my possession: a dream and six million pounds."

But you must also have luck. You can be born a prince, and have a genius level of intelligence, get the best education and be set to inherit the absolute power of the crown and the huge wealth of a treasury, but a single encounter with the wrong virus can kill you before you do.

Luck, be it an innate property of nature, or the unpredictable result of a huge number of deterministic interactions, is needed to bring and sustain your fortune. But here's the question I wish to address. To what extent can the wealth inequality in our society be explained by chance alone?

Let's make things hard for ourselves and imagine a society of perfect equality in which everyone has the same amount of wealth. Yes, I know this is far-fetched, but humour me for a while and treat it as a fairy tale.

Now imagine that the king decrees all wealth is void, including his own, and everyone must have perfect equality of opportunity. The king locks up all existing forms of wealth in a secret cave and to each person he gives one coin. Possession of one coin is enough to make sure your kingdom will provide you with life's basics of food and shelter, but no more. Your life is austere. The king abdicates and accepts the same austerity.

But the coin is magic. Once a year the owner can toss it and if it comes up heads, a second non-magic coin appears. Those with two coins are now provided with goods and opportunities beyond the basics. Perhaps some coffee or a chance to travel beyond their home town or city and see more of the world. (Let's not worry too much how the kingdom provides this stuff, and as this is a fairy tale let's assume the king left this task to the Genie of Goods and Services as his final act.)

The next year, each person gets to toss their magic coin again. If they get heads, another two non-magic coins appear, doubling their wealth to four coins in total. And if they get tails they just keep whatever they've got.

This process continues. Each year those lucky individuals who get heads double their wealth, that is, the number of coins. After three years, the luckiest people who got heads all three times will have 8 coins (2×2×2), some will have 4 coins (2×2), some will have 2 coins, and a few unfortunate souls will continue to lead a basic existence with just one coin, albeit a magic one. Our initially perfectly equal society has now become somewhat unequal, and all through chance. No skill is involved at all.

If the magic coin is also a fair one and has a 50:50 chance of heads or tails then it is fairly easy to figure out what fraction of the population has 8 coins. The probability of getting one heads is a half, so the probability of getting three heads is ½ × ½ × ½ = ⅛. So one eighth of the population, or 12.5%, will have eight coins. The exact same argument can be made for three tails, so an eighth will have only one coin.

For those with 4 coins, they must have got two heads and one tails. Now this requires a bit more care to understand. The probability of getting a particular sequence, say, heads then heads then tails, or HHT for short, is also one eighth. In fact, any specific sequence of three coins has probability of an eighth. But, there are three sequences that have two heads, HHT, which we've seen, but also HTH and THH. So the probability is three-eighths or 37.5%. Again the same argument can be made for two tails.

So, to summarise, there are only four possible levels of wealth:

An eighth or 12.5% of the population have 8 coins.
Three eighths or 37.5% of the population have 4 coins.
Three eighths or 37.5% of the population have 2 coins.
An eighth or 12.5% of the population have 1 coin.

If you add these fractions up you get 1 or 100% which means we have indeed accounted for everyone and all possibilities.

After three years, this society is still quite an equal one, and there are only four levels of wealth because there are only four possible outcomes, so it bears no resemblance to any reality. But let's now imagine we let this go on for a generation, say 30 years.

There are now just over a billion outcomes (2 raised to the power 30), and we can prove something that is not at all intuitively obvious: in a population of about a billion people it is likely that there is one person who got heads every single year. This hyper-lucky person will be a billionaire with their number of coins being 2 raised to the power 30.

Now if you think there is something wrong with this, then it might help to restate it in a more intuition-friendly way: in a population of a billion people, it is fairly likely that one person will experience an event that only has a one-in-a-billion probability.

Further, we can expect about 30 people will have half this wealth because they got 29 heads and one tails. You can understand this because there are 30 sequences in which just one tails can occur, compared to just one sequence that is all heads.

The distribution of wealth from coin tosses is the well-studied binomial distribution that I described in an earlier post on the statistics of political polling. The 'bi' in binomial refers to the fact that there are two outcomes in each coin toss. This graph shows the distribution after 30 tosses, but the profile of the distribution is much the same for any number of tosses, resembling a bell centred on half the number of tosses (the fact it looks like a bell has a significance I'll return to in a future post).

As we saw for three coin tosses above, if you add up all the probabilities on this graph you get 1 which again means we've accounted for all possibilities. But more interesting is the fact that if we add up all the probabilities from 10 to 20 inclusive, then we get 0.96. In other words, 96% of the population will find their number of heads being between 10 and 20 after 30 years. The remaining 4% are split evenly between being either very lucky and wealthy, or very unlucky and poor.

After 30 years of magical coin tossing in our mythical kingdom, we can use the binomial distribution to calculate a statistic that measures the wealth inequality. First we work out the average (specifically the mean) wealth of the wealthiest 5% of people in the kingdom. Next we work out the median, which is the wealth that divides the kingdom into two equally sized groups. Our measure of inequality is to divide the average of the top 5% by the median. Without boring you with the numerical finagling of the binomial distribution, the result for Cointoss Kingdom is 24.

So what? Well, we can now compare this with the same statistic produced for our own world by the OECD. This graph shows data from Table 6.3 in their In It Together report and, as you can see, Cointoss Kingdom sits very close to the OECD average.

In summary, starting with each individual having the same wealth and applying a simple scheme with perfect equality of opportunity, determined only by chance — or luck, if you prefer — it is possible to produce observed levels of real wealth inequality.

But, that said, Cointoss Kingdom's distribution of wealth is quite unlike a real society because it has most people on middle wealth and almost no-one on zero or low wealth.

An interesting subsequent question, which I'll turn to in the next blog post, is can a similar luck-only model be used to produce the kind of distribution that we see in reality? And can we do it without invoking magic coins and genies?

Currency and the Venezuelan Crisis

2017-08-21T20:28:00+01:00

Before getting stuck into the main point let me say that what's happening to the people of Venezuela is nothing short of a tragedy. If you, like me, live in a stable and affluent society then I doubt we can really imagine what Venezuelans feel as their society descends into turmoil. The situation and its historical causes are not simple, and people far more knowledgeable than myself have written about it, such as here or here.

With that said, I'll focus on one aspect where I do have some understanding which I'd like to share.

You may have come across a statement like this recently. If not, let me introduce you to it:

A country with its own currency and central bank cannot go bankrupt because of large debts (held in its own currency).

This is true, but less profound than it sounds, mainly because the bit in brackets is often left out.

Now consider this:

An explorer leaving the North Pole must go south.

This is similar in that it's true and sounds profound.

Both sound interesting because they take a familiar notion but put it in a special context. The North Pole is a unique place on the globe because it's the source (or destination) of lines of longitude and so notions of north, south, east and west lose their usual meaning. Likewise, a central bank is a special place in an economy because it's a source of money. Well, base money which includes notes and coins and private bank reserves but not what you might see on your bank statement, that's broad money, but I digress.

With that in mind let's pull the first statement apart and see what it means.

For starters, what does a country going bankrupt mean? More usually it is called a 'default' and, apart from some technical differences in law, it boils down to the same thing as for a person or a company: a declaration that debts cannot be repaid on time.

But, unlike a person or a company, if a country's debts are in its own currency then it can repay them at any time by creating money at the central bank. It can do this literally by minting coins or printing paper notes, but more usually it just makes the money appear in one of the accounts it manages. In the past this was done by writing an entry in an old-fashioned ledger book but these days it'd be in a database managed by some specialised software.

Whatever the mechanics of it, the key point is that a central bank can just create base money in its own currency and then spend or distribute it via private banks as it pleases¹.

But doesn't creating money cause inflation to get out of control? In other words, if several billion pounds are created out of nothing, won't the pound be worth less in dollars or euros, and won't this cause prices to shoot up in the shops?

Not necessarily. It depends on how it is spent and the circumstances of the economy. For example, quantitative easing (QE) in the UK's central bank — the Bank of England — created £435 billion that was used to buy back chunks of government debt called bonds. This is a huge amount of money — about a quarter of the UK's public debt, and a fifth of annual GDP — but it didn't cause problematic inflation. Of course, this is an extreme and unusual example but central banks routinely do something similar on a smaller scale with what they call open market operations. Also, if you stop to think about it, a result of QE is that the UK public sector, which includes the Bank of England, owes itself a quarter of its total debt²!

If you're confused at this point, it's likely because everyday ideas of debt, and even money itself, take on counter-intuitive meanings when talking about public finances in a country with its own central bank and currency. Common sense is likely to mislead you, so it's important to think it through carefully. I've only offered a partial explanation of why in this post as I don't want to deviate too far from its main point, but if you're curious see the links in the footnotes for more detail.

Let's turn our attention to Venezuela. It has its own currency and central bank, so how come it's in trouble³? The explanation involves the bit in brackets in the statement at the start. It ran up debts priced in dollars, not in its own currency, the bolivar fuerte — the strong bolivar which replaced the previous bolivar which also fell foul of inflation. Having your own currency and central bank won't help with debts held in another country's currency. If your debt is in dollars, you must pay in dollars.

Dollars, and indeed other currencies, are also needed for trade. Venezuela relies on imports of many essential goods, including basic staples of food. To buy these things from the USA requires US dollars and so importers supplying shops go to the central bank to exchange bolivars for dollars⁴. But reserves of dollars are running low for two reasons: they're needed to pay interest on dollar denominated debt, but also the supply of dollars from oil exports dried up after the oil price crash in 2014-15. The net result of all this is that there is a shortage of goods in the shops and inflation is rampant, that is, more and more bolivars are required to buy one dollar.

So while it's undeniable that having a central bank with its own currency gives a country considerable domestic flexibility in its public finances, this advantage is limited in a world increasingly interconnected through trade. In fact, if a country has a dominant trade partner, or relies on trade in something priced in a foreign currency — oil is priced in dollars, for example — then its currency is not really its own. Such a country will be constrained by actions of the trade partner, or perhaps an international cartel such as OPEC. This blog post by France Coppola goes into much more detail on this point.

The statement at the start is worthy of serious thought, and if properly understood it can and should be used as a basis for countries to take sensible approaches to public finances especially during an economic downturn. There are actually only a few countries that can be deemed to have a currency that is really their own: most obviously, the US, the UK, Japan and China. But even for them there's an international tangle. Japan and China vie with each other for holding most of the US's debt, and the US dollar floats on waves made by decisions in oil-exporting countries, mostly in the middle east. And of course, none of the countries using the euro have their own currency except, perhaps, Germany because of its dominance in exports.

So beware of the statement made at the beginning being deployed in a glib and over-simplistic way, especially if the bracketed qualification is omitted. In such cases it becomes about as profound as

A car with its own controls and engine can't crash into other cars (if there are no other cars on the road).

^{1. Private banks can create broad money and do so whenever they lend. If you dispute that please go and read this paper published by the Bank of England and argue with them.↩}

^{2. The debt still exists technically to avoid violating the Lisbon Treaty that is the constitutional basis of the EU. Article 123 forbids repaying debt using money created at a central bank. To work around this, the Bank of England simply says it owns the bonds with the intention of selling them back to the private sector at some point in the future. If you think further on this counter-intuitive situation you'll realise that public debt is not debt at all and is actually more like a form of savings used by the private sector.↩}

^{3. A quick web search will yield many articles explaining the situation in Venezuela. This one is over a year old, but it gives one of the clearest explanations I've read of the economics of the crisis.↩}

^{4. The other option is to exchange bolivars for dollars on the black market and it is this plus bolivar-creation at the central bank that is fueling inflation.↩}

Dealing with uncertainty

2017-03-16T21:01:00+00:00

Uncertainty can be confounding when thinking rationally. After many reasoned steps forward you may arrive at a fork in the road where your imperfect knowledge of the facts prevents you from knowing the correct direction of your next step. And if it's wrong, all subsequent steps will be wrong too even if they are each executed with impeccable logic.

Twice in the 20^th century fundamental discoveries of uncertainty shocked science. First in quantum physics when it was realised that you can never be certain of all properties of a particle. For example, you cannot know both where a particle is and where it is going. And from perplexingly erratic computer models of the atmosphere came chaos theory, which said that tiny and unknowable errors in the current state of a system can lead to a myriad of very different future outcomes — the so-called butterfly effect.

Although some physicists, including Einstein, were tempted to reject quantum uncertainty, they did accept it in time, and it blossomed into new avenues of research in particle physics, mathematics and beyond.

Chaos theory also found applications in many disciplines, and even gave rise to fractal works of art such the the famous Mandelbrot set.

The Mandelbrot set. Screenshot from xaos.

Uncertainty is a fundamental fact of life that should not only be accepted but embraced. I first met it in science, but it completely wrapped its arms and tentacles around me when I took an interest in the maelstrom of politics, society and economics.

But let me start with the science.

Sunspots

Rudolf Wolf introduced a system for recording the number of sunspots in 1848 which was adopted by observatories across the world. I believe it to be the oldest international collaboration in science. Data from before 1848 is far more uncertain but is still an important part of the record.

My PhD was on analysing and predicting solar activity using neural networks and other weird and wonderful methods. In return for generous funding, I had to report my progress to the European Space Agency (ESA) at regular intervals. As a young, rather startled researcher I could manage little more than an embarrassed "dunno" at first, but I soon learned how best to present examples of my predictions with a few sentences outlining why the uncertainties were so enormous. In some cases the methods could even predict a negative number of sunspots on the Sun, which is, of course, silly.

My academic elders were patient because they understood that an essential part of a PhD — an academic rite of passage even — is in dealing with the bewildering uncertainties involved in real-world research.

My goal was to reduce the size of those uncertainties, but to do that required estimating and understanding them. Was it because the number of spots on the Sun (and other related time series) were inherently unpredictable, or was it simply that our prediction techniques were rubbish? As you might expect, it was a bit of both.

(Incidentally, I did publish a prediction in 1995 that the next solar maximum in 2000 would be smaller than the two before it, which turned out to be right, but as it's only one data point I'm not over-egging this claim.)

Solar flares

A solar flare is an event that takes place in the Sun's atmosphere. It releases a huge amount of energy across all wavelengths of light and in the form of fast particles. A typical flare lasting a few minutes could, if we had means of harnessing it, supply our civilisation's energy needs for a year. Big events release more energy than we've used in all human history. For those who like numbers, a big flare releases around 10²⁵ Joules.

In addition to being mind-bogglingly powerful, solar flares can be quite pretty.

This is an ultraviolet image of our star, the Sun — the white patch at the 10 O'clock position in the image is a solar flare (I'm afraid Hard X-ray images are much less artful). Note how the structures are reminiscent of fractals. From NASA/SDO/AIA - NASA Image of the Day, Public Domain, Link

Once I'd gained my PhD and started supervising PhD students of my own, I found myself working on making images of solar flares in hard X-rays (shortest wavelength and highest energy form of X-rays) for NASA's RHESSI space mission. The snag with this is you can't do optics for such high energy forms of light. My favourite analogy is that making an image from hard X-rays is like playing table tennis, blind-folded with an opponent that's firing bullets at you. Not only that, you have to figure out what he or she might look like from the bullets you managed to deflect with your reinforced bat.

Well, it wasn't as dangerous as that of course. Less dramatically, imaging in Hard X-rays is akin to ultrasound imaging of a foetus in the womb, or echo location used by bats. And here we return to uncertainty. The images of solar flares that we were able to reconstruct were not unique. Given the raw data from observing a particular solar flare with the RHESSI satellite, our Earth-bound computers and algorithms were capable of reconstructing many very different images.

So what to do? Shrug and give up? No, of course not. The data was telling us something, but careful work was needed. When you peer at an ultrasound image and have trouble seeing the baby, that's your brain struggling to make sense of uncertain information even with the aid of significant computer power.

For solar flares we concentrated on extracting what reliable information we could from this highly uncertain data and then used it to sleuth our way towards a better set of models of what was going on in the solar atmosphere.

The big problem in solar physics was that to produce X-rays in the quantities seen, the models required impossibly large amounts of energy in the Sun's atmosphere. In other words, there was a big energy budget deficit.

Couldn't we just blame our very uncertain data? No, because we knew what the uncertainties were from well-proven statistical theory and we could do sense checks against information from other observations of the Sun. The bottom line was that we were forced to accept the data was good to within measurable uncertainties and improve the models.

This is fine. Every good scientist should seek to falsify models with data. Especially their own. That's how you improve models and progress science. Of course, scientists who developed them may well be unhappy if new data demolishes their life's work, but many will take it on the chin and see the bigger picture. Even if they don't, their younger peers will in time. Science proceeds one funeral at a time, as physicist Max Planck once said (well, almost said in more long-winded German prose).

Society

I left my science career behind many years ago. I won't go into the details on why here, except to say that leaving a secure and successful career was not a trivial decision, and that despite great uncertainty about my own future, I felt compelled to venture out and explore a wider world with the skills I'd picked up.

These days I'm studying all kinds of public data on tax, spending and the economy. I was motivated initially by the 2008 financial crisis but more recently by upheavals in British politics. The uncertainties here, such as on tax receipts or GDP, are usually no more than a few percent. In astronomy, it was not uncommon in a new line of research to be faced with "order of magnitude" uncertainties and to an astronomer that means a factor of ten. You read that correctly. We had to work with numbers that might be a factor of ten too small or too large, or if you prefer percentages ±1000%.

So coming from my astronomical perspective, public data on public spending, tax and the economy are blissfully accurate. Well, the numbers are, but the uncertainty extends beyond accuracy of numbers.

Economic data would be meaningless without models of how it relates to the society around us and the history that's brought us to this point. And of course there are ideas to consider for the future, such as basic income, a job guarantee and new forms of credit and money that may reinvent that old bugbear of society, debt.

In other words, unlike astronomy and physical sciences, data on our society cannot be separated from ideas about what our society should be like and the spectrum of moral values of its people. If you are libertarian, you will view current public spending as too high; to a socialist, it will seem too low.

There's no economic model or paradigm of society and politics that fits the data. But as the data is uncertain, it's tempting for us humans — even highly intelligent and educated ones — to twist the data, disguise aspects we don't like with complexity, and even ignore particularly awkward bits, all to make a favourite model or ideology fit. But that would be fooling yourself, and if you are a persuasive communicator, you might fool others too, at least for a while until reality clobbers you.

But not only is our knowledge of reality uncertain, a society, just like the individuals that form it, is quite capable of behaving irrationally and might react to similar circumstances in surprisingly different ways at different times. Even a well-intentioned government that takes an evidenced-based approach to policy cannot be sure that its actions will fulfill its aims.

Bertrand Russell probably best explained how we should deal with uncertainty:

I think nobody should be certain of anything. If you're certain, you're certainly wrong because nothing deserves certainty. So one ought to hold all one's beliefs with a certain element of doubt, and one ought to be able to act vigorously in spite of the doubt... One has in practical life to act upon probabilities, and what I should look to philosophy to do is to encourage people to act with vigour without complete certainty.

I believe it should be the task of all citizens to try and think through the reality of their society and face up to uncertainty without letting it cloud what can be established with some certainty. If you don't attempt to do this then you're effectively placing blind trust in your peers, or worse still, in your government. Either way, you may well regret it when they begin acting with vigour in a way that you do not like.

My father

2017-02-11T17:22:00+00:00

These are the words I spoke at my father's funeral. I've altered them very slightly, mainly to remove names of family members.

My father was a kind and gentle man. He was a man of few words but words that mattered.

If you stand in the sitting room of his house — our former family home — you would notice that one wall is covered with bookcases, and most of the books are either about history or historical novels. On the opposite wall hangs a Persian carpet and scattered about the room are other beautiful objects chosen by my mother. There are also shelves of DVDs, and laser discs, and yet more CDs and LPs of classical music. Following my mother's death, my father filled the room with photographs of her and their children and grandchildren.

This collection, which my sister and I will soon be tasked with dismantling, says more about my parents than I realised.

My father rarely gave advice but he set a good example to follow. He was polite and courteous in his dealings with people, perhaps anachronistically formal at times, but that didn't prevent him from being straightforward and honest. He disliked confrontation but would stand his ground, not to appear tough or strong or for gain, but because it doesn’t occur to a person of sincerity and integrity to do otherwise.

I can really only think of one piece of direct advice he ever gave me: don’t write text out by hand and then enter into a computer; learn to type directly into the computer. This doesn’t seem contentious now but was far from received wisdom when he told it to his 10 year old son in the early 1980s.

Although not advice as such, more an observation, I also remember him saying that a home should be primarily treated as just that: a home. Not as an investment nor an asset. This again seemed remarkably prescient as he said it a decade before the 2008 financial crisis.

He was modest but could sometimes surprise you. One day while driving me to school, he mentioned in passing about how he had discovered a Helium 3 nucleus with something called a lambda particle inside it. This was during his PhD on particle physics. His interest in the bizarre quantum world was also expressed by his liking of surreal and non-sequitur humour such as the Goons and Monty Python.

My wife recalls my father's welcome to the family. “Welcome” he said awkwardly to her at our wedding. I half expected my brother-in-law to get a Vulcan salute on his wedding day.

He didn’t wear his heart on his sleeve but that it was a heart with much love to give was never in doubt to those closest to him.

His inhibition at expressing his feelings in words made the 400 miles between him and his family in London an even greater emotional distance. But it is clear to me that he loved his mother, and his brother, and the premature death of his father in a road accident in the 1960s was a profound and quite literally unspeakable loss.

Our family home in Glasgow was a very happy one. Like all families we had our ups and downs. When I was upset, he’d come into the room, say little, but give me a hug. He didn't attempt to help solve my problems but his calming influence allowed me to see what needed done with more clarity.

My son is remarkably similar to my father in character. One time I took them both out for lunch and realised how similar they were: both people of few words — I did all the talking. Yet, there was always something companionable about their silence.

In 2010 my father was diagnosed with cancer and endured the treatment that removed it with real stoicism. His recovery owes much to the care he received from my mother and I do not think it is an exaggeration to say that doing so cost my mother her life. It was her nature to love and care no matter what. That was why he loved her, and that was why he lost her. This tragedy wounded him deeply but he still approached life philosophically, declaring that, whether he liked it or not, he must now embrace the third and final phase of his life.

He did well for a couple of years following my mother's death in 2011, but when his health deteriorated further we were fortunate to have found Margaret, a private carer he employed but who provided care in the proper sense of that word.

Margaret took care of him for most of the last two years until she fell ill with pancreatic cancer in November and died before Christmas. I then chose to care for him myself, and although that was a strain for us both at times, and I struggled to meet the high bar set by my predecessors, I am glad I did it because I got to spend every day with my father for the last 3 months of his life, with the exception of two days when my sister kindly volunteered time from her busy life to relieve me so I could spend a weekend gazing at stars in the highlands.

My daughter only knew her papa when he was — to use his own words — in a decrepit state. But she didn’t see that, and was not even slightly phased by the lop-sidedness of his face caused by the major cancer surgery. She just threw her arms around him in love and he would fold her in his. This wee flame of love was precious to us all in his final years.

My father was a kind and gentle and loving man. If more people behaved as he did, our world would be the better for it. I will miss him.

Anthony Patrick Conway 1941-2017

Music

Here are the pieces of music played at the funeral. The second one was unknown to him and was chosen by my son. The first and third pieces were favourites of his and chosen by me with advice from my father's brother and my wife.

As a boy I remember asking my father how Vaughan Williams gave his pieces of music such gravitas (not the word I'd have used then). His explanation involved a scale called a Dorian mode. Remembering this, I did a bit of reading around whilst mulling over which pieces to use and found that the Fantasia on a Theme by Thomas Tallis and Glass's Evening Song both use something called the Phyrigian mode. Not only that, but it is also used in Rimsky Korsakov's Scheherezade which my father picked for my mother's funeral.

Why did the economy grow post-Brexit?

2017-01-11T21:03:00+00:00

Note: The main point of this post is on sketching a simple and plausible reason why growth held up in 2016 despite uncertainty from the result of the June 2016 Brexit vote together with the ensuing speculation on when (or if) Article 50 would be triggered. This is quite distinct from longer term analysis such as that made by the Treasury on differences to the economy a decade or so after the UK has left the EU. Thanks to Peter Shaw for alerting me to this potential confusion.

Some economists, politicians and other commentators warned we might see negative short-term consequences of Brexit by the end of 2016. And the reasoning was perfectly sensible, going something like this:

Uncertainty will knock business confidence slowing investment and prompting cost cutting on wage bills.
And that will fuel loss of consumer confidence as people become nervous of the future and their job security.
Which will in turn cause them to slow their spending and save more: a car purchase is postponed, a holiday skipped, a tired kitchen lived with for another year and suchlike.
This lack of spending erodes business profits... go to step 1.

In this way a vicious cycle can be established that slows the economy, possibly into recession.

But this did not happen in 2016. Why not?

There's a simple reason which was so obvious I didn't realise it until recently: people who are worried by Brexit are in a minority. Roughly speaking, a third of the electorate in the UK voted Leave, a third voted Remain and a third did not vote. So about two thirds of the electorate (65.3% if you want precision*) were either in favour of Brexit or weren't sufficiently motivated to vote against it. This is a plausible explanation of why a majority of the population just carried on as normal.

Although a defence of the statement "non-voters don't matter" can just about be made in the context of an election or referendum, it's clear they do matter when it comes to the economy: non-voters may not feed the ballot box but they have to feed themselves and are consumers in the shops.

And while there are some very vocal individuals railing against Brexit, it's a fair bet that many Remain voters have not altered their lifestyles very much. I voted Remain but I don't believe Brexit will necessarily be a disaster, though my imagination is stretched in seeing it as a positive development. I carried on living my life much as I did before. In fact, I've brought forward big ticket purchases as I believe prices will rise due to the pound devaluing post-Brexit. (This provides another possible explanation for recent economic growth, and is touched on in the FT article on consumer spending mentioned below.)

The same argument works in Scotland too. Although the Leave vote percentage was lower than in the rest of the UK, the proportion of non-voters was higher, so there's still a clear majority of the electorate (precisely 58.3%) are either in favour of Brexit or insufficiently motivated to vote against it.

Scotland's slower economic growth relative to the UK is almost certainly due to the oil price shock and nothing to do with Brexit. There's some good evidence for this which I'll set out in another post. But in brief, according to the Scottish Government's Quarterly National Accounts (see table C), Scotland's economic divergence began in early 2015, just after the oil price shock hit. Also, the FAI/SCER's Labour Market report shows that unemployment in the Aberdeen area — the hub of Scotland's service industry to North Sea oil and gas — has now risen to near 2009 peak levels.

Across the whole UK, it's also clear that economic growth is being driven by consumer spending which is in good part enabled by a growth in credit, i.e. people are taking out more loans. We are now heading back to 2007 private credit levels and that should be a real cause of concern.

And if you're not willing to take my word for it, then listen to the Governor of the Bank of England, Mark Carney, who has just said he doesn't see Brexit as the biggest risk to the UK economy; instead he cites financial instability and external (i.e. foreign) factors as his main concerns.

So while the longer term warnings of Brexit's impact on our economy may still come to pass — remember, we've not yet left the EU — it may well be swamped by bigger issues that extend across Europe and the rest of the world. I doubt the Euro monetary union, and perhaps even the EU itself, could survive another crisis. The memory of 2008 and the fact that interest rates remain stuck near zero means we won't simply see a repeat of that crisis. Predicting when it might occur or what form it will take is a mug's game, but we can anticipate that a troubled time lies ahead, prepare for it and perhaps even address its causes. But it seems our governments — both central and devolved — along with most politicians and commentators are fixated with Brexit and missing the bigger and much more worrying picture.

Post-Trump thoughts

2016-11-16T21:43:00+00:00

The pollsters were wrong about Trump. This should not have come as too much of as a surprise. I hoped the predictions showing he would lose were correct, but following the 2015 UK General Election and the Brexit vote I was wary of slim margins in times of unusual political movements, especially with contorted electoral systems.

The examples of Trump, Syriza, Brexit, Beppe Grillo's Five Star, the SNP, Corbyn, Podemos and others tell us that electorates are searching for alternatives to established politics. That there are multiple causes behind this that vary from country to country is not in doubt. The timing, however, cannot simply be a coincidence.

I've recently abandoned the belief that politics driven by democratic mass voting will inevitably improve society. I protected the belief for some time with exceptions such "this government is not improving things but this will be only a 4 or 5 year departure" or "this event is inarguably bad but will shake things up and lead to something better". Of course, these can be true and mass democratic voting can drive progress; what I am recanting is my belief in the inevitably of political progress.

As John Gray observed in his introduction to Gray's Anatomy:

Progress in science and technology is a fact, whereas progress in ethics and politics is a fiction.

Yet, hasn't worldwide inequality improved markedly in the last few decades with significant falls in absolute poverty rates? Yes, this is true. But it's not clear this was a deliberate political outcome and, even if it was, it's certainly not one stemming from mass voting. Voters in Europe or the USA never voted for anyone who promised to increase prosperity in China, and voters in China effectively have only one choice.

The change in world inequality came about over decades as labour costs rose in the West whilst people came to expect ever-improving standards of living. And this was coupled with growing world trade driven by technological progress, notably the huge efficiencies of containerised shipping along with advances in air transport and, to a lesser extent, near-instantaneous electronic communication. The net result was that manufacturing moved from West to East leaving an industrial employment hole that has been unsatisfactorily filled with jobs in services, many of which are low paid and precarious.

To a large extent the decline of Western prosperity, from which the wealthiest mostly insulated themselves, was hidden from view by the financial system. The large but invisible bubble of private credit burst in 2007-08 and the debt was shifted into the public sector. Economies such as the US or UK with their own currency and trusted bonds could roll with that punch, but smaller countries, particularly those with importing economies constrained by the Euro endured its full force. Rather than spreading and softening the impact, EU fiscal rules concentrated it on countries with the weakest economies.

Recovery from the 2008-09 recession was atypical by historical standards. There was no growth spurt making up lost ground, instead growth returned in slow and faltering style, with wages falling or remaining stagnant. The details vary from country to country but broad patterns are similar.

And while this happened, people noticed that wealth was continuing to pile up with the top 1% (or 0.1%), and this inequity caused an anger that registered across the left-to-right spectrum of politics. In fact, this is one reason amongst many why the concepts of left and right now seem out-dated.

But we should also realise that most of us in western countries have recently lost only some of the gains from decades of growth, and that we are still relatively very wealthy compared to the workers in China and many other countries. That we have exported our jobs to such places and we import so many of our goods from them should be a strong clue as to what has happened.

In essence, worldwide growth has continued with an increasing share going to poorer countries and less going to low and middle earners in the West. Changes in global trade have also created new international flows of money. The very wealthiest people in the world can straddle national boundaries with ease and so they have adapted their finances to capture a share of these flows in a way that is near impossible for country-bound people on low and middling incomes.

Of course, we humans are not rational. Our instinct is to compare ourselves with those around us, especially those wealthier than ourselves. Also, human nature means that we notice only the relatively sudden recent fall in our fortunes and take for granted the prior, more slowly accrued gains. The result is that the partial but true account of losing out to the 1% has taken hold and obscured the full story of what has happened.

Separate to this is the apparent puzzle of why voters in different countries have reacted so differently. How can global economic anxiety cause one voter to turn xenophobic in north England and vote for Brexit or UKIP, but another voter a hundred or so miles to the north in Scotland vote remain or for the SNP?

Of course, the question is on an absurd premise: economic anxiety via a fall in perceived income or wealth did not cause people to become xenophobic or to support the SNP. Instead, the campaigning in the referendum and elections served to bolster pre-existing views. There's clear evidence that an authoritarian predisposition lay behind anti-EU feeling in England, and that although underlying attitudes to the EU are not that different in Scotland, it's electoral expression was altered by the SNP's vision of independence in Europe.

So the rise of non-mainstream politics in western countries is undoubtedly related to the change in the worldwide economic climate, but manifests itself in each country according to pre-existing local politics. But there is a common element behind all the movements listed at the start: blaming some other, whether it is another political ideology (neoliberalism), another party (Tories), another country (Mexico or China or Eastern Europe), another political power-centre (Westminster or Brussels), or in its ugliest form, another race. In some cases there is an element of truth in blaming the particular other, in others there is none whatsoever.

Several times I've heard the fall-out of Trump's election or Brexit described as a kind of darkness falling across western society and although that is a tad dramatic, it feels apt to me, especially in describing the xenophobic and racist elements. However, it is too simplistic to blame the establishment for this, or to summarise it away as neoliberals getting their comeuppance. Whatever else they did wrong, this darkness did not come directly from present or recently governing politicians. They do however deserve indirect blame insofar as they unwittingly sowed the seeds of this situation, and then upon it becoming manifest, denied its existence. This latter failure allowed it to grow and fester thanks to vote-hungry populist politicians and certain amoral sales-hungry newspaper owners who peddled rhetoric based on identity and grievance.

But although unscrupulous politicians and media outlets exploited it, they were not the source of the darkness either. It came from within society itself. It came from the humans that we live and work with. These people, ones who harbour racist and xenophobic views, are a minority but they have never been classed as one of the minorities, and have felt silenced for years, both in what they could say in public and what they could vote for via the ballot box. Many who were attuned to the liberal mainstream of politics expressed surprise and horror that Britain had become xenophobic and perhaps even racist overnight. There is some hyperbole on this point for sure, but Britain had this element in its society all along. Until recently, it remained quiescent. Now it has found a means of expression that not only enables it to be heard, but has allowed it to influence the UK's government.

And of course, there is no reason to suspect this situation is unique to Britain or the US. Nor is past electoral success from seemingly left or centrist parties (Syriza, Podemos or the SNP) a guarantee against such darkness emerging in the future. The fall in the fortunes of a governing party can lead to a very different party or movement gaining power or influence. Both Brexit and Trump's victories were narrow but their effects were anything but.

What can be done? Trying to understand why people voted for Trump or Brexit is worthwhile, but only so much can be learned from listening to unshakeable beliefs and irrational fears. And fighting irrationality with reason is futile. If that rankles, you are in good company: Bertrand Russell entertained such a notion until it was corrected by John Maynard Keynes and others.

But it's important to realise that there are people who voted for Brexit and Trump who did apply reason and think through their decision. But to brand them as irrational would be a mistake. We have to live in a society with other people whose values we do not share. Telling them their values are invalid or wrong will not help, nor will telling them to shut up. What can be done is to limit the actions of a minority and its expression in political power. That, of course, can work both ways.

It's likely that there's more darkness to come. When it crosses thresholds of households and affects daily lives then those who have chosen irrationally will be forced to reason with reality. Then, I think, or at least hope, matters can improve.

Kelly criterion and statistical weirdness

2016-11-04T19:53:00+00:00

I was prompted to write this after happening across this fascinating article in the Financial Times. Unhappily it's behind a paywall so I'll recount the gist of it here.

Imagine you had a coin that had a 60% or 0.6 chance of tossing to give heads and a 40% or 0.4 chance of tails. Now imagine you have $25 to gamble and you can choose to bet all or some of this amount on each toss. If you stake $20 and lose you're down $20, but you'd be up by $20 if you win. You then get to do it again with the $5 or $45 you'll have for the next toss, and so on.

The question is: what's the best strategy for maximising your winnings after a given number of tosses? Rather than idly muse on this, have a go at it on this website's simulator. Alas, there's no longer actual money winnings on offer but there is still an intellectual reward.

I recommend you do it now before reading further.

There now follows a brief intermission in this blog post.

How did you do? I did badly when I first tried it, but my heart, and more importantly, mind, were in the right place. I only bet on heads and concentrated my thoughts on what fraction of my current pot I should stake on the next toss. I didn't last the full half hour because a) I dropped down to a few dollars, and b) was mentally salivating at the prospect of resolving my confusion with mathematical statistics.

Yes, I know. A bit sad.

Sans maths

I'll start with a non-mathematical account.

Firstly, here's the answer to the question above: the best proportion of your pot to bet is a 20% or 0.2. This comes from the fact that the probability of getting a heads is 0.6. You double 0.6 to get 1.2 then subtract 1, which gives 0.2.

What's the significance of the double and subtract one thing?

Well, that's a little hard to explain. To get an idea imagine you have a $1 stake. There's 0.6 chance of being up $1 and a 0.4 chance of being down $1 after the toss. So you might expect, in some kind of average sense (averaging over many parallel Universe where this toss took place!), to be up $0.20. And the same applies for any stake. Bet X dollars and you should expect, on average, to be 0.2 times X dollars up as a result.

You can also sense check this result for a fair coin — one with a 0.5 probability of a heads (or tails). Two times 0.5 minus 1 is zero, so you would expect, on average, to be up (or down) by zero dollars on each toss.

However, this only explains why the 0.2 probability is significant. It doesn't explain why you should use that as the fraction of your pot to maximise your winnings.

Alas, I cannot think of an intuitive explanation for that. Time for maths I'm afraid.

Go maths!

First, let the probability of getting heads be $p$ and, as we're considering an unfair coin, let it be greater than 0.5. Also, let's suppose we bet on Heads on every toss (because we're not stupid).

Let the starting amount in the pot be $z_0$ and suppose we bet a fraction $a$ of whatever is in the pot for every toss. Then, using caps to denote a random variable, the amount in the pot after the first toss is:

$$ Z_1 = z_0 + a z_0 T_1 $$

where $T_1$ is the random variable representing the first toss. Being a random variable we cannot say which value it will have until the result of the toss is known. Beforehand all we can say is that $T_1$ has a probability $p$ of being +1 if Heads comes up (a win!) and -1 if it's Tails.

We can re-write the above equation as

$$ Z_1 = z_0 (1 + a T_1) $$

and for $Z_2$ we will have

$$ Z_2 = Z_1 (1 + a T_2) $$

where $T_2$ represents the second toss. It is defined exactly like $T_1$ but it is statistically independent of $T_1$ and for that matter any other $T_n$. In other words, the outcome of one toss cannot affect any other.

At this point note that we've multiplied two random variables together. We'll return to the significance of this later, but for now I'll let Robby the Robot pass comment:

Combining the last two equations gives:

$$ Z_2 = z_0 (1 + a T_1)(1 + a T_2) $$

and we can extend this to any $n$ by doing this over and over again to obtain:

\begin{equation} Z_n = z_0 (1 + a T_1)(1 + a T_2) \cdots (1 + a T_{n-1}) (1 + a T_n) \label{Zn} \end{equation}

Now this is all fine and dandy, but random variables cannot be directly compared with real world results. To do that we need to use the expectation operator. It will extract the mean (more colloquially known as the average) from the random variable.

Let's start with $T_n$. To find its expectation we multiply each outcome by its probability and add all of these up, thusly:

$$ E[T_n] = p\cdot 1 + (1-p)\cdot (-1) = 2p-1 $$

and voila, it's the first appearance of the mystical probability $2p-1$. This is such an important probability in these workings that we'll call it $q$ to make things a bit neater, i.e.

$$ q=2p-1 $$

We'll need the expectation of $1+aT_n$ which is simple to work out because neither 1 nor $a$ is a random variable, so:

\begin{equation} E[1+aT_n] = 1 + a E[T_n] = 1 + aq \label{expect} \end{equation}

Next we make use of the fact that each $T_n$ is statistically independent from the rest. If we have two independent random variables $U$ and $V$ then the expectation of the product is the product of the expectations: $E[UV]=E[U]\cdot E[V]$.

Each term in brackets in our expression for $Z_n$ in (\ref{Zn}) is an independent random variable so

$$ E[Z_n] = z_0 E[1 + a T_1]\cdot E[1 + a T_2] \cdots E[1 + a T_{n-1}]\cdot E[1 + a T_n] $$

and because (\ref{expect}) tells us all the expectations are $1+aq$, we have our result

$$ E[Z_n] = z_0(1 + aq)^n $$

To state this in English: the expected result of making $n$ bets on heads with fraction $a$ of the pot staked each time results in us multiplying our starting pot by a factor of $1+aq$ raised to the power $n$.

This can be very large. Clearly it's maximum if we set $a=1$, and for 20 tosses of a coin with a 0.6 probability of heads we can expect to multiply our starting pot by 1.2 raised to the power of 20, which is 38. That's a pretty amazing return.

But there's a problem. Setting $a=1$ means you're betting your entire pot on heads on each coin toss. If there's just one tails at any point, your pot is emptied and it's game over. How likely is it that you'll get through 20 coin tosses without a single tails? Not very.

Have we made a mistake in the Maths? Nope. Is one of the statistical steps wrong? Nope. The problem is that $Z_n$ is a random variable with unintuitive properties. In other words, our expectation is correct but useless for our stated aim: finding the best value of $a$ for maximising our winnings. We should've heeded Robby the Robot's warning about multiplying random variables.

The Central Limit Theorem to the rescue

So our random variable $Z_n$ is weird. Specifically its distribution is weird. In fact, $T_n$ is also a bit weird if you think about it. It can only ever take on two values -1 or +1 and, for $p=0.6$, the expectation operator tells us to expect a value of 0.2 — a value that $T_n$ can never have.

The working above is all ruthlessly logical and correct, but we got into trouble because we didn't just start with a weird distribution but we compounded it $n$ times to produce a weirder distribution for $Z_n$. It's still possible to work in this intuition-free zone and compute probabilities of winning over a certain amount but to do so would be climbing the north face of mount statistics.

There's a beautiful thing in mathematical statistics called the Central Limit Theorem. It says that if you add together many random variables that have the same distribution then the resulting random variable will end up approximating a normal distribution. The term "normal" is somewhat unfortunate as it is the name of the distribution not a description of it. The normal distribution is also known as a gaussian, or more colloquially as the bell curve because it looks like a bell when graphed. The Central Limit Theorem applies even if the individual distributions are not bell curves themselves and, with some conditions, even if the individual distributions are different.

In our case the distributions of the random variables $T_n$ are all identical, but the problem is that we multiply them together to form $Z_n$. We can enlist the help of the trusty logarithm function to turn a product of random variables into a sum because $\log UV = \log U + \log V$. Doing this gives

$$ \log Z_n = \log z_0 +\log (1 + a T_1)+ \log (1 + a T_2) + \cdots + \log (1 + a T_n) $$

Next we need to compute the expectation of each term in the sum:

$$ E[\log (1+ a T_n)] = p \log (1+a) + (1-p) \log (1-a) $$

As before, there are two outcomes with $T_n$ being +1 with probability $p$ and -1 with probability $1-p$. As this expectation is the same for any $n$, we find:

$$ E[\log Z_n] = \log z_0 + n E[\log (1+ a T_n)] $$

which leads us to this

$$ E[\log Z_n] = \log z_0 + n \left( p \log (1+a) + (1-p) \log (1-a) \right) $$

For $a=0$ (never bet anything) this is just $\log z_0$. As $a$ approaches 1, the expectation decreases without limit towards $-\infty$. This means $Z_n$ approaches zero and accords with our observation above that betting the entire pot will be game-over when the first tails is thrown.

So somewhere in between these two useless extremes is our desired best value of $a$. To find it we need to take the derivative with respect to $a$ and find the value of $a$ which makes it zero. So,

$$ \frac{d}{da} E[\log Z_n] = n \left( p \frac{1}{1+a} - (1-p) \frac{1}{1-a} \right) $$

and with a little bit of algebra that I leave as an exercise to you, dear reader, this is zero when

$$ a=2p-1=q $$

QED.

I'm not going to do it here but it's not too difficult to show that as $n$ increases, the spread of values about the expected value becomes proportionately smaller.

In other words, the laws of probability and large number statistics say that not only will your winnings increase with more and more tosses but you can be be relatively more certain of how much you will win. How nice.

Deficit identities

2016-10-12T21:49:00+01:00

It's my birthday today and as a treat, I've turned off my phone and am ignoring emails and social media for a short time so I can write up some stuff on macroeconomic identities. Yes, I know it sounds a bit sad, but it's what makes me happy.

In this post I want to derive an identity that relates two deficits. An identity is an equation that has to be true by definition. As it makes no assumptions and doesn't build on any theory, it provides a solid starting point for understanding an economy.

Let's start with the macroeconomic identity for GDP expressed by types of expenditure:

\begin{equation} Y = C + G + I + X - M \label{income-exp} \end{equation}

It's worth taking a moment to be clear on the definitions of the expenditures involved:

$Y$ is GDP — the total expenditure on all new products created, where products can be goods or services.
$C$ is consumption — spending by all households.
$G$ is government spending — excludes welfare payments for pensions and benefits.
$I$ is investment — investment is spending by businesses excluding that on wages plus the value of all stock waiting to be sold (Note: macroeconomic investment has a meaning separate from the everyday one associated with buying shares).
$X - M$ is net trade — money received from exports less money spent on imports — the trade deficit is the negative of this.

What this identity tells us is that GDP is the sum of all these types of spending into the economy. But, since one person's spending has to be another's income, GDP $Y$ can also be understood as the total of everyone's income. Also, it's important to remember that all these quantities are totals over a given time period, usually a year.

We can write another identity that expresses GDP as a total of different kinds of income:

\begin{equation} Y = W + P + V \label{income-use} \end{equation}

where $W$ represents monies paid to employees for their time spent working (i.e. wages, also known as compensation of employees in national accounts), $P$ represents gross profits (referred to as Gross Operating Surplus) and $V$ represents taxes on production and products less any subsidies from government (this is mainly VAT, but includes a number of less well-known taxes too).

You can think of this identity as the sum of employee income, business income and government income. Notice that employees don't get all of $W$ as income because some goes to paying tax, mainly income tax and national insurance. Likewise, business pays some of $P$ to the government as corporation tax. As such, $V$ is not total government income, just that from taxes on products and production.

Let's now focus on the income of the private sector.

$W$ and $P$ represent incomes to the private sector from supplying new goods and services in the economy, but there is another source of income that we'll call transfers that are not associated directly with any productive activity.

The bulk of transfers come from the government in the form of welfare payments such as pensions and various kinds of benefits, but they also include monies from interest payments from the public sector, notably government bonds. We'll denote transfers from the public sector by $U$.

There will also be income from returns on assets held overseas and returns to foreign-owned assets going in the other direction. Let's denote the net income from this (which may be negative) by $Z$.

We can now write an identity that expresses the balance of the private sector:

\begin{equation} W + P + U + Z = C + S + T \label{private-sector} \end{equation}

The left hand side is the total of all income to the private sector and the right hand side represents what the private sector can do with that income. Some of the income must go to the government as tax $T$, mainly income tax, national insurance and corporation tax and the remainder is either spent as consumption $C$ or kept as "savings" $S$. The colloquial notion of savings can easily mislead at this point, so I prefer to think of $S$ as just the income that is not-consumed.

Using equations (\ref{income-use}) and (\ref{private-sector}) we can express GDP as

\begin{equation} Y = C + S + T - U - Z + V \label{income-gdp} \end{equation}

and then equating the right hand sides of (\ref{income-exp}) and (\ref{income-gdp}) we have

\begin{equation} C + G + I + X - M = C + S + T - U - Z + V \label{balance-int} \end{equation}

and this can be simplified to give

\begin{equation} (G + U - T - V) + (I - S) + (Z + X - M) = 0 \label{balance-final} \end{equation}

As with all economics identities, (\ref{balance-final}) is a statement of the fact that if someone parts with money, someone else must receive it (even if either party doesn't realise it!).

However, this identity does allow us to understand how the public, private and foreign sectors are entwined and offers a rock-solid understanding of various deficits.

The first term in brackets is the deficit of the public sector, more usually referred to as the fiscal or budget deficit. It is the total of money leaving the public sector $G+U$, less revenue received $T+V$. This is the most famous deficit.

The second term could be called the deficit of the private sector, but since private sectors are rarely in deficit, $S-I$ is more usually referred to as net private sector savings. Again, this is somewhat at odds with the everyday notion of savings.

The third term is the current account balance — monies received from abroad less the amount we send abroad. The $X-M$ bit is the trade balance (or surplus) and its negative is referred to as the trade deficit.

So what?

I set myself a time limit to batter this out into a blog post and my time is nearly up so there's no time for me to bring this to an elegant conclusion. Perhaps I'll write a follow-up blog post.

But if you'd like to get your thoughts flowing towards an interesting conclusion, then consider the situation of the UK just now: the current account balance $Z+X-M$ is well into the negative and the private sector is about breaking even, i.e. $I-S$ is small or close to zero. Given this, what must the fiscal deficit be according to (\ref{balance-final})? I'll leave this as an exercise for the reader, but you can see a visual answer in this post on Neil Wilson's 3spoken blog.

Right, I'm off for a wee dram and I'll toast you, dear reader, for getting this far... :)

Mind and body

2016-09-16T11:43:00+01:00

I get my heart pumping at 150+ beats per minute three or four times a week in 45 minute sessions. An obvious benefit is that I generally feel fit and healthy, but that's not my main motivation: regular exercise helps keep my mental state in balance.

Immediately after exercising my mind feels clearer and my mood feels more positive. This is caused by endorphin production in my body. The word "endorphin", which is short for endogenous morphine, refers to the fact that morphine-like chemicals are produced inside the body. I could obtain the pleasant mental effects by taking opiate drugs but I'd lack the the satisfaction and physical benefits that come with physical exercise. Also, it's clear that even after the endorphins have left my system, which they certainly have by the following day, that my mood and mind remain in an improved state.

Another aspect is physical challenge. This can be pushing yourself in physical endurance to run 10 km or to cover 1 km in 5 minutes, but there's also an exhilaration in facing danger, of which there was a little when I ascended to the top of Helvellyn via a scramble up Swirral edge, as seen in the photo below.

The ascent up the lower slopes was on an easy path and I was so engrossed in chatting to my friend (who took the picture) that I hadn't noticed that the terrain around me had changed. Not only would losing my footing result in a long fall, but the steepening path disappeared ahead into a pile of boulders and rocks. I wouldn't describe it as more than a clamber, and certainly no climbing equipment was required, but a level-head and sure-footedness became essential. We encountered a couple of guys who were on their way down. One was bandaging the hand of the other who had stumbled and grabbed hold of a sharp rock: better to cut your hand open than have your skull cracked like a nut after a 100 m fall.

All forms of exercise have taught me much about coordination and body awareness. These aspects had an immediacy on Swirral Edge, but it is from swimming that I have learned most about them.

When I started attending the local gym one of the instructors — let's call him George, because that's his name — came up to me and said he knew me from somewhere. Soon we realised that I'd taught him about spectroscopes and atomic physics at an Open University summer school five years or so before. In a nice piece of reciprocity, he then agreed to teach me how to swim. Actually, I already could swim, but my front crawl technique was so inefficient that I could barely swim a couple of lengths without being hopelessly out of breath.

George taught me how to position my body in the water. He approached it in two ways. First, he drew my attention to the physics: to swim efficiently you must minimise friction and reduce the surface area you present in the direction of motion. This means that your body should be horizontal in the water with your legs kept just below the surface.

Second, he instructed me on how to achieve this: I had to keep my head beneath the water. It's hard, if not impossible, to keep yourself horizontal in the water by consciously trying to lift your legs. Any attempt to do so results in strange contortions involving the hips and torso that can't last for more than a couple of strokes. Pushing my head beneath the water however did pivot my body about my centre of mass thus making it natural to keep my feet high.

It took a bit of practice, especially in timing my breaths, but keeping horizontal turned out to be most of the battle in learning to swim efficiently. Next time your at a pool, watch the swimmers. Ones with poor technique, whether front crawl or breast stroke, will have their head out of the water and legs dangling down behind them. They'll either move very slowly or be unable to swim more than a length or two at a time. Watching people getting it wrong needn't involve smug gloating, but can really help understand one's own mistakes.

The other thing I learned is that leg kicking is in itself an inefficient way to propel yourself through the water. The hefty muscles in thighs and calves are good at moving legs up and down in the presence of gravity on land, but will quickly drain your energy if used too vigorously in the water. For an efficient front crawl you must use your arms to pull yourself forward whilst keeping the legs fairly straight and moving mostly to balance the rocking generated by your upper-body movements.

The above descriptions make the learning process sound more conscious than it actually was. I learned an efficient front crawl through practice and it took hundreds of lengths before it became instinctive. At times, when the rhythm is right, I feel like I'm flying, and there have been moments where I felt that I'm more at home under the water. Is this me tapping into some deep primal instinct rooted in an evolutionary ancient part of my brain?

What's clear to me is that body and mind are entwined and that improvement of one is limited by the state of the other. Certainly, one can have a flabby mind and a fit body, or vice versa, but I feel it's best to invest time in improving both.

Brexit — correlations, causes and philosophy

2016-06-28T10:26:00+01:00

I shared some thoughts on why I voted remain, then mused on the result and the possibility that brexit might not happen, but now I'd like to try and understand why people in the UK voted to leave in such large numbers.

There are a variety of simple stories: In large swathes of the poorest parts of England and Wales it was a vote against the establishment from people that feel neglected and unrepresented. It's a reflection of rising inequality in society. It expresses a desire to "take back control" from the EU. It was driven by fear of immigration laced with racism.

Naturally there's some truth to all of these, but how much truth and what evidence is there?

It's well worth looking at the work of John Harris as he talked to people around the country. Yes, it's vox pops and anecdotal, but John's approach reveals attitudes that don't come out so clearly in more traditional reporting, and that cannot be captured in the aggregate numbers of most opinion polls.

There are two prominent quotes in that article:

If you've got money, you vote in ... if you haven't got money, you vote out

said by a woman in Collyhurst, near Manchester. Also, the strap line of that article is:

brexit is about more than the EU: it’s about class, inequality, and voters feeling excluded from politics.

Now here's the problem. From John's interviews it's quite clear that these sentiments are strong, but can we assume that what people say made them vote leave is the real reason, or the whole reason? I'm not accusing them of lying, nor am I patronising. I ask the same question of myself all the time: why have I come to that conclusion? Have I found an after-the-fact rationalisation of it that I like to tell others, and myself, or is there some deeper reason? Is it consistent with other things I (like to) believe? Is it what my peers think? Is it some influence from my parents?

I'm sure psychologists would tell you that this is not a straightforward issue. Thankfully, they've found a way to tackle it: don't ask people direct questions on a single issue but probe more widely around their attitudes and values.

Let's now look at two sources of evidence.

Exhibit A — income and inequality

This Resolution Foundation blog post looks at the change in income and how it is correlated with the leave vote for local authorities in Great Britain.

It seems pretty clear that there is no correlation. That is, it provides no evidence for the proposition that changes in income inequality might be driving people to vote leave.

In fact, if you look at the first graph in their blog post you will see that although average weekly earnings have dropped from £490 to £470 since 2008, there's been a significant rise from a value of £430 in 2001. These are all in real terms, i.e. corrected for inflation. Nevertheless, the short-term and palpable perception is people feel poorer just now.

So if the change in incomes isn't correlated with the leave vote, might people's level of income be correlated? This graph from the same blog post plots the leave vote against hourly earnings in 2015.

The leave vote tends to be higher in lower income areas, but the scatter is very significant — it spans 30 percentage points at lower income areas and is perhaps even larger for higher income areas. So there's a correlation, but it's weak.

Exhibit B — values

An article on the Fabian Society website presents some analysis of data from the British Election Survey of 24,000 people in 2015-16. The graph below plots the probability of a leave vote against how strongly the respondent agreed with the death penalty for serious crimes.

This also shows a weak correlation between income and voting leave, but shows a strong correlation with support for the death penalty. Just in case that's so surprising that it's unclear: the more strongly people support the death penalty, the more likely it is that they will vote to leave the EU.

The association that interests psychologists here would be the the underlying values of the people involved. Support for the death penalty is suggestive of strong authoritarian values, that is a desire for an ordered society and severe punishments for those who break the order. This result is backed up by a second correlation between likelihood to vote leave and support for public whipping of sex offenders.

It surprised me at first that such questions were asked in the British Election Survey, but clearly psychologists and psephologists know enough about us humans to realise that asking an individual an explicit question about their motivations and values can yield distorted answers.

I have a few questions about the data presented in the above graph and would recommend not reading too much into it at present. For one thing, it certainly needs to be placed into a broader context with other questions from the survey. Nevertheless, it suggests that we need to consider ethical values and look beyond the commonly discussed left vs right of politics and national identity.

Scotland

I've lived most of my life in Scotland, so I'm keen to know if the above patterns are also present here. I have 35MB of British Electoral Survey data sitting on my laptop awaiting analysis, but I suspect it will show much the same correlations.

What I can say for now is that more careful probing in surveys about attitudes to Europe has shown that Scots are not much more europhile than others in the UK and that the current high levels of political support for the SNP and independence can explain much of the higher remain vote in Scotland.

If the societal issues that underly the brexit vote in England and Wales are present in Scotland too, then, although they may be suppressed at present by Scotland's distinct politics, they might emerge if Scotland becomes independent.

Correlation to cause

If we want to address the problems underlying electoral shocks, of which brexit is the biggest to date, then we need to understand why they happened. Explanations involving the poor and disenfranchised rising up against the establishment, or, to put it more simply, class war, may be valid, but are one dimensional. Searching in multiple dimensions is hard, but by looking at various correlations we can whittle down the number of possible causes that merit attention. But to go from correlations to causes we must think beyond the numbers.

The remain campaign placed heavy weight on the views of economists, and as important as economics is, I think their social scientist colleagues are due some more attention. Not just the psychologists, who I mentioned to above, but we need philosophers too. Ian Jack's article on the brexit vote goes deeper than its title suggests — "the poor turned on an elite who ignored them" — it points the reader to an interview with Harvard professor Michael Sandel, also known as the Public Philosopher. Don't just listen to him (or read his words), but pay attention to how re-assembles questions before he attempts an answer.

What I take from what he says is not that politicians should sell their answers to the public, nor should they put absurdly simple, isolated questions to them in a referendum, but that politicians should be working to articulate the questions that are of concern to the public.

So, philosophers please step forward. Your country needs you... to help us think.

The EU referendum result

2016-06-24T22:29:00+01:00

photo credit: Start button via photopin (license)

It's midnight on one of the shortest nights of the year and I'm ambling down a leafy road. There's some light summer rain and I'm pleasantly inebriated after a few pints at Glasgow's real ale festival. When I arrive home I check my phone and see that Leave were doing well in the first results. Too soon to worry about it, so I sleep.

51.9% to Leave. I am not pleased in the morning. I'm trying to see a positive here, but all I find are imponderables and uncertainty. The only clarity currently available from Brexit is that any definite prediction is a certain mark of foolishness. There's no plan. There's no precedent.

The referendum result is advisory and has no legal force. Nevertheless the politicians cannot ignore it. A good majority of UK MPs want to remain in the EU but they will now be tasked with managing the leaving process which legally starts when Article 50 of the Lisbon Treaty is invoked. Article 50 is a red button and pressing it makes Brexit real.

But politicians, even pro-Brexit ones including Boris Johnson, do not want it pressed yet. It seems likely that it will be David Cameron's successor who will be tasked with doing it. But why wait? Is it to negotiate? From what I've read, the EU leaders are saying "press that button now" and are unlikely to talk until that's done. Could it be that our governing politicians are hoping the UK might not leave the EU or is it just that none of them knows quite what to do.

Surely the next PM will have to press the button? Could she or he go against the expressed will of 51.9%^* of the people that voted? Probably not. But what if the next few months sees Project Fear become Project Fact?

Perhaps companies will hold back on investment, which is already in short supply. Perhaps the weak Pound will make imports expensive and people will start to feel a bit poorer; those going abroad on their summer holiday will notice their spending money is squeezed. This may be partly offset if exports are more attractively priced abroad due to the weak Pound, but if investment is low then UK supply may be unable rise to meet this demand. In short, it's hard to see how we'll avoid an economic downturn or a recession in coming months.

None of this is certain but arguing why these economic woes will not befall us requires a large dollop of hope. So in a few months perhaps the populous will feel leaving the EU isn't such a good idea, and as a result our next PM will seek a political get-out, such as putting it to a vote by MPs, or even another referendum, perhaps one on invoking article 50.

This is all possible and plausible, but I've absolutely no way of assessing how likely it is and I can imagine a spectrum of far less palatable outcomes. The optimist in me, though much cowed by the outcomes of politics in recent years, is grappling with the hope that such a shock can bring about meaningful and positive change, not just for the UK but for the rest of Europe. But if the populous and the politicians continue on their current course I see a break-up of the UK and the EU as distinct possibilities with potentially very serious and unpleasant consequences.

*For numerologists out there, that's 37.5% of those who could vote and is close to the percentage that voted Yes in Scotland in 2014, and 38% is also the percentage of Leave votes cast in Scotland. Happy conspiring!

The EU referendum

2016-06-18T16:35:00+01:00

For citizens used to representative democracy the belief we're voting for significant change can be intoxicating. We like to anchor our instinctive reaction for Remain or Leave, or Yes or No, with rational arguments. But instinct is a more powerful driver of thought than many of us can admit. Try as we might, there's no way to cleanly separate the emotional and rational elements of the debate.

Readers of my writings might expect me to present arguments drawn from graphs, tables and numbers. But there aren't going to be any in this blog post. With only a few days left in the EU referendum campaign it is highly unlikely any new significant facts will come to light, and every falsehood expressed will have been refuted at least once. But the falsehoods are more sensational and are announced with greater confidence and volume. So much so that nuanced analysis from academics, politicians, journalists and engaged citizens seems like a cautious whisper in the hubbub of a heaving pub.

The referenda of 2014 and 2016 are binary votes in which no one is elected. It should be no surprise that they generate division followed by confusion as voters on both sides try to hold various parties — or parts of parties — to account with varying degrees of reason and success.

Whether you like it or not, we live in a democracy in which we elect representatives to govern, and good governing involves informed decision making and negotiation, especially when other countries are involved. But, referenda allow governing politicians to outsource decision making to the populous. They then express their opinion on what must happen based on one particular question without sufficient thought going into how it can be brought to reality.

Now, hold on a minute, you might say (and I'm thinking it too or I wouldn't be writing this paragraph), is it right that our parliament should decide such important matters for us? The implicit answer to this question for almost all previous significant issues of state is "yes". This was true for going to war, for not joining the Euro, for rescuing the banks, for nationalising and privatising the railways, to name but a few. These examples are hardly trivial matters.

And more crucially, such large national and international issues cannot be viewed independently; they are deeply entangled. For example, the EU was a significant issue in the Scottish referendum and vice versa. This kind of entanglement is, at least in good part, why people who now intend to vote for Brexit did not all vote UKIP in 2015, and why people who voted Yes in the Scottish referendum were not all SNP voters in previous elections. (And no, beyond some superficial similarities, I don't see UKIP and the SNP as having much in common.)

One of the things I appreciated about Scotland's referendum, and about the Yes campaign in particular, was that it crossed traditional party boundaries. The same was true of Better Together, though it's hard to dissociate it with the image of Cameron, Milliband and Clegg delivering that last minute, panic-stricken vow.

I had hoped that the referendum would help shake up Scottish party politics. And it did. But a new binary politics asserted itself, with the Yes vote concentrating on the SNP and leading to some vocal antipathy directed at so-called Unionist parties. The Scottish Labour party, though certainly deserving of criticism, and already punished in the 2007 and 2011 elections, received the heaviest dose of scorn and vitriol from the SNP's most fanatical supporters. I was surprised by how tolerant very reasonable SNP voters and the party leadership was of them. I remember being taken aback when asked by a friend "So you quite like Jim Murphy then?" to which I replied "Not really, I just don't think he should have to put up with that guy shouting in his face."

I don't wish to imply that a tiny minority of unpleasant individuals represent the generally peaceful and positive people that came together to support a Yes vote. But nor do I find large crowds of people congregating to reinforce their views on one side of a binary vote particularly inspiring either. Of course, you can find it civic and joyous, as Alex Salmond described it, and I can simultaneously find it disturbing. There is no contradiction here. However, when a highly intelligent and knowledgeable friend told me the situation created by the referendum wasn't divisive, the fact we were divided on that very point seemed to escape his notice.

The EU and Scottish referenda have some similarities, but differ in many significant ways. The UK and EU are very different unions. The UK is much older than the EU and involves closer ties in trade and culture and shares a currency and language. Whereas the EU referendum has been dominated by the divisions in the highest ranks of the Conservative UK government, the 2014 referendum was unanimously backed by the majority SNP government in Scotland, and to their credit they produced a White Paper setting out their vision of an independent Scotland. Even if it was flawed and revenue projections for oil were wildly different from subsequent reality, it was more thorough than anything that has been produced to describe the UK outside the EU.

I'm writing this the day after Jo Cox was killed. Horrendous, senseless, heart-breaking are a few inadequate words that come to mind. I don't wish to speculate on how it might be related to the EU referendum, but it has paused the campaigns and I hope people will use this time to reflect rather than just dwell on the tragedy and read meaning where there may well be none.

I will vote Remain, not from any certainty, nor from any love of the EU, but because from what I've read and understood it seems likely that by remaining a part of the EU we will see a better future. And by "we", I primarily mean people in the UK, but also fellow Europeans and people who come to Europe to escape the horrors of warfare elsewhere in the world, most obviously in Syria.

It's hard to imagine Europe descending into such a nightmare, or that it might once again be at the heart of a world war. Perhaps we have trouble imagining it because the first world war has left living memory and the second is beginning to fade, but we shouldn't ignore the flames of conflict that still flicker around the edges of the EU. Civilisation is a thinner shell around humanity than many of us suppose, but its strength comes from its integrity, and so I believe we should be very wary when we see a crack developing.

Raising tax can raise GDP

2016-06-13T16:13:00+01:00

I've been improving my grounding in macroeconomics recently and have noticed something that isn't stated explicitly in textbooks, nor mentioned in public political debate, but I do think it's correct and liable to provoke some thought.

The main point of this blog post is expressed in the title's 5 words: raising tax can raise GDP. I'll present a plain English version first and then do it more thoroughly with mathematics.

Students of macroeconomics might well shout RUBBISH! at this on instinct. Let me explain why with an example and then explain why it's not rubbish (hint: 'can' not 'will'). I'll use ballpark figures to keep the arithmetic simple and the point clear.

Consider a household with an income of £50,000. Let's say it pays £10,000 of tax and that it chooses to save 20% of its income after tax. Then of its £40,000 post-tax income, £8,000 will be saved and £32,000 will be spent.

If the government raised taxes so that the household paid another £5000 in tax then the after-tax income would drop to £35,000, of which £7000 is saved and £28,000 will be spent.

The hike in taxes causes this household's spending to drop from £32,000 to £28,000 and the difference will be equal to revenue lost to businesses, be they supermarkets, cinemas or car dealers. Since the tax hike affects all households to some degree, the net result, all else being equal, is that GDP will drop. This is the perfectly reasonable justification behind the cry of RUBBISH!

But this scenario leaves an important question unanswered: what is the government going to do with the extra taxes it's raised? The RUBBISH! criticism stands if the government hoards the extra tax for a surplus, or indeed if it's intent on reducing a deficit. But let's consider what happens if the government decides to spend its extra tax income.

In macroeconomics, GDP can be expressed as the sum of all household spending (called consumption in macro-speak) plus government spending plus other amounts which we'll take as constant here which is what is meant by "all else being equal" (these other amounts are investment and net exports). The key point to take from this is that £1 of government spending and £1 of household spending both contribute to GDP equally (even if they can play quite different roles in shaping our society).

Before the tax rise, our example household spends £32,000 and gives £10,000 to the government. So the total private plus public spending we can attribute to that household, and which contributes to GDP, is £42,000.

After the tax rise, the household spends £28,000 and gives £15,000 to the government. So the total spending is now £43,000 and so £1000 more is contributed to GDP than before the tax rise.

This household is earning well above the median income, but with around 27 million households in the UK, it's clear that a boost of several hundred pounds per household is going to lead to a boost to the economy of several billion pounds, perhaps even tens of billions.

So, all else being equal, raising taxes can boost GDP if a government spends the extra amount raised. The reason is simple in our example scenario: households save some of what they earn whereas the government spends all it raises in tax.

In reality, not every household saves money and some do not earn enough to save anything at all, but, on average, households do save some money, and households on higher incomes save more and pay more in tax than those on lower incomes.

The UK government is almost always in deficit, meaning that it spends more than it receives in taxes. However, it's highly unlikely the current Conservative government will raise taxes, and even if it did, it would not use the extra revenue to increase spending, but instead would celebrate a reduction in the deficit.

The scenario we've considered is much more relevant to proposals made during the 2016 Scottish parliament election. Three of the four opposition parties did propose raising taxes and pledged to use the extra money raised to increase spending on education and other vital services. If they had done so, the end result of such modest tax increases would have given Scotland's GDP a small boost, which is certainly needed given the drag placed on it by recession in North Sea activity due to the low oil price. Unfortunately, those parties all remain in opposition and the largest opposition party — the Conservatives — along with the SNP Government have ruled out any significant tax rises.

There are two points worth bearing in mind. Firstly, an argument about whether public or private sector spends money more efficiently is a separate point to the one being made. As long as someone is spending the money into the domestic economy then there's a boost to GDP. Similarly, it's also a separate issue as to whether you think increasing GDP is in itself a worthy goal.

Secondly, the point I'm making is a variant on a Keynesian stimulus but one I felt it necessary to articulate because I heard a few folk claim that raising taxes in the context of austerity will lower GDP. If austerity is taken to mean simply cutting spending then this may be true, but a tax rise can offset the cut's impact on GDP. That said, a rise in GDP is small comfort if the spending cuts fall on services that people depend on.

Mathematical version

Let's start from a well known equation from macroeconomics for GDP $Y$:

\begin{equation} Y = C + G + I + X - M \label{income-full} \end{equation}

This equation is an identity. In other words, it's true by definition and makes no assumptions.

Before I define the terms, let me sketch its meaning intuitively. The left hand side represents total income in an economy and the right hand side is a total of various kinds of spending in an economy. Consider a very simple economy that consists of just you and one shop-owner. You go into the shop and spend £10. You give a £10 note to the shop-owner and she has an income of £10. What the above equation expresses is conceptually no more profound than that: your spending is another's income. The added complexity comes from adding up various kinds of spending and deciding what kind of transactions to include, and what not to include. That latter decision is mainly about avoiding double counting.

Here are the definitions:

$Y$ is the income of everyone resident in this country arising from economic activity that takes place here — for present purposes you can think of it as GDP.
$C$ is consumption — it represents all household spending on groceries, on a meal at a restaurant, on clothes, on fuel or on paying for a month's IT internet, and so on.
$G$ is government spending — money spent by the government on goods and services. This includes buying tables and chairs for schools or government offices, paying nurses. teachers and judges for services they provide, and so on. It excludes benefit and pension payments.
$I$ is investment — this is mainly money spent by businesses and as such is expected to bring a return in the future, for example on factory machinery or in installing or upgrading an office's IT system. It also includes inventories, which are goods of some value held in stock by businesses. Investment also includes some non-business spending, such as the purchase of a newly built house.
$X$ is exports — money paid by customers abroad in exchange for exported goods or services provided here.
$M$ is imports — money paid by customers in this country for goods and services provided from abroad. This is the only term to be subtracted because it is removing spending that is included in $C$, $G$ or $I$; such spending will be someone's income in, say, China, but not here so it shouldn't be included in $Y$.

But, for present purposes, we do not need all this detail, so I'll tidy the last three terms into a term called $R$ (the Rest) to keep the maths simpler:

\begin{equation} Y = C + G + R \label{income} \end{equation}

Next, let's define tax as being a fraction $\gamma$ of income:

\begin{equation} T = \gamma Y \label{tax} \end{equation}

We can define a standard consumption function:

\begin{equation} C = a + b (Y-T) \label{consumption} \end{equation}

where $a$ represents the absolute minimum consumption in an economy that might occur when everyone is nervous of some future calamity and saving every penny earned. $Y-T$ represents disposable income, that is income after tax has been deducted. A fraction $b$ of it is spent on consumption. In economic good times when people are optimistic about their future incomes, it's likely that $b$ will increase; conversely, it'll be lower in a recession.

If we use (\ref{tax}) to substitute $T$ in (\ref{consumption}) we get:

\begin{equation} C = a + b (1-\gamma) Y \label{consumption-tax} \end{equation}

Next, let's say that government spending $G$ is made of two parts:

\begin{equation} G = T + F= \gamma Y + F \label{gov} \end{equation}

This expresses government spending as the sum of tax revenue $T$ and non-tax revenue $F$. We'll consider what $F$ represents later on.

So let's substitute $C$ from (\ref{consumption-tax}) and $G$ from (\ref{gov}) into (\ref{income}):

\begin{equation} Y = a + b (1-\gamma) Y + \gamma Y + F + R \end{equation}

Then collect terms with $Y$ on the left

\begin{equation} Y (1 - b - \gamma +b\gamma) = a + F + R \end{equation}

which gives us the final result

\begin{equation} Y = \frac{a + F + R}{(1-b)(1-\gamma)} \label{result} \end{equation}

Interpretation

Firstly, note that $\gamma$ is not the same thing as income tax as it is a fraction of total income (GDP) in the economy, not just what people receive as pay. However, a lowering of the income tax rate will likely cause a lowering of $\gamma$ because income tax is the single largest source of tax; it makes up about one fifth of the total tax take.

To interpret this result, let's consider that $\gamma$ changes, and all quantities stay the same:

If $\gamma$ is reduced then $(1-\gamma)$ will increase and $Y$ will decrease.
If $\gamma$ increases then $(1-\gamma)$ will decrease and so $Y$ will increase.

In other words, this model is telling us that increasing tax rates with all other variables staying constant will raise GDP. To understand this we need to consider the two effects of changing $\gamma$ in the above equations.

Increasing taxation reduces consumer spending $C$ and we can see that the reduction is $b \gamma Y$ in (\ref{consumption-tax}). The other effect of increasing taxation is to increase government spending $G$ due to the term $\gamma Y$ in (\ref{gov}) which represents the bit of government spending funded by tax. Since $b$ is a fraction and so less than one, that means the increase in government spending is larger than the decrease in consumer spending and so all else being equal raising the tax rate $\gamma$ means that $Y$ is increased.

In other words, a government will spend 100% of the money it raises in tax. But if the government were not to take that tax and it was left in consumer hands, then they would chose to only spend a fraction $b$ of it.

What if all else is not equal? That is, what if other variables on the right hand side of (\ref{result}) don't stay constant and, most significantly, what if they change in response to $\gamma$ changing, i.e. they are functions of $\gamma$? I can't hope to address this properly in a blog post, but I'll sketch my thoughts on it.

If people trusted the government tax rise was going to lead to increased spending which benefited the economy, perhaps for the reason argued here, or else because they believed, say, spending on improved education might bring benefits in the longer term, then their optimism may be reflected in increased consumption (larger $b$) and this will boost the economy further. But, if the populous is not so enlightened, and are spooked about profligate government spending or mounting public debt, or if they simply don't trust the government to manage the economy, then this may dent their optimism and reduce consumption (lower $b$), cancelling some or all of the tax-and-spend rise boost to GDP.

Notice the use of the word 'believe' here. People do not react to reality itself, but what they believe to be true of reality. Despite the apparent certainty of starting with a mathematical identity, we've ended up having to consider not only whether $b$ is a function of $\gamma$ but also if it's a function of mass psychology! I'm not sure how you might express such a function — perhaps some stochastic Ito calculus is needed.

Another interesting point is the extent to which government can influence the economy. Let's start with the easy bit: variables which a government really has no control over. The minimum level of consumption $a$ required by the society does not obviously seem to be under government control, nor does the fraction of income $b$ spent on consumption.

Next, consider the $R$ term which is composed of investment $I$ and net exports $X-M$.

Investment can be influenced by interest rates, but in the UK the last Labour government gave the Bank of England the responsibility for setting them to keep inflation close to the 2% target. The government may have some indirect control here and can enact policies that encourage investment in specific areas, such as feed-in tariffs for solar energy generation (recently cut). But investment is prone to psychological views of the present and future in the same way as consumption. In fact, perhaps even moreso. It was one of John Maynard Keynes's major insights that although $I$ may be much smaller than $C$ in size, it can vary suddenly and trigger unpredictable swings in GDP $Y$.

A government may be able to influence exports on longer timescales with trade deals and can make the price of exported goods lower by devaluing Sterling, but it has to balance that with the above mentioned setting of domestic rates and the possibility of capital flowing out of the country.

So, that leaves us with $F$ and $\gamma$ that appear to be most directly under government control. Recall that $F$ is the bit of government spending that is not funded by tax revenue. Let's rewrite (\ref{tax}) as:

\begin{equation} F = G - T \label{deficit} \end{equation}

Written like this, it's obvious that $F$ represents a budget deficit: government spending minus its tax revenues. Equation (\ref{result}) makes it pretty clear that reducing $F$ will reduce GDP. If austerity is to work (meaning that it will one day raise GDP) then it must do so by boosting $a$ or $b$ or $I$ or $R$. I cannot think of a credible way this could happen. But there's an interesting point here. Even if the government is reducing $F$ by cutting spending, there's still the possibility that a tax rate rise, i.e. an increase in $\gamma$, can offset it, and potentially hold GDP constant or even allow it to increase. I can't see the current Conservative government doing this and so I think this will remain an untested theoretical point for the foreseeable future and besides, the more sensible thing to do according to equation (\ref{result}) is not try to reduce $F$ in the first place.

There is also another interpretation of $F$ in the context of Scotland — a fiscal transfer. I won't get into details here, as this deserves a blog post to itself, but since $F$ is supported at the UK level, and because the Scottish government is compelled by law to balance its budget with only modest borrowing powers, any rise in tax will necessarily be matched by a rise in spending. This means an increase in $\gamma$ will lead to a boost in GDP. Again, for the political reasons mentioned above (at the end of the non-mathematical section), I can't see this happening.

Final thoughts

The strength of the above model is that it starts out from a mathematical identity that must be true. The first weakness was introduced when we assumed a linear form of the consumption function $C$ in (\ref{consumption}). Even if $C$ is not linear in $Y$ and if $b$ varies a little with $\gamma$ then the result will retain some validity so long as $C$ increases with $Y$. But of course it has absolutely no relevance to reality in the circumstances of 2008 when the financial system seized up and confidence fell through the floor. I think it can be used to think about the situation of the economy outside of such crisis times and so might well be relevant to present circumstances.

The second weakness is when we used it to solve for $Y$ in (\ref{result}). This result describes an equilibrium, but as all quantities on the right hand side are continually changing no true equilibrium can ever be reached. This could be relevant if changes in variables such as $a$, $b$ and $\gamma$ are gradual enough so that we can imagine the economy going through a series of equilibria where (\ref{result}) is close to being true. However, if $I$ dropped suddenly, as it would at the onset of recession, then a proper dynamic model is required and it would somehow need to factor in the above mentioned mass psychology.

I believe this kind of simple model has some use but only so long as the assumptions it makes are clearly understood and are relevant to real circumstances.

I would welcome increased government spending on education and health because both will benefit society and the economy in the long term. If this is funded by raising progressive taxes then we can also tackle the problem of income inequality and, if the assumptions made above hold, we can also give the economy a wee boost too.

Inchmaholme priory

2016-05-25T13:47:00+01:00

Inchmaholme priory is a ruin that sits on a small island within Scotland's only lake — the Lake of Menteith. It's called a lake rather than a loch due to a mistranslation of the Scots word laich which means lowlands.

The island's air of tranquility is apparent as soon as you step off the boat, so it is not at all surprising that the religious order following St Augustine of Hippo chose it as one of their retreats from society. That said, the Augustinian canons, unlike Benedictine monks, did not lead an enclosed life because part of their duty was to minister to the population around them. Nevertheless, being isolated on an island gave them a good excuse to minimise contact with the great unwashed, or at least have a wee wash on the way back to the priory.

Inchmaholme - the island viewed from the boat. My phone's camera lens needs a clean.

The priory was founded in the 13th century and continued to be a home for Augustinians until the early 17th century. It then passed into the hands of the Graham family who created gardens and planted trees. During Victorian times it attracted the interest of well-to-do tourists who indulged in romanticising its past.

The priory - trees and ugly fences obscured the view from most directions. Access was restricted due to winter storms loosening masonry.

Inchmaholme is famous for being the temporary residence of Mary Queen of Scots. You can visit Queen Mary's Bower which is a circle of boxwood trees on some raised ground. It's unlikely Mary constructed it herself as she was only on the island for three weeks when she was just four years old. The Victorian enthusiasm to return with a piece of (dubious) history meant that the original boxwood trees were gradually dismantled. The ones you can see today are replacements.

May is a good time to visit the island as it is alive with the burst of colours and fragrances of late spring flowering. There's a remarkable variety of trees too, including some wizened sweet chestnuts that are ill-suited to the climate. At the north east of the island is an oak that fell a couple of decades or more ago.

The fallen oak on Inchmaholme.

The tree is remarkable for a number of reasons. Most obviously, its exposed root system provides a climbing wall for children. But if you look closely above my son's head, you'll see that there is the trunk of a silver birch growing on top of the oak's roots. I'd guess the silver birch is about 20 years old and so it's possible that it was hoisted into the air when it was a sapling and not only survived but continued to grow. The other remarkable thing is that the oak itself is not dead. What were branches off the original trunk are continuing to grow upward like trunks themselves and many of the branches have healthy leaf growth.

Inchmaholme priory is a special place. I can hardly do it justice in a few words and pictures taken with my phone. A visit is highly recommended.

I've written a book

2016-03-30T21:46:01+01:00

I've written a book. It is called An Active Citizen's Guide to Scotland and in it I share my understanding of the information and data that describes Scotland's economy and society.

You can visit its website activecitizen.scot or visit its facebook page.

You can also read a sample of it and buy the e-book.

GERS revenue 2014-15 preview

2016-03-06T19:55:01+00:00

I was leafing through Scotland's National Accounts, as you do, when I noticed that some aspects of the much awaited 2014-15 GERS (General Expenditure and Revenue Scotland) report, which is due for release later this week, are in fact already released, albeit tucked away at the end of a spreadsheet.

First up, the graph below shows the total revenue figures for 2014-15. The figures are nominal, i.e. not adjusted for inflation, as they will be published in GERS.

The most obvious bit of news is that in 2014-15 North Sea revenues are lower than in any year back to 1998-99, possibly longer. And remember that's in nominal (or cash) terms. On the plus side the total onshore revenue (i.e. excluding the North Sea) is up in nominal terms between 2013-14 and 2014-15.

Next, we can look at GDP which again is in nominal terms, and split into onshore and North Sea parts.

There has been onshore growth in nominal GDP since last year, but if we include the North Sea, Scotland's economy shrank by £700 million between financial years 2013-14 and 2014-15. The definition of a recession is if an economy shrinks for two consecutive quarters, and looking at quarterly, real terms GDP figures in the national accounts, Scotland's recession started in 2013Q4 with only 2014Q4 and 2015Q2 showing growth since then.

According to European ESA10 standards for national accounts, Scotland's official GDP is onshore-only and so technically we're not in recession. That said, with real onshore growth at only 0.1% in 2015Q2 and 2015Q3, we were teetering on the brink of one in 2015. We won't know 2016 figures for a while yet.

Finally, here are the figures in the first graph divided by those in the second, i.e. Scotland's revenue expressed as a percentage of GDP.

If we exclude the North Sea the news is not good: after remaining stable at about 37.0% for several years, Scotland's revenue has fallen to 36.5% of GDP in 2014-15. The news is considerably worse if we include North Sea: Scotland's revenues have fallen to 34.6% of GDP.

The reason the percentage of GDP including the North Sea is so much lower than the onshore one is that revenue from the North Sea is based on a tax on profits which have fallen much more dramatically than has North Sea GDP. In fact, profits are close to nil at present.

The national accounts do not contain enough information for me to estimate the expenditure figures that will be released in GERS this week, but they'll be much higher than the revenues of £53 billion. My guess-estimate is they'll be £67-69 billion, about £15 billion higher than Scotland's revenues.

A comparison with the UK gives this some perspective. Chancellor George Osborne plans to cut public spending from 40.6% of GDP in 2014-15 to 36.3% in 2020-21. He hopes that this will give him a balanced budget by then, i.e. revenues will equal or perhaps even slightly exceed expenditure. This means UK revenues will be around 36% of GDP too. For historical context, the UK's public spending has remained above 35% since World War 2.

So, in terms of revenue as a percentage of GDP, Scotland has already gone beyond what Osborne plans for the UK after another five years of austerity. If Scotland were independent, which it would've been in 2016 had the Yes vote carried, then we'd face a tough choice in how we'd sustain our high public expenditure against lower revenues.

One option would be to cut public spending and endure austerity much more savage than the Tories plan for the UK. Another would be to raise taxes by a large amount, probably by more than the economy could sustain. The other option — possibly the least bad — would be to seek massive borrowing on top of whatever agreed debt Scotland would take from the UK at separation. Whatever fiscal levers could be pulled to improve growth in years ahead, an independent Scotland would've started from a very difficult place.

As things stand, Scotland remains in the UK and we do not have to contemplate such horrible choices. The UK's onshore growth has been reasonable to 2015Q3 and the damaging effect of plummeting North Sea revenues is heavily diluted at the UK level (ball-park 1% of revenues for the UK, rather than 10% for Scotland).

Of course, my figures are not official and it's possible I've made a mistake in the detail. I'd be most grateful if anyone did let me know if they find any errors, especially if they lead to a more optimistic statement of Scotland's economy. The national accounts spreadsheets with my workings can be downloaded in ods or xlsx format.

As a postscript, I'm delighted to see whisky export duties are up one million percent on 2013-14, but then they always have been a nice round number.

A tale of two Dunbartonshires

2016-03-04T19:20:01+00:00

The issue of devolving powers more locally is a recurring theme in recent political debate and particularly the de-centralisation of income tax — the single biggest source of public revenue. This raises the question:

How can taxes be set and collected locally to fund council expenditure when income, wealth and needs vary so much between councils?

I'm going to explore the issues around this question by looking at two neighbouring councils East and West Dunbartonshire. It's reminiscent of the East Lothian question in that both raise problems associated with different levels of government. And, of course, the issues of tax and representation cannot be separated, and are hardly trivial, such was the cry of "no taxation without representation" in the run up to the American Revolution.

Numerical details and references to sources I've used in what follows are in this spreadsheet, available in ods or xlsx format. Bewarned though, those spreadsheets are quite involved: untangling finances from multiple levels of government isn't easy.

First, let's look at council tax raised by each council.

As you might expect, poorer areas collect less council tax than richer areas. West Dunbartonshire has the 5th lowest council tax take per person, and its eastern, posher neighbour has the 5th highest.

Next, let's look at the non-domestic rates (NDR) that are collected from each council. You can think of NDR as being a tax on businesses with premises in a council area.

And here's a surprise. The situation is reversed. Now West Dunbartonshire is ranked first — it collects more NDR per person than any other council. East Dunbartonshire comes out second lowest, ahead of its rival comfy Glasgow suburb, East Renfrewshire.

The plausible explanation is of course that rich, residential areas tend to have little heavy industry or large places of employment. Property values are high precisely because they are leafy, green and quiet suburbs. West Dunbartonshire on the other hand offers lower land costs and so is a more attractive area for large businesses to occupy.

Our third graph shows gross expenditure per person, and breaks it down into the major revenue sources that fund it.

Let's look at the total expenditure first. East Dunbartonshire spends £2700 per person whereas West Dunbartonshire spends £3400. That's £700 or 26% more — a significant difference.

I think it's reasonable to suppose that the councils with the highest expenditures per person are spending more because of need rather than inefficiency or profligacy. The graph contains some supportive evidence of this: the highest spending councils are those with islands, followed by urban areas that have high poverty rates, such as Glasgow and Dundee.

Before returning to the Dunbartonshires, the sources of revenue other than council tax need some further explanation:

the General revenue grant is the block grant from the Scottish government, and forms the largest part of most councils' funding.
NDR is labelled received because councils don't get to spend the NDR they collect, but instead it is sent to the Scottish government and then redistributed back to the councils.
Revenue income is composed of two distinct elements and should really be called other income. Typically, half of it is raised by councils themselves from things such as parking charges, fines and rentals for use of council land or premises. The other half comes mainly from the UK government's Department of Work & Pensions to fund housing benefit (listed under non-HRA housing on council finances).

With all this in mind we can break down the funding for the Dunbartonshires a little further:

This shows us two things. Firstly, West Dunbartonshire gets almost three times more funding for housing benefit than East Dunbartonshire. Secondly, the blue plus red plus green bits are raised locally, whereas the dark-red plus orange bits are funded by central government. (Actually, some central government funding is probably still mixed in with the green bit, but it's a small amount and fiddly to separate out.)

And this leads us to this table showing an estimate of the breakdown between local and central funding.

	Local funding	Central Funding
East Dunbartonshire	£1000	£1700
West Dunbartonshire	£1500	£1900

Even with West Dunbartonshire's extra ability to raise revenue locally from businesses, it still needs more central funding to meet its expenditure per person. And this leads to the question:

If funded by a local income tax, how can West Dunbartonshire raise more money from its residents who have lower incomes than its richer neighbour to the East?

The question gets even tougher to answer if we were to consider a council that isn't fortunate enough to have a high income from NDR, such as West Dunbartonshire's other neighbour Inverclyde.

The obvious solution is to collect taxes locally, send them into a central government and redistribute them back to the councils based on need, much like is done with NDR, and indeed implicitly with income tax already via block grants.

And admitting that this is the solution is also an admission that centralisation is necessary, at least to some degree, to ensure funding gets to where it is needed. It makes more sense to me to avoid the needless complexities of a local income tax, and the increased potential for tax competition between councils, by keeping income tax centralised.

A similar point could be made about devolving tax to countries and regions within the UK, but some care is needed because that differs in many important details. That said, the machinations around the Scotland Bill and the ensuing framework highlights the extreme fiscal yoga positions politicians have to adopt when compromising de-centralisation with "no detriment".

So, am I saying that de-centralisation is impossible? No, though I think it is impossible to do it fairly within the confines of our current system. If the true natures of fiat money, credit and public borrowing were understood more widely then new possibilities will emerge, but that's for another blog post.

There's an old joke which makes the general point more simply:

Patient: Doctor, doctor... it hurts when I do this!
Doctor: Well, don't do that.

Scotland - full fiscal flows

2016-02-23T20:37:01+00:00

Scotland's fiscal flows are the monies related to tax and public spending that wend their way through all three levels of government — UK, Scottish and local — and sustain our public infrastructure and services.

I created the diagram below to illustrate the flows, drawing on a number of sources. The figures relate to 2015-16 and should be regarded as estimates as we're still in that financial year. They're rounded to the nearest half billion pounds, though the uncertainty in my estimates may well be a bit higher for the larger figures. Numerical details and full references to sources are in this spreadsheet available in ods or xlsx format.

The main part of the diagram is the oval that represents Scotland's people, its businesses and the public infrastructure and services that allow them to live their lives and do their work. As they work they will pay income tax, national insurance, VAT, corporation tax and so on. This is collected along with all UK taxes by HMRC.

Next, look to the UK government box. From it are two transfers going down to the Scottish Government: the block grant of £30 billion and the AME component of £4 billion. The block grant is set according to Departmental Expenditure Limits (DELs) that represent more predictable costs which can be agreed on at spending reviews that occur every few years. In contrast, AME (Annually Managed Expenditure) is demand-led funding that must be determined each year. The Scottish Government also receives £0.5 billion via Revenue Scotland from the new taxes it started levying from April 2015. These are the landfill tax and LBTT which replaced stamp duty on property and land sales.

Local government's major source of funding is the £11 billion block grant from the Scottish Government. This is supplemented by £2 billion raised from council tax and £2.5 billion of fees from various council services.

Just as local government collects council tax from residents who live in homes, it collects a tax from businesses called non-domestic rates (NDR). The £3 billion of collected NDR monies is transferred to the Scottish Government who then redistribute it back to the councils (included in the block grant in this diagram). This is done because there is significant variation of the NDR raised from one council to the next.

The spending by each level of government is shown by the dotted arrows. The Scottish Government spends the most at £27 billion. This is dominantly on health, with the rest divided between pensions, education (universities and colleges), infrastructure and justice. The UK government spends £23 billion, and this is almost entirely on pensions, welfare payments and defence. Finally, local government spends £16 billion, with most of this going on school education, social work and housing.

We can now summarise how tax and spend balances overall:

£ billion 2015-16	Tax	Spend
UK Government	42	23
Scottish Government	3.5	27
Local Government	4.5	16
Total	50	66

Note 24/2/16: When this post was first published the tax figures in the above table were incorrect and the 42 which appears in the diagram was wrongly shown as 38. The same figures have also been corrected in the spreadsheet. Thanks to @McCaigsTower for bringing it to my attention.

It's clear that tax does not come close to equaling spending at any level, and that £16 billion more is spent in Scotland than is raised in tax. This is sometimes referred to as the net fiscal balance (closely related to, but not quite the same thing as the deficit).

The first line of the table appears to show the UK government collecting much more tax from Scotland than it spends. This isn't the UK government screwing over Scotland, or the mystical whisky export duty or such-like, it's just that this table is a summary of tax and spending only. A glance back at the diagram above shows that £34 billion is transferred to the Scottish Government and it in turn transfers £11 billion of that to local government. In this way, the block grants and other transfers ensure that Scotland has the funds needed for its spending at all levels irrespective of what tax revenue is raised.

By law, the Scottish Government has to balance its books each year. In recent years it underspent its budget, but it now has some borrowing powers and can draw on reserves, so has a little more fiscal wiggle room. Local government too can borrow money, but in both cases the amounts borrowed for 2015-16 are small and lost in the rounding in the diagram above.

It'd take a giant illegal wiggle to square off the missing £16 billion if it were to appear in the Scottish budget reconciliation, and thankfully it isn't necessary because it gets swallowed into the UK's overall budget deficit.

Deficits are not necessarily a problem for states like the UK which have their own central bank and currency. In fact, the UK has run a deficit in 54 of the 60 years between 1956 and 2016 and, if you care to count them, you'll probably find that Conservative governments have run more years with deficits than Labour governments.

But I digress. The reason I wanted to test my understanding by creating this diagram is that mother SNP and father Tory have, after much huffing and puffing become the proud parents of a newly born fiscal framework. Amongst other things, this provides the new rules for calculating the block grant to the Scottish government given the tax powers that will arrive once the Scotland Bill comes into effect. I'm not sure what the outcome of all this will be, but my next task is to create an animated version of the diagram in which the arrows move around Dad's Army-style.

Is raising the Scottish rate of income tax progressive?

2016-02-02T18:37:01+00:00

A progressive tax change is one that it benefits those on lower incomes more than higher incomes.

Let's consider what raising the Scottish rate of income tax by 1p means. The bar below shows the income of someone who's on the minimum wage, earning about £14,000 (before tax) per year. This is in the bottom 25% of incomes in Scotland.

The green bit on the left is the personal allowance. You pay zero tax on that. It's £11,000 (using 2016-17 figures), so tax is only paid on the £3000 shown in red. At the moment the basic rate of income tax is 20%, so the tax that needs to be paid on this is:

0.2 × £3000 = £600

If this is bumped up to 21%, as proposed by the Lib Dems and Labour in Scotland, then the tax due goes up to

0.21 × £3000 = £630

That's an extra £30 a year or 58p per week.

Let's now consider someone who's on a much higher income of £41,000 per year. This is just enough to be in the top 25% of incomes in Scotland. The bar for them looks like this:

As before, the green bar is the personal allowance of £11,000 on which you pay no tax, but now a much greater proportion of the income is red. This person will pay tax on £30,000:

0.2 × £30,000 = £6000

Let's pause to appreciate what these calculations are telling us: this person has a taxable income that is ten times greater than the minimum wage person, and so they pay ten times the amount of tax. Putting the tax rate up to 21% gives:

0.21 × £30,000 = £6300

We can summarise the tax rise for these two people as follows:

Income	Tax rise	Tax rise as % of income
£14,000	£30	0.2%
£41,000	£300	0.7%

So there is a larger rise in tax for the person with the higher income whether we look at it in pounds, or as a percentage of income. In other words, this is very definitely a progressive tax change. That said, the rise is fairly modest, being only 0.7% for the higher earner.

The person on the lower income is having to pay more tax, but they are still likely to benefit overall for a couple of reasons. Firstly, some of the increased tax take can be returned explicitly to low earners, as Scottish Labour propose with a £100 annual bonus. Even without this, folk on lower incomes are likely to rely more on state services (education, health, social care etc) and so will benefit if the extra tax is spent on those things.

Both the examples above involve only the basic tax rate. Because of the way the Scottish rate change works, all tax rates get bumped up by 1 percentage point: the higher tax rate will go from 40% to 41%, and the additional tax rate will go from 45% to 46%. Also, as you go to higher incomes, the personal allowance becomes a smaller proportion and so the tax rise as a percentage of income will rise from the 0.7% in the table above to approach 1.0% for salaries over £100,000. (The personal allowance actually goes to zero for incomes above £122,000.)

Clearly, 1% of a big income is much more than 0.2% of a minimum wage income. But aren't there far fewer high earners? There are, but that's a subject for another blog post. For now, mull over the implication of the following fact: higher-rate tax payers — that is, those earning over £43,000 — contribute about two-thirds of all income tax collected.

The question is therefore not whether such a tax rise is a progressive policy — it clearly is — but whether to support it. Personally, I do, as I am willing to pay a bit more tax to help those less well off and to fund local services, particularly schools and care for the elderly. There are possible downsides, such as higher income folk moving out of Scotland, but I think those risks are fairly low and outweighed by the likely benefits.

A spreadsheet with all above calculations, references and more background data can be downloaded in ods or xlsx format.

Absolute poverty

2015-12-07T19:37:01+00:00

Absolute poverty has different meanings in different contexts. I thought it'd be interesting to compare the UK meaning with the international one and consider the extent to which they are comparable.

First, here is the United Nations 1995 definition of absolute poverty:

A condition characterised by severe deprivation of basic human needs, including food, safe drinking water, sanitation facilities, health, shelter, education and information. It depends not only on income but also on access to services.

This is also referred to as extreme poverty, so to avoid confusion I'll use that to mean international poverty and absolute poverty to mean the UK definition.

The World Bank provides an income-based definition of extreme poverty which is often associated with the UN definition:

An individual living on less that $1.90 per day is in extreme poverty.

This amount is not in normal US dollars but in purchasing power parity dollars (PPP) that are stated in 2011 prices. The PPP system accounts for the fact that prices of common goods and services in a richer country tend to be higher than those in a poorer country. For example, a loaf of bread that costs $1 in the USA may cost the equivalent of $0.50 in Rupees in India. PPP conversion rates are different to exchange rates and are much less variable over time (the reason for that is an interesting story in itself).

PPP conversion factors from the World Bank shows that $1 PPP equates to £0.76 in 2011. So the extreme poverty line for 2011 equates to 1.90 divided by 0.76 which is £2.50 per day.

Next, here's the current UK definition of absolute poverty:

An individual living in a household with an income that is less than 60% of the inflation adjusted UK median for 2010/11 is in absolute poverty.

(It's worth digressing a little to explain why it's called "absolute". The other measure of poverty in the UK is relative poverty which is defined in a similar way except the threshold is 60% of the current year's median. That is, it is defined relative to the income distribution that year. The absolute threshold does not depend on changes in the income distribution; it only changes due to inflation.)

According to the Households Below Average Income (HBAI) report the absolute poverty threshold in financial year 2013/14 (most recent available) is an income of £14,600 per annum, or £40 per day. This is after tax and includes benefits but before housing costs.

A standard household in HBAI has two adults and no children. (Actual household incomes recorded in surveys are converted to this standard using a Dr Who-esque sounding process called equivalisation.) This gives a per person threshold of £20 per day.

The value of £2.50 from above is for 2011, so we need to adjust for inflation. The best choice here is RPI as it is used in HBAI to adjust the absolute poverty threshold itself. Once this is done we find the value for 2013/14 is £2.70.

So in 2013/14 prices, the UK threshold for absolute poverty is £20 per day and the international one is £2.70, a factor of 7.4 lower. As you might have guessed before seeing these figures, the UK and international definitions represent two very different levels of poverty.

The UK adopts a much higher threshold for poverty because it is one of the richest countries in the world. This is best quantified with Gross National Income which can be thought of as being similar to GDP except that it only includes incomes of residents. For example, the UK's gross national income per capita is about fifteen times that of Cameroon in 2011 PPP dollars. Also, a third of Cameroon's population is estimated to live in extreme poverty, whereas 17% of the UK's population live in absolute poverty (before housing costs).

Of course, as stated in the 1995 UN definition, we cannot assess poverty using numbers alone. A rich country uses its high gross national income to provide high levels of public services such as education, welfare, health, transport and justice. Many of those living in extreme poverty have little or no access to such services.

I'm concerned that 10 million people in the UK live in absolute poverty, but I'm even more concerned that about a hundred times that number live in extreme poverty worldwide. Latest estimates for 2012 put this at 896 million people — 12.7% — of the world's population.

There is some good news though: the level of extreme poverty is falling. It was 37% in 1990 and 44% in 1981. When you see these numbers, the UN goal of ending all extreme poverty in the world by 2030 doesn't seem as far fetched as many might think.

If you're interested in learning more about how the numbers connect to the reality of people living with extreme poverty then watch the film Don't Panic by Hans Rosling of Gapminder.

A wee bit more on electricity in Scotland

2015-11-20T16:37:01+00:00

This is a just a quick follow up to my last post to tighten up a few details on the estimates I made. The numbers and graphs presented in this post supersede the ones in the last one for a few reasons which I'll detail below. But, in short, the estimate of non-wind renewables has been refined and this has resulted in less wind power being required by 2023 to meet the Scottish Government's renewable target.

New graphs

The graph below replaces the one in the last post.

The 2015 column is little changed, except that non-wind renewables have increased by 0.6 GW, and wind reduced by a similar amount. Whereas the 2023 column was noticeably larger than the 2015 one in the last post, it is now slightly smaller having been reduced by 3 GW (as explained below).

The two de-rated columns weren't included in the last post — they show the average generation we can expect in reality. The 2023 de-rated column is much smaller than the 2015 one because of the loss of coal and nuclear, but also because de-rating significantly reduces wind capacity. As such the 2023 total de-rated capacity is much less than the 2015 value.

The next graph is new and illustrates what was described only in words in the last post. A factor of 0.85 (representing transmission losses and generators' own use) is applied to all the de-rated values in the above graph. This allows us to compare available supply with the average and peak demand values — 3.7 GW and 5.5 GW respectively.

In addition to the average wind power, the possible above average amount is shown as the outlined box at the top of the columns. So on any given day, the available supply to meet demand can vary from just above the top of the brown pumped storage segment (calm day) to the very top of the outlined box (windy day).

This graph makes it clear that average and peak demand can always be comfortably met in 2015 with room to spare for exports. However, in 2023, a peak demand at average or low wind levels would require imports. In fact, even average demand might not be met on very low wind days. Exporting in 2023 would only be possible when the wind was above average.

So although some of the figures have changed quite substantially, notably the required wind capacity, the conclusion and bottom line figures remain much the same.

Note: As explained in the last post, if Scottish supply fails to meet Scottish demand that will not cause a "lights out" situation as long as demand is met across the entire GB grid.

New details

The method and so figures have changed for the following reasons:

I improved the reconciliation of DUKES with EIS renewable figures and brought the latter up to date with data from Figure 2 in the Renewables routemap update 2015. This meant that the amount of non-wind renewables is upped by about 0.6 GW.
Table 1 in the renewables update report states planned renewable capacity, so the 2023 prediction now assumes 30% is constructed, whereas before this value was assumed to be 0% for non-wind renewables. Since 95% of planned new capacity is wind, this doesn't have much impact.
National Grid de-rating factors for various energy generation methods are now used: Table 8 in the National Grid EMR electricity capacity report. These multiply on to the maximum capacities and reduce them to values you might expect in reality. They range from 85% for diesel up to 97% for pumped storage, and I used a value for wind of 29.4% estimated using Scottish historical data. This is an improvement in assuming 90% for all non-wind sources (stated as a 10% decrease in the last post).
Finally, armed with 1-3, I refined the calculation of the wind capacity so that it now just meets the 100% renewable 2020 target (before the target was slightly exceeded).

The end result of all this is that about 3 GW less wind power is needed in 2023. The most part of this comes from including more non-wind renewables because at a capacity factor of around 90%, as compared to 30% for wind, 0.6 GW of non-wind capacity equates to about 1.8 GW of wind capacity.

The full workings behind all of the above are in this spreadsheet: ods or xls formats. Please do get in touch if you find any mistakes in it.

Other things

There are a few factors worth considering not accounted for in the above:

During a long, low-wind spell with high demand, it may happen that the stored energy in pumped storage runs lows, in which case about 0.7 GW of supply wouldn't be there to meet demand. It's also possible that low rainfall restricts hydro power as happened in 2010, which might be a particular problem due to a correlation of wind and rainfall on longer timescales.
Non-wind renewable figures used above include about 0.2 GW of solar power. At Scotland's latitudes this would be reduced to near zero in winter which of course is when demand is highest.
The capacity of wind turbines declines with age.
The renewables target is based on 100% 2013 gross consumption. If gross consumption falls as it has done in recent years it may well be somewhat lower in 2020. However, the gradual decline in energy use may halt or even reverse as efficiency improvements reach a limit and population increases. The National Grid's Future Energy Scenarios predicts UK electricity consumption in 2030 to be larger than now in all four of its scenarios.
Finally, what if the 100% renewables target isn't met, or what if it is exceeded because more than 30% of renewable projects are constructed by 2023? You can answer that for yourself by tweaking the the values highlighted in yellow in the spreadsheet mentioned above.

A question on electricity in Scotland

2015-11-15T14:37:01+00:00

Note: More refined estimates and better graphs are provided in this post, but the conclusions remain unchanged.

Scotland currently has four large conventional power stations. In descending order of capacity they are:

Longannet - coal - 2.260 GW
Torness - nuclear - 1.185 GW
Peterhead - gas - 1.180 GW
Hunterston B - nuclear - 0.965 GW

Longannet is scheduled to close in 2016, and the two nuclear stations by 2023. Between them they comprise 4.4 GW — about a third — of Scotland's generation capacity. So a question many might ask is:

Will there be a problem in meeting Scotland's electricity demand when these three power stations close?

(The data above comes from DUKES 2015 and unless otherwise stated other data given in this post comes from the Energy in Scotland 2015 (EIS) Scottish Government report.)

Sharing across the grid

The question is in fact based on a false premise. In reality, there is no Scottish demand met by Scottish electricity generation. A problem will only arise if demand across the whole GB national grid exceeds its supply of electricity from all power generation connected to the grid, and by inter-connectors to other countries.

That said, there are still reasons to consider the Scottish balance between supply and demand by estimating the amount generated and the amount used by consumers located in Scotland. One is political: the Scottish Government makes decisions on energy policy in Scotland. Another is economic: currently Scotland exports electricity and these sales bring benefits such as jobs and taxes raised in Scotland. So the "problem" mentioned in the above question may not be a lights out issue, but something more nuanced. Will the loss of these three power stations push Scotland into becoming an electricity importer with negative economic consequences? And if this happens, how does becoming dependent on decisions made in England and Wales sit with the Scottish government, and those who support independence? I'll leave that question hanging and concentrate on the facts and figures.

Great Scot!

Power is defined as energy flowing per second. 4.4 GW is a lot of power. It's roughly equal to each of these:

2 million standard kettles boiling
600,000 cars driving at urban speeds
4.4 million 1 kW electric bar fires
Raising all water in Loch Lomond at 60 cm per hour*
Two flux capacitors from Back to the Future's time travelling Delorean

* EDIT 30/3/16 — Originally I had this as 4 mph, but I'd got my sums wrong.

It is about a third of Scotland's current total generation capacity when the wind is blowing, and about half of it when it isn't. In terms of the entire GB grid, it is about 6% of total capacity.

Although there are plans for more generation capacity in Scotland by 2023, it's hard to estimate exactly how much because it depends on private companies deciding to invest and planning permissions being granted. However, the Scottish Government's target is to have 100% of gross consumption in Scotland (that's electricity, not deep fried Mars bars) equalled by renewable generation by 2020. By my estimates, this means Scotland will need to add 8 GW capacity of renewable generation by then, and almost all of that would be from new wind farms.

What follows assumes this ambitious target will be met before 2023, and if it isn't my estimates involving wind power would need to be reduced.

Generation capacity

This graph shows Scotland's electricity generation capacity in 2015 and my estimate for 2023 assuming the Scottish Government's renewables target is met by new wind farms and all coal and nuclear capacity is lost.

Source: compiled from the Energy in Scotland 2015 Scottish Government report and DUKES 2015.

Some simple arithmetic tells us that overall capacity will have increased by 8 - 4.4 = 3.6 GW. On the graph, this corresponds to the removal of the black and orange segments at the base of the 2015 column, and expansion of the blue-grey segment at the top.

But there is of course a crucial qualitative difference between the 4.4 GW we are losing and the 8 GW we are gaining. Wind capacity of 8 GW is the maximum we can expect when the wind is blowing. In the UK capacity factors for wind farms are about 30%, so we can, on average, expect 2.4 GW from this extra 8 GW of wind farm capacity. This leads us to the question: will having an average of 2 GW less generation mean we need to import electricity on days of low wind?

Meeting demand

Averaged throughout 2013, the latest for which figures are given, demand was 3.7 GW. Scotland's peak demand is about 5.5 GW. This will most likely occur on a cold winter evening at around 6pm when people are returning from work. Although not stated explicitly, these figures can be estimated from the total consumption figures and seasonal variations given in Chapter 3 of EIS.

Electricity demand has declined in recent years, in part due to gradual improvements in efficiency (e.g. use of low energy and LED light bulbs) but there was also a noticeable drop in non-domestic usage due to the recession post-2008. I've not found any clear predictions for Scotland in particular, but according to the National Grid's Future Energy Scenarios report the GB grid's peak energy demand is forecast to rise from about 55 GW now to between 59 GW and 66 GW in 2030. With this in mind, and for simplicity, I'll use the above average and peak demand figures for Scotland in 2013 in what follows.

For 2015, the above graph's data shows that Scotland can supply 7.9 GW if zero energy is generated from wind. This assumes all non-wind sources are at maximum capacity and there are no losses. If we allow for 15% losses (see gross vs total consumption figures), and reduce capacity by another 10% to reflect reduced capacity due to faults and maintenance, then the supply available to meet demand is about 6 GW. So at present, even in this worst case scenario — zero wind, highest demand — Scotland can meet its own consumption and export 0.5 GW elsewhere.

For 2023, the non-wind supply figure drops to 3.5 GW, which becomes 2.6 after deducting for losses and other factors. About 0.9 GW of imports are needed to meet average demand, and 2.9 GW to meet peak demand.

Of course, these figures are the worst case scenario on windless days. In 2023, my estimated capacity for wind is 12.2 GW and applying the 30% capacity factor gives 3.7 GW and deducting 15% losses gives 3.1 GW. So total supply will be 5.7 GW which is enough to meet peak demand and export 2 GW on average.

Nevertheless, we'll have moved from a situation now where we're almost always able to export, to being dependent on imports at times of high demand and low wind. (I might come back to how often this might be in a future blog post.)

Change, but for better or worse?

I am not alone in coming to the conclusion that Scotland could well become reliant on some degree of electricity imports in coming years. The Institution of Civil Engineers arrived at much the same conclusion. But is this necessarily a bad thing?

Firstly, does the GB grid have sufficient generation capacity in years ahead? According to ofgem's Electricity security of supply report, the winters of 2015/16 and 2016/17 do give some cause for concern, though a number of measures should ensure a sufficient supply in all but the very worst case scenarios. From 2017/18 onwards, the situation improves as mothballed power stations are returned to service and new power stations come online.

From the point of view of the GB grid as a whole, Scotland's move away from electricity exporting to a position of balance with occasional importing makes some sense. In 2013, 28% of electricity generated in Scotland, some 14 TWh, was exported to the south of the GB grid. Whilst this may be of economic benefit to Scotland, it is wasteful in that energy is being lost along the hundreds of miles of wires. It makes sense to locate generation closer to the demand.

There are environmental benefits too. Longannet is an inefficient, coal powered station and its closure will significantly reduce Scotland's greenhouse gas emissions. Some may also welcome the fact that nuclear power is being banished from Scotland, though we'll still be dependent on the ones in England when importing and, to small extent via a 2 GW inter-connector, on the many nuclear power stations in France. The big downside of course is that the closure of these three large plants will involve the loss of many jobs. It is vital that action is taken to help those people affected.

There are some infrastructure improvements that would fit very well with the Scottish Government's renewables strategy, their politics and the overall needs of the GB grid. Specifically, the construction of more pumped storage facilities. These use excess electricity to pump water to a high level reservoir during times of low demand, and can later use that water to spin up turbines in a matter of minutes to meet large jumps in demand. At present we have the Foyers pumped storage facility at 0.3 GW, which dates back to 1896 for aluminium smelting, though much upgraded since, and the ground-breaking and mountain-hollowing Cruachan facility at 0.44 GW. I hope proposals to expand Cruachan or build a new one at Coire Glas are progressed.

I very much doubt the current Scottish Government's intention was for Scotland to become reliant on imports, and consqeuently more dependent on decisions made in England and Wales. Nevertheless, by prioritising the renewables target over replacing traditional power stations, that seems to be the end result. But, viewed across the GB grid as a whole — and for reasons of infrastructure and the environment — I have to conclude that the coming changes are, in the round, positive.

Scottish public funding - fair or needed?

2015-09-23T16:20:00+01:00

A report on local government funding was published yesterday by the Scottish Parliament Information Centre (SPICe). The BBC and the Scotsman, amongst others, reported on these two findings:

The funding of local government as a percentage of the central Scottish government budget has not significantly decreased.
The Scottish government has adequately funded — in fact "over-funded" — councils for its council tax freeze.

My previous analysis of data from the same source came to a conclusion that apparently contradicts the first point. Also, I wasn't convinced on the second point because it relies on a comparison with council funding growing with inflation (RPI), but does not address growing population with growing needs due to aging and schooling. What follows is my interpretation of the data.

Local government funding

Local government funding refers to the money given to the councils by the central Scottish government. What local government can spend is determined by this amount plus what they collect in council tax (and a small amount of revenues and funding from other sources).

Source From the spreadsheet associated with the SPICe Local government finance 1999-2016 report. Note 2013-14 is estimated, and 2014-15 and 2015-16 are budgeted values.

This graph shows the funding of local government from central government over the history of the Scottish parliament. The figures are in real terms, i.e. inflation adjusted to be in 2015-16 prices. For the first decade local government funding rose, peaking at £13.0bn in 2009-10. It then fell to a minimum of £11.5bn in 2013-14 and is budgeted to rise to £11.9bn in 2015-16.

From 2013-14 police and fire services were brought under central government control and so for that financial year onwards approximately £1.5bn of local government funding is included so we can easily make comparisons with the past.

Impact on council services

The graph tells us that real terms local government funding was cut by over £1bn in recent years and is now sitting at the level it was at in 2004-5 — about £12bn. Since then the population of Scotland has increased by 280,000 or 5%. Not only that, but a large cohort of people are moving into old age putting a particular strain on council services. For example, the number of people aged 90 or over was 27,808 in 2004 and 41,838 in 2015, a rise of 50%. Demand for education is increasing too, with the number of children aged under 10 having risen by 5%. These data are from the National records of Scotland.

To keep services at a constant level given these demographics council spending should have risen, perhaps by 5% or even more given the aging population and their care needs. The fact that there has been no increase is likely to cause resources of councils to be stretched, increasing the load on staff such as teachers and carers.

Percentage of central government spending

The SPICe report looked at local government funding (shown in the last graph) as a percentage of the Scottish government budget. Importantly, they didn't calculate the percentage of the total budget (known as Total Managed Expenditure or TME), but the sub-total over which the Scottish government has devolved control (referred to as DEL+NDRI). They found that this percentage did not significantly decrease over time. This means that the central Scottish government funding of local government has not lost out in its share of the budget compared to other areas, such as health or justice. The blue line on the graph below shows these percentages — they are identical to those provided in the SPICe report (Figure 1 and Table 1).

Source Local government and DEL+NDRI data are from the spreadsheet associated with the SPICe Local government finance 1999-2016 report. Data on TME is extracted from various Scottish Government draft budget documents. No single source of Scottish TME is available and I couldn't find figures for 2001-02 and earlier. Note 2013-14 is estimated, and 2014-15 and 2015-16 are budgeted values. Full detail can be found in this spreadsheet: download as ods or xlsx.

The red line on the graph plots the local government funding (shown in the first graph) as a percentage of TME. Specifically, it is the amount shown as the "Total Scottish Government Budget" in each year's draft budget. Despite the name "draft" these budgets are definitive and are in fact approved by a vote on a Bill at the Scottish Parliament. To see these figures in full context see, for example, Annex G in the 2015-16 draft budget document.

As you can see, the choice of how you measure the Scottish government budget has a noticeable effect on the trend in the percentages: unlike the blue line, the red line has been decreasing since 2006-07.

The SPICe conclusion based on the blue line was (page 3):

"Over the period 1999 to 2016, local government's share of the Scottish Government budget...has decreased by only 0.2% to 36.0%."

My conclusion based on the red line is:

"Since 2006-07, local government's share of the total Scottish government budget has been decreasing. It was 35.8% in 2002-03, peaked at 37.6% in 2005-06 and in 2015-16 is expected to be 31.9%."

There's actually no contradiction here though — we need to understand what these two sets of percentages are measuring.

Fair or needed?

The SPICe report's conclusion shows that the Scottish government's allocation to local government is fair in the sense that it is historically consistent with what they and previous governments have done in allocating the budget under their devolved control. This means that the UK government cuts are reflected in the Scottish budget (specifically in DEL) and those cuts are passed on proportionately to the councils.

The percentages I've provided are of the total Scottish budget, or to give it's full title, the Total Managed Expenditure (TME). TME is larger than the measure of the Scottish budget that the SPICe report used (DEL+NDRI) because it includes money that's earmarked for specific purposes such as welfare, tax credits and public sector pensions — this amount is called Annual Managed Expenditure (AME). AME is demand-led which means that the TME total reflects the needs of the population.

This allows us to resolve the apparent contradiction with the percentages. The decreasing red line on the above graph is telling us that council funding from central government is not reflecting the increasing needs of the population. This accords with the points made above about increased demographic pressures on council budgets. The fact that the blue line is not decreasing just tells us that the Scottish Government is allocating budgets in the same way as has been done in the past.

Political comment

I believe we should judge a government on how it serves the changing needs of the population and not on historical consistency in spending patterns. And in making this judgment we should realise that power is being centralised in the Scottish government: obviously in the case of the police and fire services, but also implicitly by using a council tax freeze to control what local government can raise and spend. Although the current funding cuts are due to the austerity policies of the UK government, the Scottish government is not using its powers to alleviate them and is preventing local government from using its powers by imposing the council tax freeze.

Corbyn: My evidence-based prediction of impact on UK politics

2015-09-12T16:20:00+01:00

I have no idea.

See also this animated visualisation and this spreadsheet of data.

Scotland's age distribution

2015-09-09T19:20:00+01:00

I've been looking at the age distribution of people in Scotland and made this animation to visualise trends between 1982 and now - a period of 33 years.

Source: National records of Scotland

Additions to the population are highlighted in blue and people leaving Scotland are in red, including those who have shuffled off this mortal coil.

The conveyor and baby boomers

The graph appears to be on a conveyor belt, with features remaining recognisable as they move from left to right. This is of course because people age by one year every year: this year's 31 year olds were last year's 30 year olds.

The two most notable features are a broad hump aged around 20 in 1982, who end up in their 50s in 2015, and a narrow spike of people around age 34 in 1982 and 67 in 2015. These correspond to baby-boomers born in the late 1950s and 1940s respectively.

The famous late 1940s baby boom is associated with the ending of World War 2 but there was also a similar one after World War 1. You can see this as a "cliff" at age 62 in 1982. This cohort reached age 90 in 2010 causing a big jump in the 90+ bar.

Notice there are gaps between peaks in age. In 2015, 12 and 13 year olds are rarer than any other ages under 70.

Birth and death

The height of the bar for age 0 is shown in blue because this is always an addition to the existing population. It reflects the annual birth rate which rises and falls, but has an overall decreasing trend over this time period. For any age above zero, the population will decrease slightly from year to year because of deaths: sadly, not all 30 year olds make it to 31.

The death rate becomes higher at older ages and so the end of the distribution is sloping and shows a strong red fringe. You'll notice that the height of the death slope is higher in 2015 than in 1982, i.e. there are more older people now than 33 years ago. In particular, the number in the 90+ category have more than trebled in the last three decades.

Migration

The broad, baby-boomer hump that's at young adulthood in 1982 shows a strong red fringe. It continues to decrease through the 1980s before stabilising by the mid-1990s. This decrease is due to net emigration, i.e. more people leaving Scotland than arriving.

From the mid-1990s onwards you can see evidence of net immigration as a blue fringe for ages 20-30. This becomes more prominent through the 2000s to now.

Some details

When a red bar is present the population for that age is equal to the height of the grey bar only. The red bar represents the decrease, specifically, its height is last year's population at age n-1 minus this year's at age n.

For all other bars the population is the height of the grey bar plus the blue bar, where the height of the blue bar is this year's population at age n minus last year's population at age n-1. The bar for 90+ is always grey.

I made the animated gif using the excellent and venerable gnuplot. The gnuplot script, some java code for pre-processing and the CSV file it used (data from NRS link above) can be found in this zip file.

A simple argument against austerity

2015-08-30T19:20:00+01:00

The argument for austerity goes like this:

The public debt is too high so businesses and consumers lose confidence which hampers economic growth.
If the government is seen to be reducing public debt then confidence will be restored.
Confident business will invest and become more efficient, and sell more products and services because of consumer confidence.
Hence the economy will grow.

What follows is my attempt to simplify the argument against austerity to its essentials.

Public and private sectors

The public sector of a country will spend into the country's private sector for many reasons, including paying a nurse, building a school or constructing a warship. Let's call the total spent in a year G.

Private sector funds flow to the public sector as taxes such as the VAT when a car is purchased, the income tax when a person is paid and corporation tax on a company's profits. Let the total amount of a money that flows from the private to the public sector in a year be T.

Since G goes into the private sector and T comes out, the increase in private sector wealth W is

Change in W = G - T

and the shortfall in the public sector — the budget deficit — is met by borrowing, i.e. increasing the public debt D, so

Change in D = deficit = G - T

In plain English, the above means that the increase in private wealth is equal to the deficit. Conversely, if the deficit is negative — a surplus — the private wealth must decrease by an amount equal to the surplus.

The decrease in private wealth is equal to the surplus.

This is an accounting identity. There's no clever (or stupid) economics going on, just simple arithmetic. Also, this argument does not assume anything about growth (or inflation).

The aim of austerity is to reduce the public debt. To do this you must first reduce the deficit and in time create a surplus. The surplus can then be used to "pay off" the debt. This entails either a cut in public spending G or a rise in taxes T. Either way the effect on private wealth will be the same:

Austerity will cause private wealth to decrease.

And this is the crux of the argument. A drop in private wealth will cause consumers to spend less on products and services, and this will in turn reduce the revenues of businesses who will also be less likely to invest for the future. This is a recipe for recession, not economic growth.

Paul Krugman criticises austerity saying that it requires a belief in the confidence fairy. If the confidence fairy does exist then its magical powers lie in persuading businesses to invest more, and consumers to consume more as they get poorer.

Austerity in the UK

Growth is currently respectable in the UK economy. This is because the government is nowhere near achieving its desired annual surplus which, according to its own plans, should've happened by 2015. Wages adjusted for inflation are only just back to their pre-recession levels and so the growth must in part be fueled by consumers funding their purchases with debt, which is consistent with the rise in overall private debt. It is also possible that some consumers are funding their purchases by spending their savings (dissaving in economist lingo). Neither is sustainable.

In discussing the private and public sector balances above I didn't include the effect of imports and exports. I won't get into that here, but the bottom line for the UK is that the private sector is currently having its wealth drained because the UK spends more on imports than it receives from selling exports. So if the surplus is achieved this will be in addition to the wealth drain due to foreign trade.

Public debt isn't just debt

Separate to the argument just made, there is a more fundamental point that fells the case for austerity at its base: high public debt isn't necessarily a problem for a country like the UK that has its own currency and central bank. This blog post by Frances Coppola explains this point very well.

Inequality mathematics

2015-07-29T13:47:00+01:00

This post describes the mathematics of inequality in income distributions. It is sufficiently abstract that it's applicable to wealth and many other distributions. You can find the ideas explained without mathematics in this post over on the Rational Intuition blog. The essential point is that commonly stated income fractions, such as

The top 30% of the population in terms of income receive 51% of the total income.

make for poor measures of inequality because they are insensitive to changes in the distribution. The mathematical reasons for this are given at the end of this post.

Definitions

Consider an income distribution $f(x)$. The fraction of the population in income bracket $(x,x+dx)$ is given by $f(x)dx$ and so we have

\begin{equation} \int_{-\infty}^\infty f(x) dx = 1 \label{population} \end{equation}

This admits negative incomes, but doesn't require them as $f(x)$ can be set to zero for $x<0$ if desired.

The mean income of this population is given by $\mu$, defined as

\begin{equation} \mu = \int_{-\infty}^\infty f(x)x dx \label{mean} \end{equation}

The standard deviation of the population is given by $\sigma$:

\begin{equation} \sigma^2 = \int_{-\infty}^\infty f(x) (x-\mu)^2 dx \label{sigma} \end{equation}

Let $x_n$ be the upper boundary of decile $n$, then we have

\begin{equation} \int_{-\infty}^{x_n} f(x) dx = \frac{n}{10} \label{decile-n} \end{equation}

and also define $\mu_n$ as

\begin{equation} \mu_n = \int_{-\infty}^{x_n} f(x)x dx \label{mean-n} \end{equation}

This is the mean of the group with $x<x_n$, i.e. all deciles up to and including $n$. The fraction of income of this group out of the total population's income will therefore be $p_n$

\begin{equation} p_n=\frac{\mu_n}{\mu} \label{fraction-n} \end{equation}

Income groups

We can divide the population into three groups in terms of income. The low group will have $x<x_3$, i.e. the bottom 30%, and the high group will have $x>x_7$, i.e. the top 30%. The remainder form the middle 40%. We can define the fraction of income going to these groups as follows:

\begin{eqnarray} p_L &=& p_3 \nonumber \\ p_M &=& p_7 - p_3 \label{p-groups}\\ p_H &=& 1 - p_7 \nonumber \end{eqnarray}

Note that by definition

$$ p_L+p_M+p_H=1 $$

Uniform distribution

For a uniform distribution which is zero except in $(a,b)$ we have from (\ref{population}) that

\begin{equation} f(x) = \frac{1}{b-a} \end{equation}

and from (\ref{mean}) we have the mean

\begin{equation} \mu = \frac{a+b}{2} \label{uniform-mean} \end{equation}

and from (\ref{sigma}) we have the standard deviation

\begin{equation} \sigma = \frac{b-a}{2\sqrt{3}} \label{uniform-sigma} \end{equation}

The upper bound for decile $n$ from (\ref{decile-n}) is

\begin{equation} x_n = a + \frac{n}{10} (b-a) \end{equation}

and $\mu_n$ from (\ref{mean-n}) is

\begin{equation} \mu_n = \frac{n}{20} \left[\left(2-\frac{n}{10}\right)a + \frac{n}{10}b\right] \label{uniform-mean-n} \end{equation}

From this and (\ref{uniform-mean}) and after a little algebraic reduction we can write out the fractions of incomes going to the three income groups in a form convenient for calculating percentages:

\begin{eqnarray} p_L &=& \frac{51a + 9b}{100(a+b)} \nonumber \\ p_M &=& \frac{40}{100} \label{uniform-p-groups}\\ p_H &=& \frac{9a + 51b}{100(a+b)} \nonumber \end{eqnarray}

Notice that $p_M$ is always 40%, independent of $a$ and $b$. The most extreme equality occurs for $a=0$ for which the trio of percentages will be 9%/40%/51%, independent of $b$.

We can re-express these in terms of the mean and standard deviation using (\ref{uniform-mean}) and (\ref{uniform-sigma}):

\begin{eqnarray} p_L &=& 0.3 - \frac{21\sqrt{3}}{100} \frac{\sigma}{\mu} \nonumber \\ p_M &=& 0.4 \label{uniform-p-musigma}\\ p_H &=& 0.3 + \frac{21\sqrt{3}}{100} \frac{\sigma}{\mu} \nonumber \end{eqnarray}

Notice that as $\sigma/\mu \rightarrow 0$, $p_L$ and $p_H$ both tend to 0.3.

Gaussian distribution

For a Gaussian distribution with mean $\mu$ and standard deviation $\sigma$, (\ref{population}) gives

\begin{equation} f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} \end{equation}

and $\mu_n$ from (\ref{mean-n}) is

\begin{equation} \mu_n = \frac{n}{10}\mu - \sigma^2 f(x_n) \end{equation}

which leads to the following income fractions:

\begin{eqnarray} p_L &=& 0.3 - \frac{\sigma}{\mu} g \nonumber \\ p_M &=& 0.4 \label{gaussian-p-musigma}\\ p_H &=& 0.3 + \frac{\sigma}{\mu} g \nonumber \end{eqnarray}

where $g=f(x_3)\sigma = f(x_7)\sigma$ is a constant independent of $\mu$ and $\sigma$. By solving (\ref{decile-n}) for $x_3$ (using the inverse of the error function erf) we find that $g=0.348$ to 3 decimal places.

Comparison

For both uniform and Gaussian distributions $p_M$ is 40%. For the uniform distribution, the low and high fractions are:

$$ p_\frac{H}{L} = 0.3 \pm 0.364 \frac{\sigma}{\mu} $$

and the equivalent for the Gaussian is very similar:

$$ p_\frac{H}{L} = 0.3 \pm 0.348 \frac{\sigma}{\mu} $$

For a uniform distribution starting at zero ($a=0$) we have $\sigma/\mu=1/\sqrt{3}$ which gives a trio of percentages for the Gaussian as 10%/40%/50% which is only slightly different for the uniform case we saw above with 9%/40%/51%.

Symmetry invariance

If $f(x)$ is symmetric about the mean then $p_M=0$. This can be proved as follows. First, rewrite (\ref{mean-n}) as

$$ \mu_n = \int_{-\infty}^{x_n} f(x) \mu dx + \int_{-\infty}^{x_n} f(x) (x-\mu) dx $$

Then, because $\mu$ is constant, we can use (\ref{decile-n}) to replace the first integral to get

$$ \mu_n = \frac{n}{10}\mu + \int_{-\infty}^{x_n} f(x) (x-\mu) dx $$

This is the general form of expressions derived in (\ref{uniform-p-musigma}) and (\ref{gaussian-p-musigma}).

\begin{eqnarray} \mu_7-\mu_3 &=& \frac{7-3}{10}\mu + \int_{-\infty}^{x_7} f(x) (x-\mu) dx - \int_{-\infty}^{x_3} f(x) (x-\mu) dx \nonumber \\ &=& 0.4\mu + \int_{x_3}^{x_7} f(x) (x-\mu) dx \nonumber \\ \end{eqnarray}

First, note that $f(x)$ is symmetric about $x=\mu$ and so $f(x)(x-\mu)$ is anti-symmetric about $\mu$. The symmetry also means that $x_7-\mu$=$\mu-x_3$ and so the integral must be zero. From (\ref{p-groups}) and (\ref{fraction-n}) we now arrive at the result

$$ p_M = 0.4 $$

Scale invariance

Suppose we create a new distribution $g(y)$ from scaling $f(x)$ as follows:

$$ g(y=\alpha x)\alpha = f(x) $$

then it follows that all income fractions for $g$ will be the same as those for $f$, i.e. $p_n$ and so $p_L$, $p_M$ and $p_H$.

The new distribution can be thought of as that obtained if everyone in the original population has their incomes increased by a factor of $\alpha$. Note that $\alpha$ appears on the LHS above so that $g(y)$ satisfies (\ref{population}).

It can be shown from (\ref{decile-n}) that the decile boundaries $y_n$ for this new distribution are given by

$$ y_n = \alpha x_n $$

and from (\ref{mean-n}) the means $\nu_n$ for this distribution are given by

$$ \nu_n = \alpha \mu_n $$

since this is true also for the mean of the whole distribution, i.e. $\nu = \alpha \mu$, then we have

$$ \frac{\nu_n}{\nu}=\frac{\mu_n}{\mu} $$

and from this it follows that all income fractions for the scaled distribution will be the same as the $p$ values for $f$.

Insensitivity to distribution shape

The income fractions involve integrals over $f(x)$ and so are only weakly dependent on how income is distributed within deciles or income groups. This is why the income percentages for the same $\sigma/\mu$ ratio are so similar for the Gaussian and the uniform distribution even though the two distributions have significantly different functional forms.

All symmetric distributions will have the form

\begin{eqnarray} p_L &=& 0.3 - \frac{\sigma}{\mu} g \nonumber \\ p_M &=& 0.4\\ p_H &=& 0.3 + \frac{\sigma}{\mu} g \nonumber \end{eqnarray}

where $g$ is typically around 0.36 for many realistic distributions. Note that $p_L+p_H=0.6$.

Real income distributions are asymmetric and so $p_M$ will not equal 0.4. Usually they are skewed so that the distribution peaks at incomes below the mean and has a long tail to large $x$. This can cause $p_M$ to be less than 0.4, which in turn means that $p_L+p_H$ must take on a higher value than the 0.6.

Poverty

2015-07-23T16:09:00+01:00

A very pleasant Friday afternoon. My son and I went to the cinema, then had a good lunch at a cafe, with amazing baked-on-premises cake, and then browsed a few shops, coming away with a box set of Alien films and a Dr Who and the Web of Fear DVD. That all cost £45.

As we walked down Byres Road in Glasgow we passed a beggar sat by a bin. My son asked me for some money and I gave him a pound and he went back and dropped it in his cup. I went back too and said hello to the guy. This was repeated twice more as we walked a few hundred metres down the road.

I once asked a friend of mine who works with homeless folk, particularly those with drug problems, whether giving money helped them. She wasn't sure, but assured me that, whether I did or not, acknowledging their existence would be appreciated. Begging, she said, can leave you feeling invisible and sub-human.

I asked my son to consider why these people were begging on the street. He speculated that they had been made redundant, lost their jobs or had never had a job. I wondered about going back to ask the last guy we'd met, but decided against it, partly because I didn't want to intrude, but also because, quite selfishly, I didn't wish to risk being drawn in. The more I think on it, the more the latter seems like the main reason.

It's been bouncing around in my mind all afternoon. On another day, I'd have felt a pang of guilt then dismissed it, but perhaps I'm sensitised to it because I've recently been researching data on income distributions and poverty. (It's interesting that data can affect your mental state.)

There's one graph in particular that has grabbed my attention. What it contained was so surprising that I almost missed it. It shows the weekly income distribution for people in the UK, after tax, including benefits but before housing costs are deducted:

Source: HBAI data from the UK Government's Department of Work and Pensions.

Can you see it? Look next to the axis at the left of the graph: the first bar is for the income range £0 to £10 per week. It has about 360,000 people in it. The next band - £10 to £20 per week - has less than a tenth that in it. How can this be in a rich country like ours? Why aren't these people receiving benefits? Do they choose not to? Is their plight so bad they can't even make a claim? Or are they ineligible? I don't know the answer to these questions, but intend to find out.

Notes on Piketty's famous book

2015-06-17T21:20:00+01:00

Here are some rather raw notes on my reading of Thomas Piketty's famous book Capital in the twenty-first century.

Part one - Income and capital

Chapter 1 - Income and output

Domestic output equals GDP less depreciation; depreciation is significant, typically around 10% of GDP.
National income is equal to domestic output plus net income from abroad, i.e. returns on capital owned inside the country but located elsewhere. Net income is near zero for most developed nations because other countries own assets located there.
Global income must equal global output, by definition (unless Martians are involved!).
Capital is defined by Piketty as any form of wealth that can be owned. Capital and wealth are used interchangeably by Piketty.
Human capital is excluded.
National income is equal to income from capital plus income from labour.
National wealth (or capital) can be split into as public plus private wealth, or alternatively domestic plus foreign wealth.
The capital-income ratio is given the symbol β
β can be thought of as the number of years of a country's income held as wealth, so β=6 means a country holds six years worth of income as its wealth.
The income from capital as a fraction of national income is denoted by α.
α = r × β is an accounting identity and follows from the definition of r.
i.e. income from capital equals r times amount of capital; then divide both sides by national income to get above equation.
Figures 1.1 and 1.2 contrast shares of world output and population respectively for each major continent.
Europe and America's share of output greatly exceeds their population share, at expense of Asia and Africa.
Piketty uses Purchase Power Parity dollars (PPP) to convert currencies. More stable, and gives a more realistic estimate of inequality.
PPP accounts for what a given amount of currency can buy in a country, rather than how many dollars it can be exchanged for, e.g. 100 USD will buy much more in a poorer country than it would in the US.
National income distribution is more unequal than output distribution because wealthy countries are more likely to own capital in poorer countries and so draw a return from that capital.
Developing nations have caught up with richer ones because of diffusion of knowledge not because they received capital investment from rich countries; in fact, China and South Korea have protected themselves from foreign capital flows, and competition from foreign companies in order to allow native companies to grow and catch up.

Chapter 2 - Growth: Illusions and reality

Growth can, with some difficulty, be decomposed into a demographic growth component (growth with population) and an economic growth component. Only the latter results in improved living standards.
Global growth is about 3%, whereas population growth is 1%.
Table 2.1 shows global growth was only demographic at 0.1% before 1700, and dramatically increased due to large economic component over the industrial revolution.
Cumulative effect of growth over time needs to be appreciated: 1% growth over a year seems small but equates to 33% growth of thirty years (a generation).
Demographic growth has been very high since 1700, but now shows signs of slowing.
High demographic growth dilutes wealth-concentration effect of inheritance.
Output per capita, called productivity, has increased dramatically since industrial revolution.
This means purchasing power of average person is typically ten times higher now than a hundred years ago.
Productivity gains have mainly been in production of goods due to automation, whereas it's hardly changed in services, hence latter's growth gathers an increasing share of the workforce.
Typical growth in developed countries in 20th century was 1 to 1.5%; 3 to 4% in poor nations that are catching up.
Growth rate of 1% must involve significant social change.
Fig 2.3: Europe growth was high post WW2, but this ended when it caught up with US in the 1970s.
Policies of Thatcher and Reagan in 1980s didn't seem to affect growth.
Population growth peaked earlier in the 20th century, before economic growth.
Fig 2.4 and 2.5 shows growth falling in 21st century.
Growth rates quoted in Piketty are real, i.e. inflation corrected.
Growth brings new and changing products, which makes it hard to assess changing prices, and so effect of inflation on growth.
Price changes can drive inequality, if owners of certain items see a jump in their value.
Inflation erodes debt.
There was almost no inflation before WW1.
Novels no longer refer to amounts of money due to inflation.

Part two - The dynamics of capital/income ration

Chapter 3 - The metamorphoses of capital

Fig 3.1 Agriculture now negligible, national capital to income ratio high and climbing.
Foreign capital owned by Britain and France fell post colonial times.
Public net wealth is small compared to private net wealth.
Public debt is owned (not owed) by a minority and so important to inequality
I find the net measures in this chapter misleading. In particular, public assets cannot be sold, so what's the sense in offsetting them against debts?

Chapter 4 - From old Europe to the new world

Fig 4.4 Germany's relatively low private capital is due to apparently lower stockmarket valuation of its firms, since it runs a stakeholder approach to firms: eg representatives for workers, community, gov and environmental groups sit on boards of directors. Rhenish capitalism/social ownership.
Fig 4.6 USA national capital much more stable than Europe from 1790 to now.
In fig 4.10 ownership of slaves is included and makes up about a quarter of national capital at its peak. Even though it goes to zero by 1880; it makes the profile of national capital vs time even more stable looking.

Chapter 5 - The income/capital ratio over the long runs

Why did capital/income in Europe return to historic highs in late 20th century and why are they higher than for US?
Net public capital is small compared to private capital in all major economies
2nd law β=s/g - in words: the ratio of capital to income (β) is equal to the rate of increase of capital (s for savings) divided by rate of increase of income (g for growth)
This law is true in the long run, not necessarily precisely obeyed at any given time if rapid flucuations are occurring.
National income = GDP - depreciation + net foreign income
Return to higher β is due primarily to small g, but also privatisation and increase of asset prices.
Table 5.1 Higher β in Europe vs N America is due to higher population growth in the latter.
Table 5.3 depreciation significantly reduces gross savings.
Table 5.4 public savings are negative because private savers having been buying government bonds.
Fig 5.6 market value is sum of shares of company ; book value is assets - liabilities. Ratio is Tobin's Q.

Chapter 6 - The capital-labour split in the 21st century

Britain & France have best data.
Fig 6.1 and 2 show both countries have fairly stable split over 2 centuries with labour being 60% to 80% and a noticeable jump up at WW1. Not sure how he sees a U shape in those graphs!
Figs 6.3 and 4 show rate of return 4 to 6% except for spikes after 2 WWs.
The "pure" rate of return accounts for labour expended on managing investments.
Most household wealth is in real assets, eg a house, that have a real role in the economy, unlike money which is a nominal asset, vulnerable to inflation.
Inflation can redistribute wealth and erode debt, but has a lesser role in determining the return on capital.
Marginal productivity is extra output from adding one unit of capital. It decreases as β increases. But since α = r x β, how α (share of income as return from capital) behaves depends on how r varies with β. This is measured by elasticity of substitution, which is when r goes as 1/β which means α stays constant ie any capital-labour split gives the same return α. If zero, labour has to be set according to capital, and α decreases with increasing β and vice versa if elasticity is large.
Evidence is that elasticity is a bit above 1 and so α grows gradually with rising β, share of income from capital increases as ratio of capital to income grows. It was less than 1 in 18th century agricultural economy.
Long term decrease in capital and mirrored increased in labour's share of income suggests human capital is more important, ie knowledge and skills possessed by workers. But traditional capital is still important and has not been displaced by increased human capital.
Marx's prediction that accumulation of capital by few owners would kill the return on capital makes sense if low growth persists (then historic ownership is dominant). Ie income would mostly go to capital, workers revolt etc But he didn't account for growth from increased productivity.
Last page of chapter: "Progress toward economic and technical rationality need not imply progress toward democratic and meritocratic rationality." Possibly true :(
Technology and knowledge have caused increases in growth that have avoided Marx's apocalypse, but not altered the structure of capital, eg the capital-labour split.

Part three - The structure of inequality

Chapter 7 - Inequality and concentration: preliminary bearings

Tables 7.1-3 give broad stats in different regions, at different times of inequality in labour income and capital.
If top 10% have 20% of income, by mathematics, their average salary is twice that of whole population average.
All figures are pre-tax here.
In Inequalities wrt capitalism section: "wealth is so concentrated that a large segment of society is virtually unaware of its existence, so that some people imagine that it belongs to surreal and mysterious entities."
Inequality of wealth is greater than inequality of income. Europe has "medium" inequality: top 10% has 60%, middle 40% has 35% and bottom 50% has 5%.
Over the last century, inequality has reduced markedly creating the middle class. 100 years ago in Britain: top 1% had >60%, Middle 40% had 5% and bottom 50% had <5%.

Chapter 8 - Two worlds

French case:

Fig 8.1 and 2 - share of income for top decile & percentile in capital dropped sharply during WW2 whereas top wage decile & percentile remained the same.
Inequality changes in the 20th century were not gradual or planned but resulted from war crises and political shocks and changes.
Fig 8.3 and 4 - in France, point where income from capital overtook that from labour moved from about top 1% in 1932 to top 0.5%
In the top decile, the top 1% is different from other 9% - the latter earn their income by working. This kind of effect is not visible if relying on a single number measure of inequality like the gini coefficient.
Share of wages going to the top centile increased significantly from the 1990s in France - high pay packages for top managers.

US case:

Inequality was reduced by policies that froze managers pay during ww2 and boosted the lowest paid.
Inequality remained constant until the late 1970s, after which it increased sharply, mainly due to huge pay to top managers. Spikes in income from capital gains are apparent before the dot com bubble burst in 2000 and before crash of 2008.

Chapter 9 - Inequality of labour income

All wages in 20th century France have been lifted; wage inequality remained constant. In contrast, inequality increased in US.
Both examples plus Scandinavia point to investment in education acting against rising inequality.
Rhenish capitalism involves stake and stock holders, already discussed in part 2.
Regarding imperfect competition and minimum wage: "the goal is to make sure that no employer can exploit his competitive advantage beyond a certain limit".
Last para in minimum wage section is great! "best way to increase wages and reduce wage inequalities in the long run is to invest in education and skills."
In Inequalities in emerging economies section: "deterioration of the tax data after 1990... due in part to... computerised records."
Marginal productivity theory cannot explain supermanagers' pay rises.
Pro business policies from early 1980s onwards included cuts to top rate of tax. As big business leaders/owners get wealthier, their influence over politicians grows, hence more pro business policies and tax cuts. Positive feedback for inequality growth.

Chapter 10 - Inequality of capital ownership

Historical reduction in inequality occurred with the rise of a middle class.
Figs 10.1-6 show wealth inequality in France, Britain, Sweden, US and Europe. All show similar pattern: big drop in inequality during mid 20th century, followed by slow rise thereafter, though US is different in that inequality started lower but ended up higher.
Fig 10.7 shows r>g since 1820.Together with fig 10.8 which plots α and s; it's possible to verify second law of economics.
Fig 10.8 shows r>g since 0 AD.
Figs 10.10 and 11 show that r<g once tax and losses are accounted for, but only in mid 20th century.
Economic models usually use time preference (e.g. whether to spend $100 today or $105 tomorrow?) to explain why r is historically 4-5% and >g. Piketty argues this is tautological and over simplistic.
If g=1% and r=5% then wealthy person will keep pace with average growth of income if they reinvest (save) a fifth of capital income each year.
Laws on entails - default inheritance of eldest son - ended in major economies by early 20th century
Pareto power law distribution of weath distribution is a useful mathematical description
Inequality hasn't returned to 19th levels despite r>g because not enough time has passed and taxation has changed.

Chapter 11 - Merit and inheritance in the long run

Demographic growth is predicted to slow, and since r>g, inequality will rise in 21st century.
French inheritance tax historical record is the "best known".
Fig 11.1 shows France's inheritance was about 20% of national income before WW1; feel to 4% post WW2; now rising through 15%.
Two alternative ways to estimate inheritance: economic (wealth and age distribution) or fiscal (tax). Both give reassuringly similar answers, though latter is less than former, probably because some inheritance avoid taxs (legally or otherwise).
Accounting identity of inheritance: inheritance/national income = deaths per year × (average wealth at death / average wealth of all) × β.
All three of these factors decreased to cause mid 20th century dip in French inheritance flow.
French annual mortality of adults decreased from 2.2% in 1900 to 1.2% in 2000.
Partly explained by baby boom introducing a large cohort of young adults in latter half of century.
Gifts pre-death should be included with inheritance flows as this is now increasingly important.
Older people have generally been wealthier than young adults, except for around WW2.
Piketty predicts share of total wealth that was inherited in France will return to being between 80-90% in 21st century from a low of 45% in 1970.
Fig 11.10 living standards of top 1% inheritors is now roughly equal to that of top 1% of those who derive income from work.
Data for other countries on inheritance is less good, but UK and Germany exhibit similar trends over last century.
Much harder to assess US and developing countries.

Chapter 12 - Global inequality of wealth in the 21st century

Return on capital varies with amount owned - a fact often neglected.
Large funds can expend huge sums on managing the fund, e.g. managing risky, high-yield investments.
Data on individual fortunes hard to assess, or imperfectly estimated, e.g. Forbes magazine.
Bill Gates's fortune grew as a rapidly as a non-working heiress Liliane Bettencourt of L'Oreal, and continued to grow at same rate after he stopped working.
Entrepreneurs turn into rentiers.
Table 12.2 shows returns on University endowments (i.e. their capital) grows most, as high as 10% per annum, for the largest funds. Best data available on such large fortunes and their returns.
Inflation does not affect returns on capital very much.
Sovereign wealth funds: Norway's fund worth $700 billion in 2013, and offers good data, but despite being double that of all US university endowments combined, it is not growing as fast, possibly due to public/democratic scrutiny. Middle east wealth funds comparable in size, but data is more opaque.
Will sovereign wealth funds own the world? Will some countries own other countries?! Will China own the world? Probably not.
Loss of democratic sovereignty as capital and its power shifts across national borders avoiding tax, and promoting a feeling of dispossession of the majority of people who have little capital. Particularly acute in Europe with many small states with no united fiscal regime.
The net wealth (assets less debts) of some rich countries appears to be negative, and indeed the world overall appears to be in debt! "Earth owned by Mars"! The explanation is probably that some private wealth is "invisibly" held in tax havens.

Part four - Regulating capital in the 21st century

Chapter 13 - A social state for the 21st century

Wars of 20th century wiped away past and its structure of inequality.
Tax exposes wealth to democratic scrutiny as well redistributing it.
2008 crash was not as bad as 1929 because governments and central banks stepped in and ensured there was liquidity, ie money still flowed.
Fig 13.1 - taxes were less than 10% of country income until early 20th C. Now stabilised at 30%(US) to 55% (Sweden).
Role of governments is at an all time high in social and economic matters.
Education & health account for 20% of developed country GDP, mix of public & private provision.
Half national income in developed countries is spent on social issues - debate needed about how it's organised.
Social mobility better in Europe than US.
Average income of Harvard parents is $450k; represents top 2% of income distribution in US.
Free access to grand ecoles in France benefits children of parents in top 10% income bracket; free tuition not good enough in itself.
Pensions are currently pay as you go, but now growth is lower, a capitalised pension scheme is needed, but investing now would deprive current retirees.
After 1980, ultraliberal wave from developed countries forced poor countries to cut their public sectors.
Decline of poor countries revenue in 80s due to custom duties
The kind of fiscal and social state emerging in developing world is of utmost importance to future.

Chapter 14 - Rethinking the progressive income tax

Tax is not technical but social and philosophical.
Tax can trigger social unrest eg poll tax,or war,eg American revolution.
Globalisation leans heavily on least wealthy workers in wealthy countries, justifying progressive tax there.
Fig 14.1 and 2 shows high tax rates of mid 20th century.
Taxing high income or wealth at 70 or 80% does not raise much revenue, but it does curb growth of high pay and so inequality.
US and Britain had highest tax rates and distinguished between earned and unearned income.
In mid 20th century top managers in UK or US were less inclined to fight for big rises because 80% or more would just go to government as tax.
Dropping top rate of tax shows no increase in productivity nor GDP growth from 1980 onwards in UK or US compared to Germany.

Chapter 15 A global tax on capital

Global tax on capital viewed as dangerous and idealistic but so was income tax a century ago.
Piketty proposes a progressive annual tax on global wealth.
Tax must cover all forms of wealth, unlike current systems.
Global tax is not about funding social state, but restraining growth of inequality and will force greater transparency of asset ownership.
No technical difficulty to do so: USA does it for hundreds of millions, and FACTA is an attempt to extend reach outside USA
0.1% rate would force transparency without any inequality reduction.
Progressive tax on capital is missing in most countries.
Wealthy capital owners like Gates or Bettencourt cannot spend most of the capital return, so it just gets added to their capital stock and not taxed, without any evasion or breach of the law.
Annual tax on capital of a few percent enforced across nations would be fair and provide a significant extra public revenue. 2% GDP in Europe.
Protectionism offers a possible alternative in some situations. It is useful to protect a developing economy from established foreign competition, eg S Korea or China.
China controls its currency and restricts outflows of capital and foreign ownership.
Unequal distribution of petroleum resource causes wars, eg middle east.
Redistribution through immigration is a partial solution to inequality.

Chapter 16 - The question of public debt

Rich countries have high private wealth but poor (indebted) governments.
Governments of poor countries are less indebted.
Europe has high private wealth but struggles with public debt.
Three methods to tackle public debt: tax on capital, inflation or austerity.
Inflation worked before; austerity is the worst.
What follows applies to Europe.
Public assets equals public debt
National wealth is 6 years of income, mostly owned privately, by households. Half is real estate; half is financial assets, including public debt.
European assets owned by foreign agents equal foreign assets owned by Europeans.
Public assets could be privatised, but then rents would still need to be paid to use them, effectively replacing debt interest, and this would have some socially and morally unacceptable consequences, eg private police.
Simply cancelling public debt may hit the current owners of bonds unfairly; may not be those who purchased bonds during the crisis.
Better to introduce a progressive tax on capital which could bring debt to zero in 10 years.
Inflation can be redistributive but is difficult to control and might have unintended consequences.
Central banks main role is to stabilise the financial system by providing liquidity in times of need.
Central banks don't create capital, but can inject money into the economy by creating an asset and debt simultaneously.
Europe has a currency, but no state, and a central bank, the ECB, that cannot exercise a monetary policy to satisfy 17 states with separate fiscal regimes.
Implement tax according to location of asset rather than where owner lives?
Cross-border agreement on data sharing, interest on debt, taxation is no more Utopian or idealistic than a stateless currency. In fact it's needed by it.
To decide the appropriate level of public debt requires answering the question: what value of β is desirable?
The golden rule r=g provides a theoretical upper bound on β, the amount of capital in units of income.
To achieve this would require that capital is accumulated until r falls to g - requires an unrealistically huge stock of capital.
There is no rational argument behind why Europe has set limits of 3% and 60% of GDP for public budget deficits and debt respectively.
Europe needs a budgetary parliament to go with its single currency.
Public debt is often repressively redistributive; specifically, wealthy lend money to gov and receive interest instead of paying tax.
Debt flexibility is necessary in a crisis, so arbitrary or legislated rules are not wise.
Public and private wealth are at record high levels - the problem is not public debt being too high, but wealth is unequally distributed.
Funding solutions to climate change will impact public debt decisions.
Free market with private companies vs public ownership are two polar opposites - many types of organisations/companies exist in between.
We need more transparent accounts of big private companies if we wish to hold them to account.

Conclusion chapter

Without government intervention our societies will revert to r>g and inequality will grow, i.e. wealth from the past (r) grows faster than the active and productive economy involving output and wages (g).
It is possible to increase growth, investing in education, knowledge and non-polluting technologies. But this cannot raise growth to 4 to 5% per year to compete with r, except temporarily in developing economies.
A hefty income tax may harm motor of the economy (entrepreneurs) and limit growth.
A progressive tax on capital is needed, and this requires a high degree of cooperation between nations. European small countries are a particular challenge.
Last 2 sentences: "...all citizens should take a serious interest in money, its measurements, the facts surrounding it, and its history. Those who have a lot of it never fail to defend their interests. Refusing to deal with numbers rarely serves the interests of the least well-off."

Scottish government cuts?

2015-05-28T20:58:00+01:00

I was writing a blog post on education when I noticed something rather odd about the Scottish government's budget for 2015-16. In the overview I found on the Scottish government's website it states that the budget has been cut "as a result of the UK government's austerity programme". Fair enough, that accords with what I've read elsewhere, and would seem plausible to anyone who's paid any attention to what's been going on in Scotland and the UK. But then I spotted the graph below elsewhere on the Scottish government's website - ignore the colours, look at the total amounts for each year:

Source A screenshot of an interactive graph illustrating the Scottish Budget 2015-16 from the Scottish government website (follow this link to see what the colours mean). Figures are inflation-corrected to be in real terms at 2014-15 prices.

In apparent contradiction to what I'd read, this graph shows a real terms spending rise in 2015-16. So who to believe, the Scottish government, or, err... the Scottish government? Let's find out.

The exact words

The Scottish Budget Bill for 2015-16 was passed in February 2015 and the key document on which it is based is the Draft Budget. An overview is given in Chapter 1, and near the start is this:

Over the course of 2010-11 to 2015-16, the Scottish Fiscal DEL budget has been cut by around 10 per cent in real terms with the Capital Fiscal DEL budget facing a particularly challenging real terms reduction of around 26 per cent as a result of the UK government's austerity programme.

This statement is correct, but notice it refers only to the DEL part of Scottish government expenditure. DEL stands for Departmental Expenditure Limit and it's the part of the budget that's planned ahead of time during a spending review that occurs every three years. As I've discussed before in more detail, the rest of the expenditure is called Annual Managed Expenditure (AME) and unlike DEL it is demand-led and so is harder to plan for ahead of time.

And here's the problem. Usually you'd find that the total expenditure figure is discussed in the overview of a budget. In the jargon this is called Total Managed Expenditure (TME) and TME is equal to DEL plus AME. The UK budget document leads with TME and in fact the very first chart in it is for TME, i.e. all UK expenditure. Only in later chapters does it discuss the DEL/AME split. So why does the Scottish budget give prominence to DEL?

Perhaps mention of TME has been dropped to emphasise the cuts driven by Westminster? Sure enough, when I check back to older Scottish budgets prior to the coalition cuts from 2010 I find that TME was mentioned in the overview, such as this one. The 2015-16 budget does give AME and TME figures but they're tucked away in later chapters and Annexes E and G, and there's no mention of them in the strategic context introduction.

Scotland's Total Managed Expenditure

Here's the data from Annex G on the Total Managed Expenditure in real terms at 2014-15 prices along with its year-on-year change [EDIT 18/8/15: Fraser Whyte pointed out that these real-terms data are not explicitly given in Annex G but can be found in the accompanying spreadheet.]:

	2008-09	2009-10	2010-11	2011-12	2012-13	2013-14	2014-15	2015-16
TME	36,263	37,249	36,916	35,437	35,136	34,393	35,417	36,853
% change		2.65%	-0.90%	-4.17%	-0.86%	-2.16%	2.89%	3.90%

This table matches the data displayed in the graph above. As you can see, although Scotland did experience the coalition cuts from 2010, the last two budgeted years show real terms increases in TME.

Local government cuts

Look again at the graph above, and notice the brown section at the bottom of each column. This is the amount given to local government (i.e. Scottish councils) by the central Scottish government. You can see that from 2009-10 onwards there have been a series of cuts.

According to this spreadsheet on the Scottish government's website, from a peak of £13.1bn in 2009-10, the local government allocation has fallen to £10.6bn in 2014-15 and is budgeted at this level for 2015-16 also. These figures are all in 2014-15 prices, i.e. adjusted for inflation.

So, in real terms, although the total Scottish government's total expenditure is now above what it was in 2008, there has been a £2.5bn real terms cut in what it gives to local government. About £1bn of this occurred from 2013-14 when police and fire services were brought under central control, but that still leaves a real terms cut of around £1.5bn. In addition to this, the 2015-16 budget enforces a council tax freeze for the eighth year in a row, i.e. the tax councils collect from their residents has been decreasing in real terms for eight years.

Political comment

If I take a step back from all the detail above, I see a clear picture in which powers are being increasingly centralised in the Scottish government. I certainly do not agree with the austerity cuts being implemented by the UK's Conservative government, but the Scottish SNP government are hypocritical for making misleading statements about it in their budget whilst forcing substantial cuts on councils in Scotland.

Good on you Linux Voice

2015-05-25T11:45:00+01:00

The year before last I was listening to one of my favourite podcasts and at the end the genial bunch of guys on it announced that it would be their last episode. I think I said WTF out loud. Not only that, the podcast was associated with the magazine Linux Format and they were leaving that too. A bit of a blow for that magazine as they were most of the editorial team.

But they didn't go away. No siree. Soon they announced that they were going to crowd source a new magazine called Linux Voice. They aimed to raise £90,000 and after a month they'd raised a whopping £127,603 from 2347 people. I had a tiny hand in that, by chipping in some cash myself, and by plugging their cause any way I could, including on my own podcast Tuxjam when my co-host Kevie invited Linux Voice's Andrew Gregory on during the fundraising campaign.

As impressive as that success was, I think they topped it because not only have they survived a year (and hopefully many more), but they produced an excellent magazine and they actually made a profit which, true to their word, they shared with a number of free and open source projects as voted for by their readers. Not just that, they've released all content under a Creative Commons license nine months after it was published. To be clear what that means - not only can you read it for free, but more importantly, remix and reuse it to create new content to share with others.

I should declare I have a particular interest in their venture as I'm now listed as a contributing editor. I've written a number of articles over the last year on topics ranging from comet hunting with Python, through to Linux kernel parameter tweaking and writing code for analysing the stockmarket. Doing so has been nothing but a pleasure and I've enjoyed corresponding about these articles with Andrew Gregory along the way.

If you are curious about linux, do go and buy a copy of Linux Voice. It's on the shelf in many newsagents, or you can head over to linuxvoice.com and have a look at free content and if you like it buy a subscription. Their podcast is very good too, and infinitely good value for money (literally, it's free!).

Well done Andrew, Ben, Graham and Mike! (That's in alphabetical order of first name, which they can note down and use next time they get confused in their podcast.)

Pollsters and tossers

2015-05-16T21:15:00+01:00

I've written about pre-election polling quite a lot in the last year and although I promised not to blog about politics for a while, because I've had enough of misleading and inconsistent nonsense from politicians and parties such as... sorry, let me take my anti-rantimine pill... ah, that's better, let me start again.

Polling the voting intentions of a population is vulnerable to many kinds of uncertainty, but one fundamental one you cannot avoid is sampling uncertainty. For a poll of 1000 people, it's commonly stated that the uncertainty involved on the percentages will be ±3%. I'll explain where this comes from, what it means and why it's not always valid. But let's enter into the subject with a simpler example.

Coin tosses

If you toss a coin you can expect an equal chance of it being heads or tails, i.e. the probability of each is a half. So if you toss a coin four times you'd expect, on average, to get two heads or two tails.

Let me pose a question:

If you were to place a bet on the outcome of four coin tosses, should you bet for or against there being two heads?

Most people would say for, but let's work it out properly.

Statistics says that for repeated random outcomes that are independent of each other, you can multiply probabilities to calculate the probability of all outcomes occurring. Since the result of one coin toss does not effect the next coin toss, the probability of getting two heads in a row is a half times a half, or a quarter. So for four tosses, the probability of getting four heads is 1/16. And the probability of getting four tails is also 1/16. In fact, the probability of getting any specified sequence of four tosses, e.g. heads tails tails heads or HTTH for short, is also 1 in 16.

But how can this be? Didn't we say above that getting two heads was more likely? We did, but we said nothing about the order of the results. To calculate the probability of getting two heads, we need to work out all the possible ways that can occur. All the orderings are shown below:

HHTT TTHH HTHT THTH HTTH THHT

There are six ways to get two heads, so the probability is 6/16=3/8. Similarly, we can work out all the ways to get three heads:

HHHT HHTH HTHH THHH

so the probability of getting three heads is 4/16=1/4. And in the same way we can list the probability of getting three tails:

TTTH TTHT THTT HTTT

which, unsurprisingly, also has probability 4/16=1/4. So, to summarise

Result	Permutations	Probability
4 Heads	1	1/16
3 Heads	4	4/16=1/4
2 Heads	6	6/16=3/8
1 Heads	4	4/16=1/4
0 Heads	1	1/16

Notice that the probabilities add up to 1, which reassures us that we've listed all 16 outcomes. Also, the result of two heads (which is the same as two tails) is the most probable line in the table with a probability of 3/8. This accords with what we thought above. But - and this may be a surprise - the probability of not getting two Heads is in fact greater at 5/8.

So, to the betting question above, the answer is that it's better to bet against the outcome being two heads.

Who gives a toss?

It turns out that the probabilities of coin tossing are described by what's known as the binomial distribution. This allows us to calculate probabilities for any number of coin tosses. The graph below shows the distribution for 100 coin tosses. Notice that the probability of getting exactly 50 Heads is only 0.08 - less than 1 in 10.

At large numbers of tosses we can start looking at confidence intervals. For example, by adding up the probabilities of getting from 40 heads to 60 heads, we can state that there's a 0.97 probability of getting a result within that range. If we go up to 1000 tosses then we find that there's a 0.97 probability of getting heads in the range 470 to 530.

Polling

So let's return to polling and imagine 1000 people are contacted out of a much larger population and asked if they supported the blue party or the red party. Let's suppose that people in the population are actually evenly split between these two parties. Then selecting a person at random is much like tossing a coin - there's a 0.5 probability they'll be for the blue party and 0.5 probability they'll be for the red party. And the results of sampling them will be governed by the binomial distribution.

In a sample of 1000, there's a 0.97 probability of finding the number of blue party supporters to be between 470 and 530 (and likewise for the red party). In percentages, this means blue party support is likely to be between 47.0% and 53.0%. This is the origin of the well-known ±3% sampling uncertainty.

Rule of thumb

The 47-53% interval is called a 97% confidence interval because if we were to conduct 100 such polls we'd expect 97 of them to come up with a value in this range. If we increase the sample to 10,000 - a factor of 10 larger - then then confidence interval narrows by a factor of the square root of 10, which is 3.2, which gives 97% confidence interval of ±1%. Likewise, if we reduce the sample to 100, then we increase the interval by 3.2, which makes it about ±10%, just as we saw above with the coin toss, i.e. 50±10%.

So the rule of thumb to estimate the size of the 97% confidence interval is this: divide the sample size by one thousand, take the square root and divide 3% by the result.

Bias

If two polls conducted over the same time period, each with sample 1000, put the blue party on 51% and 49%, then I'd have no trouble in saying they agreed to within sampling uncertainty. However, if the results were 53% and 47% then I would be confident (about 97% confident, in fact) that they disagreed with each other. In this case I'd be investigating both polls for sources of bias.

Biases occur when there's some effect that will cause you to overestimate support for one of the parties. So, for example, a survey that phrases a question in a way that favours the blue party, say

Prof McClever's study shows that 99% of red supporters are cretins. Do you intend to vote blue?

will likely show a bias that overestimates blue's percentage. Another source of bias is in failing to ensure your sample is representative of the population, e.g. by picking a geographical region to survey where the blue party is disproportionately supported.

There's no way to avoid sampling uncertainty for a given sample size, but a good pollster will do everything they can to avoid bias creeping in. In fact, that's very nearly the definition of their job.

An asymmetric, multi-party state

It's probably occurred to you that most polls don't just involve two parties that are equally popular. It turns out that the binomial distribution can easily handle the case where probabilities are not 0.5. If the probability of finding a supporter of the blue party is $p$ then it follows that the probability of finding a red party supporter is $1-p$. It turns out we need to multiply the width of the confidence interval by the square root of $4p(1-p)$ (though there's no easy way to show why without getting into some mathematics relating to the variance of the distribution).

Let's suppose $p=0.7$. This means we expect the larger party to poll 70% of the vote and smaller party 30% and

$$ 4p(1-p)=4\times0.7\times0.3=0.84 $$

and the square root of 0.84 is 0.92. In other words, the confidence interval only narrows slightly, so ±3% is still a reasonable estimate. Even for p=0.9, i.e. one party polling 90% of the vote, the confidence interval reduces to about ±2%. But notice one implication of this: ±2% may be small compared to 90%, but it's relatively large compared to 10%.

The next question is how we deal with more than two parties. Can we appeal to a trinomial or quadranomial distribution? The answer, I think, is yes, but I'm not quite clear on the details. I understand that most of the above results are still applicable but I've yet to convince myself of that directly from the mathematics.

Want to know more?

If you want to get stuck into the mathematics of the binomial distribution then take a look at the wikipedia page. You can also find binomial calculators on the web like this one. You can have a look at the examples for 4 and 100 coin tosses I mentioned above by downloading a spreadsheet, either in ods format or xls format.

General Election 2015 Scotland Results

2015-05-08T11:15:00+01:00

In my last post and the one before I made predictions on overall vote share and seats in Scotland for the 2015 General Election. Here's how they compared with the actual result.

I predicted the SNP would get 43-48% of the vote and Labour would get 30-35%. In reality the SNP did better with 50.0% and Labour worse with 24.3%. This gave the SNP 56 seats out of 59, just above the 40 to 55 range I predicted. My "gun-to-the-head" prediction was an SNP/Labour vote share of 46%/32% with SNP having 45 seats - so quite a bit off the mark (but still better than being shot in the head).

My stab at predicting constituency results fared a little bit better, with all 46 of my highly probable seats going to the SNP. Below is the updated table from my last post showing the remaining 13 seats. They are listed with the hardest wins for the SNP coming first:

All three seats that the SNP didn't win are in the above table, with two of them being at the top of the list. Many of the SNP leads in the above table are slender, especially compared to the enormous leads in many of the other 46 seats.

There is a pattern in how my predictions missed the mark. Actual turnouts in these seats were mostly around 70% whereas I predicted higher turnouts of around 80%. In most cases my prediction of the number of SNP votes was pretty good, but I predicted too many votes going to other parties. The most obvious interpretation of this is that many of the undecided voters who had said they were very likely to vote, simply didn't vote.

Based on arguments in my previous posts, I have some reason to believe many of these non-voters were No voters who perhaps couldn't bring themselves to vote tactically or else felt the SNP surge was so large that their vote would make no difference; certainly true in Ayr and Aberdeen North. However, please regard this as informed speculation in the absence of more direct evidence. Whatever the cause, the result was that although some tactical voting did occur, it didn't have the numbers to beat the SNP lead in most seats.

If you'd like to see how I produced the prediction then have a look at this spreadsheet, available in xls format or ods format. My decisions were mostly on how to implement the method of prediction and very little guessing of numbers or percentages was needed. To do this I drew on the 2010 results, the referendum result in each region and combined them with various pre-election polls and data on the registered electorate. For example, I didn't make a prediction on turnout directly, but instead let that be implicitly determined by voting likelihood from polls together with the changes in electorate between 2010 and 2015.

Personal comment

The SNP success in Scotland was not much of a surprise. The much bigger shock of this election was that the Conservatives achieved a majority. This was not predicted by any of the polls beforehand.

I mean "shock" literally. The prospect of a majority Conservative government in Westminster and the SNP majority government in Scotland is my worst case scenario. For reasons I explain in previous posts, both parties propose policies that I believe will be deeply damaging to the economy of the UK and Scotland, and both parties now have a stronger mandate to pursue them in their respective governments.

This is democracy in action of course, and not for a moment do I question the legitimacy of the result, nor the expressed will of the people. But, given what I've learned about society and the economy since the financial crisis of 2008, I feel very pessimistic about the future of both Scotland and the UK.

Although I feel unrepresented by both governments, I think my family and I will be OK. My concern is more for those people who already have the least; they are usually the ones that suffer the most under the economic contortions brought about by governments driven by ideology.

Resigned despondency is not an option. But is there anything that I could do? I'll let dust settle, spend some time reflecting on the situation and see what floats up from my subconscious. For that to work I'll need to distract myself, so expect non-political blog posts for a while.

General Election 2015 Scotland Constituency prediction

2015-05-07T20:45:00+01:00

In my last post I made a prediction on overall vote share and seats in Scotland for the 2015 General Election. I'm going to stick my neck out even further and make a prediction on which parties will win the Scottish constituencies. I'm prepared to be laughed at quite a lot on this one.

Below is the list of 13 Scottish constituencies that I think can realistically be won by Labour, the Conservatives or the Lib Dems, ordered from most to least likely. The 46 remaining seats have a high probability of being won by the SNP, and the worst result I can envisage for the SNP is that they'd win "only" 30 seats.

Colours on the left indicate the 2010 result and those on the right indicate my 2015 prediction.

The method draws on a number of sources, but does not try and account for the popularity of the incumbent MP. The ICM poll of Nick Clegg's Sheffield Hallam seat conducted the weekend before the election suggests naming the candidate can make a significant difference. If so, I suspect that a few more Lib Dem seats might be held.

My full list of 59 seats in descending order of the chance of a non-SNP party winning can be seen in this spreadsheet.

I may expound on how I arrived at these predictions, but only if they don't turn out to be embarrassingly wrong.

EDIT (7th May 21:50): The four Conservative wins will be much less likely if Labour voters are as reluctant to vote tactically for the Tories as Lord Ashcroft found in his most recent polls. (And yes, this is a bit of last-minute arse-covering on my part!)

General Election 2015 Scotland prediction

2015-05-01T15:14:00+01:00

I'm going to take a stab at predicting the results of the 2015 General Election in Scotland. There are two aspects which make doing so extremely difficult:

how many undecideds are there, and how might they vote?
to what extent will tactical voting take place?

The undecideds

The undecideds are the undead zombies that lurk at the edges of election polling. There's no good way to deal with them so they are usually excluded from headline polls results. If you think about it, this is the same as assuming that either they won't vote or will vote with the same split across parties as the decideds.

The best way to detect undecideds (or don't knows) seems to be through face-to-face polling. The most recent such poll is this one from TNS. It shows that 29% of their sample remain undecided as of mid-April, which is the same as was found in their March poll. The undecideds in the TNS poll say they are as likely to vote as the decideds.

Although we cannot reliably predict how the undecideds will vote, we can assess whether they might have a significant effect. TNS and other polls suggest a 75% turnout from a registered electorate of 4,000,000 people resulting in 3,000,000 votes. This means that 870,000 voters currently remain undecided but are very likely to vote.

Dividing these numbers by 59, the number of seats, tells us that the average number of votes per constituency is 51,000, of which 15,000 are undecideds. Of the 36,000 decided voters, John Curtice's poll of polls suggests that 18,000 will vote SNP and 18,000 will vote for the other parties, of which 9000 will go to Labour.

This tells us that undecideds are important because at 15,000 they significantly exceed the average SNP lead of 9000.

Recent TNS polls contain evidence that undecideds are less likely to be past SNP voters. However, I'm reluctant to rely solely on this because it depends on voter recall of the 2010 and 2011 elections and also there have been big changes in both turnout and voting intentions since then.

A better indication can be found in YouGov's poll conducted between 16-20 April. This poll (page 3) shows that 70% of the undecideds voted No in the referendum, and 30% Yes. A similar split can also be found in a YouGov poll conducted in March.

Let's suppose that undecided Yes voters back the SNP and No voters are spread amongst the non-SNP parties. This assumption could be wrong in either direction and it's important to realise that the prediction hinges on it, but in the absence of better data we have to assume something and this seems most plausible.

If we apply the assumption to the numbers above, we can conclude that the 15,000 undecideds split 5000 for the SNP and 10,000 for non-SNP, which brings the totals to 23,000 and 28,000 respectively. This split is 45%/55% and so tells us that the assumption just made is at least consistent with the referendum result.

Next we need to consider how No voters might distribute their votes amongst parties, and in particular assess tactical voting.

Tactical voting

The 16-20 April YouGov poll gives results for questions about tactical voting (from page 7 onwards). For example, if respondents are asked to imagine voting in a constituency where only Labour or the SNP can win, the SNP/Labour vote share changes from the headline 49%/25% to 50%/38%. It's clear from this that tactical voting can have a significant effect on the vote share, but its impact on the number of seats won by each party requires analysis at the constituency level.

The polls conducted by Lord Ashcroft between 10-16 April in Scottish constituencies show that some element of tactical voting is taking place, but the amount of it appears to vary between the eight constituencies polled. The first question (Q1) asks for a general voting intention, but the second question (Q2) asks you to think about your own constituency; differing answers suggest tactical voting is in play. The figures quoted below are for Q2 with undecideds excluded.

Let's now explore what level of tactical voting is needed to overcome the SNP lead in these seats. I'm not making a prediction (yet), just exploring the numbers to see what's possible.

Four of the constituencies show SNP leads of around 11pp (percentage points), ranging from 9pp to 13pp. Let's treat the poll as a snapshot of what decided voters in the constituency are currently thinking. As we saw above, an average constituency has 51,000 voters, of which 36,000 will be decided and so represented in the poll's headline results, whereas the 15,000 undecideds will not be (because they were excluded or because the poll failed to find some of them). Therefore the SNP lead equates to 11pp of 36,000, which is about 4000. If 5000 undecideds split to the SNP by the day of the election, then 9000 of the other 10,000 non-SNP voters would need to vote tactically to overcome the SNP lead. This seems a bit of a stretch, but let's say this happens in just one of the four seats, mostly like the one that Ashcroft shows has a 9pp lead - this is for Jim Murphy's East Renfrewshire seat, which we'll come back to later.

(I've used the constituency average in the last paragraph for clarity's sake, but the argument remains valid if 51,000 is replaced with any constituency size as long as all the numbers are kept in the same relative proportions).

In another two of the eight constituencies, the SNP has little or no lead, so even a small amount of tactical voting will put a non-SNP party in front. So out of eight seats, tactical voting can give three seats to non-SNP parties.

If we scale the results from these three of eight seats up to all 59, i.e. multiply by three-eights, then we find 22 seats could be won by non-SNP parties. However, as these seats are probably not typical, I'll drop the fraction to a quarter, which gives 15 non-SNP seats. Remember, I do not mean this to be a prediction, just an illustration of a possible scenario.

A prediction

First, vote share. The current poll of polls tells us that 50% of votes will be for the SNP. However, because it (quite rightly) excludes the undecideds, these percentages are likely to shift as they make up their minds. The evidence above suggests that most of them will not vote SNP, and so the SNP vote share will decrease, and the vote for other parties will increase. For this reason, I'd expect the actual election result to be 43-48% for the SNP. Accounting for tactical voting, I'd say 30-35% of votes will go to Labour, with around 20-25% mainly going to the Conservatives and Lib Dems.

The distribution of seats amongst parties is highly sensitive both to the vote share and how tactical voting occurs in each constituency. This makes it very hard to predict, but based on what I outlined above, I expect that the SNP will win between 40 and 55 seats.

If you held a gun to my head and insisted on specific numbers then I'd say the SNP/Labour vote share will be 46%/32% and the SNP will win 45 seats.

Comments

There are a few other imponderables that might affect the result.

One is that the turnout in this election is different from previous ones: 63% in Scotland in 2010; 50% in 2011; 85% for the referendum. This might distort the weighing schemes applied in polls, but I don't see an obvious way to assess its impact.

The spiral of silence effect might be operating. If people feel more comfortable saying they're voting SNP, and reluctant to admit they're voting Labour, then it's likely to affect the polls. This is the effect behind the "shy Tories" in 1992 and "silent No voters" in the 2014 referendum. From the referendum experience, I might guess that the polls shortchange Labour by a few percentage points to the SNP's benefit. Also, instead of completely excluding undecideds, some polls account for this by "returning" a small portion of them back to parties they voted for in a past election. Although I made no attempt to correct for this above, Lord Ashcroft's polls do make a spiral of silence correction in which a percentage point or two is transferred from SNP to other competing parties.

The Lib Dems have published private polling (performed by Survation) in East Dunbartonshire and found that their candidate and MP for the last decade, Jo Swinson, has a slender lead. In contrast, Ashcroft's poll found her to be 11% behind the SNP. This can be explained by the so-called incumbent bonus effect that arises because only the Lib Dem's poll mentions candidates by name. However, Lord Ashcroft countered the incumbent bonus effect by arguing that it's unnecessary to name a locally well-known MP because people will know who you mean in answering Q2. I am a resident of this constituency, and having some local knowledge, I'm inclined to believe the incumbent bonus is real. As surprising as it may be to those heavily engaged with politics, there are many people who cannot name their MP, nor the party they represent. Currently, I believe the result in East Dunbartonshire is too close to call and that a late swing in either direction is likely.

And finally

Just as I was writing the Comments section, Lord Ashcroft published new polls for two of the eight constituencies I referred to above.

For East Renfrewshire he finds the SNP's lead over Jim Murphy has shrunk from 9pp to 2pp, mainly as a result of Conservatives switching their vote to Labour. I also noticed evidence in the tables that some undecideds have made up their minds, which could also explain the change.

In Dumfriesshire, the SNP have increased their lead from 2pp to 11pp. Ashcroft attributes this to some of the Labour vote going to the SNP. Although it's true that the Labour vote drops by 3pp and the SNP increases by this amount, the "Wouldn't votes" have also decreased by 3pp.

For polls of 1000 people such as these, the sampling error is often quoted as +/-3pp, but this only applies in cases where there's a roughly 50%/50% split of the vote. In these polls most of the vote share is spread unevenly across three parties. It's therefore possible that a sampling error larger than 3pp is distorting the results.

What's remarkable is that both these constituencies have shown quite dramatic changes in the space of two weeks. The undecided zombies are coming and are likely to have a decisive effect in many seats!

Solar co-ordinate fun

2015-04-24T17:06:00+01:00

Transforming co-ordinate systems between pixel images and the surface of a sphere is probably not most peoples' idea of fun, but I enjoy it. To sugar the pill take a look at this fabulous image:

The Sun in ultraviolet taken by the Solar Dynamics Observatory. Credit: NASA/SDO/AIA

If you see an interesting feature on the surface of the Sun, you can specify its position by its pixel co-ordinates. The image above is 600 pixels across by 413 pixels high, so I can say that the interesting, yellow feature (a cluster of magnetic field loops in an active region) to the right of the image's centre is located at (380,210), that is, 380 pixels in from the left side, and 210 down from the top edge of the image.

This is a bit unsatisfactory, so it might be better to use a different co-ordinate system to say the feature is centred on the point (80,90) measured relevant to the centre of the Sun's disc, i.e. 80 pixels to the right and 90 pixels up from the centre.

But this isn't very useful because the Sun is a sphere, not a flat disc. We use longitude and latitude to specify our location on a spherical Earth, so we can use something similar for the Sun. Let's say heliographic latitude is zero along a line drawn horizontally through the centre of the Sun; this is the solar equator. The North pole is +90°N and will be at the top and -90°S at the bottom will be the South pole. Of course, this assumes the Sun has been correctly orientated in the image.

We can define heliographic longitude so that 0° lies on a vertical line that runs through the two poles. This is sometimes called the solar meridian. A point with a longitude of 90° lies on the right edge of the Sun (referred to as the west limb by solar observers) and a point with a longitude of -90° lies on the left edge (east limb).

Unlike the longitude and latitude system of the Earth which rotates with the Earth, this solar co-ordinate system is fixed to our viewpoint. This means that the Sun's rotation will cause features to move from left to right, e.g. the longitude of a sunspot will increase over time, though its latitude won't be affected. The Sun is not a solid body: it takes about 25 days to rotate once at the equator, and somewhat longer to complete a rotation nearer the poles.

The image below illustrates what lines of constant longitude and latitude look like on a 2D image. The central vertical red line is the solar meridian at longitude 0°, and the first red curve to its right is 10°, and the next is 20°, and so on. The solar equator is the central horizontal blue line, and the line of 10°N latitude is immediately above it.

Solar lines of longitude (red) and latitude (blue) in intervals of 10°. The plot was generated using the venerable plotting program gnuplot using this script.

Points of equal latitude (blue) form horizontal straight lines, but points of longitude (red) form curves. For example, the longitude line for -80° is a curve very close to the left edge of the Sun. This explains why features near the limb of the Sun will appear squashed, which is referred to as foreshortening by astronomers.

Some mathematics

If you don't like equations then you won't like this bit. Sorry.

Consider a point on the solar surface that's at longitude $\phi$ and latitude $\theta$. Our first aim is to calculate the pixel co-ordinates of this point. Let's take the point (0,0) to be at the centre of the Sun's disc (not the image, which may be different). A bit of spherical trigonometry and 3D to 2D projection (which I'll not get into here) yields these equations:

\begin{eqnarray} x&=& R\cos\theta\sin\phi \label{x} \\\ y&=& R\sin\theta \label{y} \end{eqnarray}

where $R$ is the radius of the Sun in pixels, i.e. half the diameter.

We can now turn these equations around to work in the opposite direction - given $(x,y)$ pixel co-ordinates, calculate the longitude $\phi$ and latitude $\theta$. Let's take (\ref{y}) first which tells us that $\theta$ is:

$$ \theta = \sin^{-1}\frac{y}{R} $$

The -1 indicates we need to perform the sin function in reverse, sometimes called arcsin or inverse sin. This can be done by using the shift or inv button on a scientific calculator. The angle obtained needs to be between -90° and +90° where minus means South (or below centre).

Now we have $\theta$ we can use it to calculate the latitude $\phi$ by re-arranging (\ref{x}):

$$ \phi = \sin^{-1}\frac{x}{R\cos\theta} $$

and again $\phi$ should be between -90° and +90° where minus means east (or left of centre).

You can also use spreadsheets to perform these calculations, though remember that spreadsheet trig functions (sin, cos etc) usually work in radians rather than degrees. This spreadsheet shows an example of calculations in either direction: download ods or download xls.

Scotland - Full fiscal autonomy

2015-04-21T12:14:00+01:00

In the wake of Scotland voting to remain part of the UK, the SNP government has proposed Full fiscal autonomy (FFA). This means expenditure will need to be met by tax revenues raised in Scotland, and the shortfall, i.e. the deficit, will need to be covered by borrowing arranged by the Scottish government. Scotland would remain part of the UK and so have to pay a contribution to the Westminster government for its share of defence and other international costs. You can think of FFA as fiscal separation or fiscal independence.

Update: the SNP have released their 2015 General election manifesto and renamed it Full Fiscal Responsibility.

Although it seems unlikely that FFA will find political approval in coming years, its implications deserve some attention. In this post I'm going to use a simple model of my own devising to look at how different assumptions of growth and expenditure will affect Scotland's deficit under FFA. The model draws on historical data from Government Expenditure & Revenue Scotland 2013-14 (GERS) to establish trends for Scotland's share of UK public sector revenues and expenditure, and combines them with Office of Budget Responsibility (OBR) UK forecasts to predict them for the next five financial years.

Deficit breakdown

The graph below shows Scotland's deficit deconstructed into various components which are useful for understanding implications of FFA. The data plotted are:

■ blue: this is Scotland's population share of the UK deficit, i.e. in 2013-14, 8.3% of the UK's population resided in Scotland, so the blue bar for that year is 8.3% of the UK's deficit (i.e. PSNB ex).
■ red: this is the extra deficit peculiar to Scotland excluding North Sea revenues. Most of it comes from Scotland having a higher spending per capita, but a smaller part is due to non-north sea revenues per head being slightly lower. It is positive and so always increases Scotland's deficit.
■ yellow: this is Scotland's share of North Sea revenue (less the small amount already included in the blue bar). It is negative and so always reduces Scotland's deficit.
━ green: this is the Scottish deficit. Adding the red, blue and yellow bars together (remembering the yellow is negative) will give the green line's value for a given year.

Source: A breakdown of the history of the notional Scottish deficit as generated by a simple model using data from GERS. The OBR's March 2015 Economic and fiscal outlook report for the UK is the basis for forecasts in the shaded area. As with GERS, all figures are in current prices, i.e. not corrected for inflation.

The data to the left of the shaded region is based on actual data on Scottish finances from GERS. The shaded area to the right are forecasts using my model which applies recent trends in expenditure and revenue from GERS to infer forecasts for Scotland based on OBR forecasts for the UK. Further detail on how these components are calculated can be found in an appendix at the end.

When the green line is above the top of the blue bar, Scotland is in a weaker fiscal position than the UK as a whole, and by weaker I mean that if Scotland had FFA, it would have a higher deficit per capita than the UK.

Over the last 15 years, Scotland would have been weaker in all but three years, and those years were when oil revenues were highest, i.e. the yellow bars are longest.

The red bars represent a potential extra fiscal problem for Scotland which has historically been mitigated by the yellow bars. It is only potential because neither the problem nor its mitigation have mattered in reality because Scotland's extra deficit is heavily diluted in the UK's deficit.

The word deficit is often used rather loosely, so to be precise, the Scottish deficit (green line) is the negative of the net fiscal balance as reported in GERS. It is current plus capital expenditure less revenue. GERS also states a UK net fiscal balance which is the negative of PSNB ex (Public Sector Net Borrowing Excluding temporary effects) quoted by the OBR report and ONS Public Sector Finances. Although not strictly correct, to be consistent with common usage and keep jargon at bay, I refer to this as the UK deficit in this post.

Forecasting full fiscal autonomy

If Scotland had FFA then it would have a deficit, and the deficit would be the green line. The forecasts I've produced in the shaded region of the graph show what would happen under two assumptions:

that the OBR predictions for the UK under the current coalition government's plans are correct;
that the tax revenue and expenditure trends that have persisted for at least the last fifteen years continue if Scotland has FFA.

Neither assumption is likely to be true, but it still makes sense to explore some possible scenarios using the above forecast as a baseline. First let's consider the big implication of the plummeting forecasts for North Sea revenue that come from assumption 1.

The graph shows that North Sea revenue (yellow) almost disappears from 2015-16 onwards. This means that there is nothing to mitigate the extra deficit (red) and consequently Scotland's deficit (green) remains well above its share of the UK deficit (blue). The size of the red bars represents the £7.6bn deficit estimated by the IFS that is dubbed the "black hole" by the media. Notice that even as the blue bars shrink, there is no reduction in the red bar. The explanation is simple: the coalition government strategies are designed only to reduce the UK deficit and so they don't address the extra deficit in Scotland.

So under FFA, Scotland would need to introduce additional measures to bring its extra deficit down. The SNP government claim they will do this by using the "fiscal levers" that they will gain control of under FFA.

Let's imagine an alternate reality in which FFA powers come into effect in 2014-15. If Scotland had voted to separate in 2014 then this would have happened in 2015-16, so it's not too far fetched.

There are two sets of levers to operate. A Scottish government could choose to pull back on the expenditure levers, i.e. make cuts. This is the coalition government approach for the UK. The SNP object to this, with some justification in my opinion, so they talk up the other option: to throw forward the levers that will increase revenues.

The SNP don't propose any tax rises, but instead express a belief that economic growth will result in increased revenues from income tax, VAT and corporation tax. Specifically, they say that Scotland's non-North Sea (onshore) annual revenue is predicted to grow by £15bn by 2020. This is true (give or take a billion) but it does not help in reducing the red bars, because that growth of revenue is simply reflecting the growth of UK revenues predicted by the OBR, and is already implied in the graph: it's part of the reason for the blue bars' reduction, with coalition cuts accounting for the rest.

Extra growth?

Having established that the growth predicted by the OBR cannot reduce the red bars, the question becomes how much extra growth is needed to do so? As North Sea revenues are a lost cause for the next five years, the extra growth would need to be in onshore tax revenues.

The graph below is identical to the one above, except for the forecasts: the extra deficit (red) is eroded to zero in 2019-20 by a growth in onshore revenues.

Source: As for the the previous graph, except now the models assumes a 2% growth in onshore tax revenues in Scotland.

The real (i.e. inflation adjusted) revenue growth needed to produce this forecast is about 4.6%. The OBR forecasts UK real GDP growth to average at 2.4% over the next five years, so extra growth of about 2pp (percentage points) would be needed from Scottish onshore revenues.

But this model uses OBR forecasts that assume coalition cuts are taking place. The SNP have said that they'd like Westminster to grow annual expenditure in real terms at 0.5%, so presumably, if they had FFA they might want to do that in Scotland. I found that adding 0.9pp to Scottish expenditure (remember these are nominal, i.e. not inflation adjusted) produces an annual growth in Scottish expenditure of 0.5% in real terms. To produce the above graph, i.e. bring the Scottish deficit to zero in 2019-20, requires 3pp extra onshore growth, making the real growth rate a hefty 5.6%.

One thing that's not included in the two scenarios I've mentioned is the extra burden of paying for borrowing, as that would be required to cover the deficit during the years it is being reduced. It would of course only increase the deficit and require more extra growth from revenues.

Enforced cuts?

But what if borrowing could not be secured under FFA? In this case cuts would need to be made. Let's optimistically assume that the UK agreed to use its borrowing to cover the blue share and shoulder the debt interest payments too, then Scotland would have to eliminate the remaining deficit by cutting the red bar down until it was at least as small as the yellow bar.

So the cuts for 2014-15 would be £6.2bn and for 2015-16 as £7.7bn. From 2014-15 to 2019-20, a cumulative total of £46bn of cuts would be required, and this is in addition to the coalition cuts assumed in the OBR's forecast. For context, this represents 11% of the annual Scottish expenditure, and equal to the spend on Education and training.

Bottom line

There is no prospect of growth on anything like the scale that the SNP would need to close Scotland's extra deficit. No major, developed economy has shown real growth of around 5% in recent years. In fact, the UK is currently one of the best performers with 2.6%.

The fact that an FFA Scotland has no track record of borrowing, that it embarked on FFA knowing that it would face such a large deficit and that Scotland does not have control over its own currency would all make borrowing very difficult and more expensive than for the UK. The likely outcome would be a mixture of costly borrowing and cuts over and above what the UK coalition government propose, with the full burden falling on Scotland.

The easiest way to avoid these intractable difficulties is for Scotland to reject FFA and continue the arrangement of pooling resources with the UK. That's not to say the UK has to remain as it is, nor is it a statement that Scotland is "too wee and too poor" to be independent (a phrase only uttered by the SNP). It is simply saying that for the foreseeable future FFA is both economically unwise and self-defeating in that it serves neither Scottish interests nor those of the UK.

Last minute update: At almost the moment I was about to hit publish on this post, this IFS report was published. It broadly agrees with what I've found, though it forecasts a slightly greater extra deficit for Scotland. For example, Table 1 shows an extra deficit of £49bn over the next five years, whereas I found it to be £46bn. The IFS predicts a real growth rate of 4.5% is needed to avoid this deficit by 2019-20, consistent with my estimate of 4.6% above. I'm reassured that my amateur and self-taught understanding of these things turns out to be consistent with a respected body such as the IFS

Appendix: Model details

The spreadsheet that generated the graphs and data discussed above can be downloaded from these links: download ods or download xls.

The mathematics below describe the simple model used to split Scotland's deficit figures into the components displayed on the above graph.

Let the UK deficit for a given year be $D$ and the expenditure (Total Managed Expenditure) be $E$ and revenue (current receipts) be $R$. Let's also split $R$ into the sum of onshore revnue $O$ and north sea revenue $N$. So, we can write the deficit as:

\begin{equation} D=E-R=E-O-N \label{UKDeficit} \end{equation}

We can do the same for Scotland using the subscript $s$:

\begin{equation} D_s=E_s-O_s-N_s \label{ScotDeficit} \end{equation}

Let the fraction of the UK population that live in Scotland be $p$. We can then write the Scottish expenditure, onshore and North Sea revenues as follows:

$$\begin{eqnarray} E_s&=&(p+e)E \nonumber \\\ O_s&=&(p+o)O \label{shares}\\\ N_s&=&(p+n)N \nonumber \end{eqnarray}$$

where $e$ is the fractional extra expenditure for Scotland above what you'd expect from it's population share, and $o$ and $n$ are similar but for onshore and North Sea revenues respectively.

For 2013-14, $p=0.083$, $p+e=0.092$, $p+o=0.081$ and $p+n=0.755$. Notice that $p+o$ is less than $p$, so $o$ must be negative, whereas $e$ and $n$ are positive, as has been the case for at least the last fifteen years. In the base model shown in the first graph, the forecasts have $p$ declining slowly at its recent historical rate of -0.24%, and $e$, $o$ and $n$ remain approximately constant over the next five years.

Inserting (\ref{shares}) into (\ref{ScotDeficit}) gives:

\begin{eqnarray} D_s&=&(p+e)E - (p+o)O - (p+n)N \nonumber \\\ &=&p(E-O-N) + eE -oO - nN \nonumber \\\ &=&pD +(eE-oO) - nN \end{eqnarray}

where (\ref{UKDeficit}) is used to substitute $D$ in after the second line. The three terms in the final line are the population share of the UK's deficit $pD$ (blue bars); the extra deficit for Scotland $(eE-oO)$ (red bars); and reduction in deficit due to North Sea revenue (yellow bars).

Notice that because $o$ is negative, the $-oO$ increases deficit. Also, the $nN$ term is not just the geographic share of North Sea revenue but the extra on top of population share. The population share of North Sea revenue is contained within $pD$.

Scotland's finances Part 3 - Balance

2015-04-11T13:37:00+01:00

This is the third post about GERS 2013-14 (Government Expenditure & Revenue Scotland). Having reviewed revenue and expenditure in previous posts I'm now going to look at how these come together to give a budget balance. You can read about the background to GERS here and find links to download the PDF and spreadsheet of data.

Previous posts in this series:

Balancing acts

First, let's set out some terminology. Talking about public sector budget balances involves many terms that have closely related or identical meanings. For example, "balance", "budget balance" and "budget surplus" all mean the result of subtracting expenditure from revenue. As this is often negative for a country with its own currency, it is more usual to hear talk of the "deficit" or "budget deficit" - the expenditure minus revenue, i.e.the negative of the budget balance.

There are two budget balances given in GERS, but I'll focus on the "net fiscal balance". This is the public sector revenue minus the total public expenditure. The other balance is the "current budget balance". This is public sector revenue minus current expenditure. As described in my last post, total expenditure is the sum of current and capital expenditure, and the capital expenditure differs from current expenditure in that it is used either to purchase or create an asset that will continue to have value or use in the future.

Net Fiscal Balance

The graph below shows the net fiscal balances for the UK and Scotland.

Source: Constructed from Table E.6 in the GERS spreadsheet. Scotland figures include a geographical share (83.8%) of North Sea revenue.

Notice that the values are all negative since 2003, which means that both Scotland and the UK have had deficits, i.e. expenditure has exceeded revenue. The jump in deficit caused by the 2008 crash is in part because expenditure rose, but mostly because tax revenues fell back - I discussed this in more detail for the UK in a previous blog post.

This graph shows the balances with North Sea revenue excluded for both the UK and Scotland. The scales in this graph are identical to the previous one.

Source: Constructed from Table E.6 in the GERS spreadsheet with data from Tables 3.3 and 5.6..

If you compare this graph with the one above, you'll notice that excluding the North Sea revenue makes very little difference to the UK's green bars, but significantly lengthens Scotland's blue bars downwards, i.e. Scotland's deficit increases dramatically if North Sea revenue is excluded.

The graph below draws on the same data but shows Scotland's deficit as a fraction of the UK's deficit. The first four years are omitted because the UK didn't have a deficit then.

Source: Constructed from Table E.6 in the GERS spreadsheet with data from Tables 3.3 and 5.6..

Look first at the red bars on this graph and bear in mind that Scotland's population share has steadily declined from 8.7% to 8.3% over the years shown. Scotland's share of the deficit exceeds its population share for nine out of the twelve years shown. This is mainly because expenditure per head in Scotland is greater than expenditure per head in the UK as a whole. As we'll see below, the three years in which Scotland did better are the three highest years of North Sea revenue (2005-06, 2008-09, 2011-12).

The blue bars, which exclude North Sea revenue, show a much higher deficit share which is in excess of the population share for all years. It averages about 25% of the UK's deficit prior to the crash in 2008, and at about 15% since then.

North sea revenue

North Sea revenue is sensitive to the oil price which is volatile because it is subject to market forces and decisions made by international oil cartels. The graph below shows UK North Sea revenue and the oil price per barrel:

Source: Constructed from Table 3.3 in the GERS spreadsheet with data from Tables 3.3 and 5.6. and the oil price is the Europe Brent Spot Price

The correlation between North Sea tax revenue and the oil price is clear. For instance, both peak together in 2008-09. However, there are departures from the correlation for several reasons related to the fact that the revenue is a tax on profits of companies operating in the North Sea.

Firstly, firms may claim tax breaks for investing in their North Sea infrastructure, or engaging in activities that may benefit future production. Secondly, the money made by oil firms depends on the amount they extract, and if that output changes, so will their profits, even for a fixed oil price. Finally, if operational costs go up, then profits will go down and so will the tax revenue.

From 2011-12 onwards, all of these factors have worked together to reduce the tax revenue from the North Sea despite the oil price remaining high. Operating costs went up and oil firms chose to invest and production declined slightly. Future years of North Sea tax revenue are expected to be lower still because since 2013-14 the oil price has halved. The Office of Budget Responsibility predict North Sea revenues will fall from their recent peak at over £10bn in 2011-12 to be only £0.7bn in 2015-16.

North Sea extra deficit

As we saw above, excluding North Sea revenues for Scotland has a much more significant effect on Scotland's deficit than on the UK's. Let's estimate the extra Scottish deficit that would have resulted if the drop in North Sea revenues had occurred over the five financial years to 2013-14.

For those five years, Scotland's deficit including a geographic share of North Sea revenues averages to £12.4bn. Without North Sea revenues it is £18.9bn. So if North Sea revenues were £0.7bn each year, then Scotland's deficit would have averaged to £18.2bn. So the annual North Sea extra deficit can be estimated as £18.2bn - £12.4bn = £5.8bn.

Bottom line

In 2013-14, and for the twelfth year in succession, both Scotland and the UK ran a budget deficit. In nine of those years Scotland's share of the UK deficit was greater than its population share, and in the other three Scotland benefited from unusually high North Sea tax receipts.

Typically, 10% to 20% of Scotland's annual revenue has come from North Sea tax receipts. However, from a high of over £12bn they have fallen to £4.7bn in 2013-14 and are predicted to fall to £0.7bn in 2015-16 and remain at this level for the foreseeable future. If this had happened in the last five years then Scotland's deficit would have been raised by £5.8bn. The Institute for Fiscal Studies (IFS) predict that this figure will be £7.6bn for 2015-16.

It's important to realise that the predicted £7.6bn is additional to the existing deficit. Scotland's total deficit would be roughly £15bn according to the above figures; this is larger than Scotland's annual expenditure on the NHS. Scotland's share of the UK deficit will rise into the 15-20% range in future years, so likely more than twice Scotland's population share.

Scotland's deficit is notional at present because of the way that revenues are pooled across the UK, and so does not impact expenditure in Scotland. If Scotland had voted to become independent, or opts for Full Fiscal Autonomy (FFA) in coming years, then the pooling would end and this notional deficit would become a reality.

It is true that these figures represent only a starting point for what would happen under FFA, and the powers of FFA would allow Scotland to make choices to promote growth and borrow to cover its deficit. That said, it is inarguable this starting point presents a fully fiscally autonomous Scotland with an immediate and very serious challenge.

My next post will be the last in the series on GERS 2013-14 and will describe my views on the wider political implications for Scotland and the UK.

Scotland's finances Part 2 - Expenditure

2015-03-31T16:06:00+01:00

This is my second post about GERS 2013-14 (Government Expenditure & Revenue Scotland) and I'm going to turn my attention to Scottish public sector expenditure. You can read about the background to GERS here and find links to download the PDF and spreadsheet of data.

Previous posts in this series:

Scotland's finances Part 1 - Revenue

As with the last post, it's important to remember that Scotland's share of the UK population is 8.3% as this provides a good benchmark for judging how expenditure differs in Scotland from the whole of the UK.

Expenditure history

This graph shows the history of public sector expenditure in Scotland and the UK.

Source: Constructed from Table 5.2 in the GERS spreadsheet.

Notice that the UK scale (right) is ten times the Scotland scale (left) and that every blue bar is higher or level with its partner red bar. This means that in all years the percentage of UK expenditure in Scotland is at least 10%, and so larger than Scotland's population share which has declined from 8.7% to 8.3% over the years shown.

Expenditure for both the UK and Scotland steadily increased from 1998, then jumped up in 2008-09 at the start of the Great Recession, and decreased from 2010 onwards due to coalition government cuts, though less markedly for Scotland.

Capital vs current expenditure

Current expenditure makes up around 90% of the totals shown in the above graph. It is money that's spent in a one-off way, such as buying food for hospitals - it's consumed once (hopefully!) and then it's gone. Paying staff such as doctors, nurses or teachers also comes under current expenditure.

Capital expenditure is where money is spent on creating an asset which gives repeated use or value in the future, such as in building a hospital. The cuts since 2010 have been heavier on capital spending in both Scotland and the UK, presumably because it is is easier not to build a new school or hospital than to cut the pay of people that staff them. There are of course serious long term consequences of cutting capital expenditure. Beyond this, I found nothing particularly noteworthy in the division of current and capital expenditure in GERS.

Expenditure breakdown

This pie chart shows the areas of expenditure in Scotland that exceeded £2bn in 2013-14 with the remaining 15% of expenditure lumped into the Other slice.

Source: Constructed from Table 5.5 in the GERS spreadsheet.

Much of the largest slice, Social protection, goes to paying state pensions with the remainder being made up of things such as housing benefit and income support. The yellow and green columns show that Social protection increased as a percentage of total revenue for both Scotland the UK, which is a result of the jump in unemployment triggered by the Great Recession of 2008.

The next largest is Health which combined with Social protection comes to about half of all expenditure. Education and training is third largest, which I'll come back to shortly.

The appearance of Accounting adjustments in fourth place at £3.8bn caused my eyebrows to raise, but it is explained in appendix B of GERS as reconciling the difference between expenditure figures from HM Treasury which are used for Scotland, and the UK equivalents from the Office of National Statistics (For the detail-hawks, it's mainly the difference between Total Managed Expenditure TME and Total Expenditure on Services TES.) The adjustment also involves another, much smaller correction to account of EU transactions.

Scotland's share

The table below gives more detail about expenditure for 2013-14 ranked so that the largest areas are at the top.

Source: Constructed from Table 5.5 and 5.6 in the GERS spreadsheet. The ranges cover five financial years, i.e. 2009-2014 is the sum over all years from 2009/10 to 2013/14.

There's a lot to take in, but first focus your attention on the pink column which shows Scotland's expenditure for 2013-14 as a percentage of the UK's.

The final row tells us that in 2013-14, 9.2% of UK expenditure went on Scotland. In other words, more money is spent per head in Scotland than in the rest of the UK. This is another way of stating some figures in Table E.4 of GERS that made headlines, such as the Scotsman's Scots public spending £1200 per head more than UK. There are a few reasons for this.

Firstly, Scotland needs more funding to maintain the same level of public services given its low population density and many islands. If you run your eye down the table to the Transport row, you can verify one fact that's consistent with this: nearly 15% of the UK's transport expenditure is in Scotland.

Secondly, Scotland needs higher expenditure because it suffers more socio-economic problems than the UK as a whole (but not, say, compared to northern England or Wales). The "socio" bit is evidenced by Social protection and Health being above Scotland's 8.3% population share at 8.9% each. Support for business on the "economic" bit is much higher with 19.9% - a fifth - of the UK's spend on Enterprise and economic development taking place in Scotland. The other high percentage in the pink column is also business-oriented being for Agriculture, forestry and fisheries at 17.5%.

The only area of Scottish expenditure below the UK level is Science and technology at 7.5%, though this is the second smallest expenditure area in this list.

Education

It is of course not an accident that international services, debt interest and defence, all sit exactly at 8.3%; they are all UK-wide and apportioned to Scotland in GERS according to population.

However, it is surprising that Education and training sits at 8.3%, because it is devolved to Scotland. If you look over to the purple columns you can check the values for the last two 5 year periods. The proportion of UK expenditure on Education and training in Scotland was 9.4% in 2005-2009, but dropped to 8.6% in 2010-14.

The graph below shows the expenditure on Educating and training for all years of the Scottish Parliament. Health, which is also devolved to Holyrood, is shown for comparison.

Source: Constructed from Table 5.5 in the GERS spreadsheet.

Expenditure on Education and training was rising until 2007-8, but unlike Health it stopped rising and has remained roughly constant at £7.5bn since then. As these figures are in current prices, i.e. not corrected for inflation, this means that expenditure on education has decreased in real terms in recent years. Using the CPI index of inflation, £7.5 bn in 2008 equates to about £8.7bn in 2013, i.e. this represents a cut in annual Education expenditure of £1.2bn in 2013 prices. EDIT 21/5/15: Using the ONS GDP deflator, which is more appropriate, the 2007/8 figure is £8.5bn in 2013-14 prices.

The only other significant reduction was in Science and technology, which is about 20 times smaller than Education and training. It suffered a cut in cash terms over the same time period from £350mn to £330mn, which is a real terms cut of 21%.

Change?

There's a curious trend, or rather, lack of trend in the above table. if you run your eye down the change columns for Scotland (yellow) and the UK (green), you'll notice that changes in expenditure of the country totals have been very small - almost all are less than one percentage point between the two 5 year periods.

This suggests to me that expenditure is determined by need rather than by political policy. Whereas there has been significant political change in Scotland and the UK governments during the last decade, it can be argued that needs of the people have largely not changed. The increase in Social protection does of course represent a change in need - the need of the unemployed in the aftermath of the 2008 crisis - but it too fits the idea that expenditure is need-driven. The one area that does not fit with this is the real terms reduction in Education expenditure in Scotland, but there is no way to determine the reason for this from GERS data alone.

Bottom line

Scotland receives 9.2% of the UK's expenditure whilst it has 8.3% of the UK's population, or, equivalently, public expenditure per citizen in Scotland is £1200 higher than the expenditure per UK citizen.

Almost all areas of the public sector are better funded in Scotland than for the UK as a whole. The big exception to this is the expenditure on Education and training - the third largest area - which has a received a real term cut of about 14% in its annual expenditure.

As I said in my previous post, I'll hold back on political comment until the last post on this series.

Scotland's finances Part 1 - Revenue

2015-03-27T17:31:00+00:00

I've written a number of blog posts about the UK public sector finances and have been wanting to give similar scrutiny to the Scottish equivalents but was holding off until GERS (Government Expenditure & Revenue Scotland) for 2013-14 was published. It was released on Wednesday 11 March 2015 and I've spent some time reading through its 98 pages. You can about the background to GERS here and find links to download the PDF and spreadsheet of data.

Note that GERS is consistent with the ONS UK Public Sector Finances (PSF) January 2015, whereas my previous blog posts are based on PSF data published in Sept 2014.

The first few blog posts will concentrate on data and facts and - as far as I can manage it - I'll keep political comment confined to the last post in the series. My aim is to make GERS more accessible and highlight points that have interested me, and will hopefully interest you too, especially where such points haven't come up in GERS coverage elsewhere.

Whilst writing this post, I came across this thoughtful and clear analysis of GERS by Kevin Hague. I recommend you read it. He takes a different approach to me in some ways (e.g. looking at rUK figures), but it's valuable to have different views on the same data. Perhaps you, dear reader, might consider writing something too? Apart from sharing your thoughts with others, and perhaps spotting something no one else has noticed, you'll find there is no better way to ensure you understand something than explaining it in words.

General points

Scotland's GERS differs from the UK's Public Sector Finances (PSF) in that it has to allocate shares of certain tax revenue to Scotland. This is most crucial for North Sea revenue. GERS offers two alternatives: one is the population share, and the other, which I'll mean unless otherwise stated, is the much larger geographic share.

I'll probably come back to the North Sea in a later blog post, but for now keep in mind that 2013-14 represents a very poor year for North Sea tax revenues and that the recent crash in oil prices occurred after this time period - i.e. next year is likely to show even lower North Sea revenue.

Scotland has 8.3% of the UK population and many of the percentages from GERS are within a few percentage points (pp) of this. It's the minority that aren't that are particuarly interesting as they highlight aspects of public finances that are unique to Scotland.

On a technical but important note, all figures here and in GERS are quoted are in current prices, also known as nominal values, i.e. they are not corrected for inflation.

Revenue history

The following graph shows the revenue of Scotland as a percentage of the UK total, and, for reference, the percentage of the UK population in Scotland.

Source: Table 3.1 of GERS spreadsheet.

The revenue including that from the North Sea (red) is always above Scotland's population share (yellow). This is because the North Sea share is allocated geographically. In 2013-14 the population share was 8.3% but the geographic share was 83.8%.

The revenue share excluding the North Sea from both Scotland and the UK (blue) is noticeably less volatile and is always below the population share (yellow). The declining trend is simply mirroring the decline in population share; Scotland's population is growing more slowly than the UK's.

Revenue breakdown

The top 12 sources of Scotland's public sector revenue are shown in the bar chart below. These come to a total of £50.4 bn or 93% of Scotland's total revenue. North Sea revenue is shown in two colours: blue is the population share, and the blue+green is the geographic share.

Source: Page 34 of the GERS 2013-14 report or Table 3.1 in the accompanying spreadsheet.

Income tax, VAT and national insurance are the top three sources of revenue. Table 3.1 in GERS shows that these are the top three for the UK too.

The fourth largest income for Scotland in this year was North Sea corporation tax revenue at £4.0 bn. Remember though, this is a bad year for North Sea revenue. In a good year, such as 2011/12, this figure was closer to £10 bn, similar to Scotland's VAT revenue. For comparison, North Sea corporation tax revenue for the whole UK is £4.8 bn and ranks fifteenth amongst its revenues.

Next up, and only a shade smaller than the North Sea revenue, is the enigmatic sounding Gross operating surplus. It is explained on page 37 of GERS as being money received from public sector corporations. (You could call it public sector profit without troubling a semantic pedant too much.) This includes Scotland's share from UK-wide bodies, such as from The Bank of England, but a significant portion of it is for Scottish Water, of which, more later.

Scotland's share

Let's start by looking at the purple column of the table below which shows Scotland's revenue from each source as a percentage of the UK's.

The percentages are generally within a percentage point (pp) or two of the 8.3% population share. Of the top 12 sources of revenue in the table, seven are within 1pp of 8.3%, and nine are within 2pp.

By far the biggest departure - by a factor of 10 - is the 83.8% geographic share of the North Sea revenue. The next biggest is for Tobacco duties, for which Scotland contributes 13.2% of the UK's total, followed by the above-mentioned Gross operating surplus at 10.4%.

Whilst VAT and national insurance are unremarkable, coming in at 8.4% and 8.1% respectively, income tax is somewhat low at 7.3%, reflecting that individual incomes in Scotland are below the UK average.

Source: Top 12 revenues for Scotland from Table 3.1 in the GERS spreadsheet.

Shares of totals

The yellow column shows the revenue from each source as a percentage of the Scottish total in 2013-14. The last green column does the same for the UK. These data give us an alternative view of how Scotland differs from the UK as a whole.

Income tax makes up 21.1% of Scotland's revenue, whereas it is 25.0% of the UK's. Again this tells us that average incomes received by individuals is lower in Scotland. This is the main reason why revenues excluding the North Sea in the above graph (blue) are below Scotland's population share (yellow).

National insurance (NI) shows this too but to a smaller extent. Since NI doesn't ramp up with income (income tax goes up from the 20% basic rate to 45% for the highest incomes), this suggests that there is a class of very high earners outside Scotland that pull the UK average up.

VAT percentages are not too different and the share in the purple column is only 1pp from 8.3%. This suggests that Scottish consumers are little different in their overall spending from the UK as a whole. I might venture a guess that this is because the higher earners outside Scotland (mentioned above for NI) are saving or investing their extra income rather than spending it.

There's little else of note in the yellow and green columns, except, of course, for the North Sea corporation tax revenue. This revenue is 7.4% of Scotland's total for 2013-14, but only 0.8% of the UK's total. Remember this is for a bad North Sea year; for a good year such as 2011/12 it was 17.0% and it reached an all time high of 20.6% in 2008/9. For comparison, the share of non-North Sea corporation tax in 2013-14 was 5.2%. So, even for a bad year, Scotland's corporation tax revenue from the North Sea exceeds that for all other business activities.

Scottish Water and public ownership

Those who think that the private sector companies are necessarily more efficient and profitable than public sector ones may be surprised by the statement on p24 of GERS that "UK public corporations ... tend to run at a slight surplus". This is clearly backed by the evidence in GERS and in the UK Public Sector Finances (PSF).

Water and sewerage services in Scotland are provided by public sector owned Scottish Water, but in the rest of the UK they are mostly provided by private companies. Thanks to the surplus generated by Scottish Water, Scotland's Gross operating surplus is 10.4% of the UK's total revenue.

Scottish Water also shows up as extra expenditure peculiar to Scotland too, which I'll describe in the next blog post. (As an aside, I recently had an interesting chat with Scottish Water's Chief Operating Officer about its history of public sector ownership, but that's also for another blog post.)

Devolution

It's clear from the above that Scotland's revenues as a percentage of the UK from each source are mostly not too different from its population share of 8.3%.

The most obvious exception is the 83.8% geographic share of North Sea revenues, but remember this is just an illustrative figure and not one that has any tangible meaning in the running of the UK or Scottish public sector.

Most of the remaining differences are of a few percentage points or less, and are caused by socioeconomic factors rather than because Holyrood has exercised devolved powers. For example, consider income tax: it is the largest revenue source, and Scotland brings in 7.3% of the UK's income tax, and is 1pp below Scotland's population share. This is because average incomes are lower in Scotland, and not because the basic rate of tax is different; Holyrood has never used its power to vary income tax.

One area where Holyrood has exercised its devolved powers is to freeze council tax. It now sits well below the 8.3% population share at 7.1% of the UK total. This doesn't necessarily mean that Scottish councils have experienced a commensurate drop in revenue, because the Scottish Government can choose to provide top-up funding. However, it does mean that some revenue raising powers have been removed from local councils and centralised in the Scottish Government.

Bottom line

Scotland's revenue, including the North Sea, as a percentage of the UK's was 8.6% in 2013-14. Excluding the North Sea, this percentage drops below the population share of 8.3% to 8.1%.

The more popular way to express this is using the figures in GERS table E.2. The revenue per person in Scotland is £400 higher than for the UK if North Sea revenue is included, but £300 lower if excluded.

Of course, this is only one part of the story. The next blog post will be concerned with expenditure.

Post Solar eclipse 2015

2015-03-20T12:39:00+00:00

I managed to see quite a bit of the eclipse this morning, despite the clouds. My children and I observed it from the balcony at the front of our house using a pin-hole camera made out of a cardboard washing powder box. We called it the HGT.

The HGT - Hubble Ground Telescope. Several pin-holes were made on the front and images projected onto the white paper stuck on the back. The altitude of the Sun at the time of this photograph was about 25° which, apparently, equates to an Ikea plate+Sky remote+Wiimote.

From the start of the eclipse at 08:30 to nearly 10:00 the sky was mostly overcast and cloud obscured the Sun (sometimes known as a Glasgow eclipse). There were however some brief moments in which the cloud thinned and the dimmed crescent Sun could be glimpsed without dazzling the eyes.

By 09:30 - 4 minutes before the local maxium of 95% was reached - the light outside took on an eerily dark feel, and outdoor lights with sensors and most car headlights came on. I noticed also that the hallway inside the house, lit only by daylight coming in from other rooms, seemed oddly dark. This peculiar light lasted until about 09:40.

Normal sunlight will provide about 1340 W of power across a square metre surface facing it. That's more than a bar of an electric fire or about twenty-two 60 W light bulbs. So, given that at the maximum of this partial eclipse the sunlight was reduced by 95%, dropping the power to 67 W per square metre, it may seem puzzling that the darkness wasn't more noticeable. The explanation is that the change in light is gradual enough that our eyes have time to adapt to the new light levels. It's only when you get close to 100% that your vision fails to compensate and it will start to seem dark. Also, because cloud cover can change much faster than the solar eclipse progresses, its effect is often more marked.

From 10:00 am onwards the clouds parted sufficiently to allow the pin-hole camera to work. It soon became obvious that the hole I'd made was too large, so I used some finer pins to make a few smaller holes. I had to take quite a few photographs with my phone to get one that was steady enough to see the eclipse clearly - the best one (and yes, it's not I know it's not great!) is this one:

Pin-hole images. Projected images of the Sun at 10:20:52 photographed with the camera on my Nexus 4 phone from a slightly oblique angle.

The first hole was too big and gave the over-exposed and blurred image at bottom right. The second hole's image is top left and is also blurred because it was too big. The smallest holes produced the middle two images which are sharper and fainter. They compare well with what Stellarium shows for that time:

Stellarium shows the eclipse at 10:20 on 20 March 2015.

I do have access to more advanced equipment for solar observing, including telescopes with solar filters. So why did I choose to go with the primitive pin-hole approach? Because it's simple, there's little to go wrong and that makes for an experience that's more fun and interesting. In fact, the best moment, although it was fleeting, was when we saw the crescent Sun through the cloud with nothing other than our eyes. That was special.

The next solar eclipse visible in the UK won't be until 2017, and it'll be much less impressive. But if you're eager for more eclipse action, then the next lunar eclipse, in which the Earth's shadow covers the full Moon, will be on 28 Sept 2015.

Solar eclipse 2015

2015-03-18T12:01:00+00:00

There will be a partial solar eclipse on Friday 20 March 2015. A solar eclipse is when the Moon moves in front of the Sun and blocks out its light. A total eclipse is where the entire bright disk of the Sun (called the photosphere) is covered. By a strange coincidence of nature, both the Moon and the Sun are about the same angular size as viewed from the Earth - about half a degree, or a thumbnail at arm's length. This means that total solar eclipses are rare and brief, and visible from only a small area of the Earth's surface. Partial eclipses will be visible across a larger area of the Earth's surface and are slightly less rare.

Although Friday's eclipse is only partial in the UK, it will be covering around 90% of the photosphere across most of the UK. In Glasgow, where I live, it will be around 95% at maximum, but that 5% of photosphere still emits a lot of light, enough to damage your eyes, so let me issue the standard but important warning:

DO NOT LOOK DIRECTLY AT THE SUN AND DO NOT VIEW IT THROUGH BINOCULARS OR A TELESCOPE UNLESS YOU'RE ABSOLUTELY CERTAIN THAT A PROPERLY RATED SOLAR FILTER IS IN PLACE. PERMANENT EYE DAMAGE MAY RESULT!!!

There's a dramatic difference between even a 95% partial eclipse and a 100% or total eclipse, which is why a total eclipse makes for a very different experience. I saw the 1999 total eclipse from Alderney in the channel islands and I wasn't alone in finding it a primal and emotional experience. A partial eclipse may not be as intense as that, but it's still a rare and interesting phenomenon.

How to see it?

I plan to use a pin-hole camera. It's so simple. Just take a piece of card and a make a small hole it. The hole doesn't need to be pin-sized, even a few mm will work, but the smaller it is, the sharper the image will be. Don't look through the pin hole but hold it so that light passes through it to project an image of the Sun - see below. It may be low tech, but it'll do the job reliably and let you see the circle of the Sun being "eaten" by the Moon (and then vomited back up).

See http://www.solareclipse2015.org.uk for more details.

When and where to see it?

I've consulted some astronomy websites and the excellent Free and Open Source planetarium software Stellarium, and have constructed this table showing the timings and azimuth and altitude of the Sun for where I am in Glasgow. Go to http://www.timeanddate.com/eclipse if you want to get the exact timings for your location.

Phase	Time (UTC)	Azimuth	Altitude
Begins	08:30	117°	17°
Max	09:34	132°	24°
Ends	10:43	150°	30°

I should explain a few things. UTC is the same as UT or Greenwich Mean Time (GMT) and is the time that clocks will show in the UK until the clocks go forward. Azimuth is just compass bearing measured east of north, so 90° is due east and 180° is due south, so the Sun will be to the southeast (but starts closer to east) throughout the eclipse. The altitude is the angle measured from the horizon, so 0° is on the horizon and 90° is at the zenith, i.e. straight up. When the eclipse starts the Sun's altitude will be 17°, rising to 30° by the end. If you want to check whether you've got a clear view of that part of the sky before Friday, hold your hand with fingers apart at arm's length and from the tip of your thumb to the end of your pinky will span about 15° (Beware of this rule of thumb if you have abnormally sized limbs!)

I put together this brief movie using Stellarium to illustrate what the eclipse will look like from Glasgow, 30 metres above sea-level.

Please excuse the arty-farty lens-flare effect - that isn't something you will see.

Here's hoping for clear skies!

This blog

2015-03-11T17:27:00+00:00

I've been meaning to write a post on how I maintain this blog, and prompted by this request, I've finally made time to do it.

Pelican

I use a static site generator called Pelican. This entails me writing my blog posts as plain old text files, formatted using markdown, and then Pelican takes these files and produces a ready-made website in the form of HTML files, CSS files, images, and indeed any other type of file, such as audio or video, that I'd like to include. This heap of files can then be uploaded on to almost any web-server. There are no requirements that the web server uses php or ruby or any kind of database.

As a matter of personal preference, I stick mainly to writing text, and use other media only when I feel it makes sense, which is generally far less often than for most people. Pelican fits well with this, and also offers a kind of simplicity that appeals to me. Don't confuse that with being simple to use: a pencil may be simple, but using it to draw is not. I consider Pelican to be simple because it doesn't have a GUI (graphic user interface) to distract me from the complex business of stringing words together.

Pelican is Free and Open Source software and is released under the GNU AGPL v3. This means you are free to read, modify and redistribute the python source code. This means that, if I so choose, and sometimes I do, I am able to understand how it is processing my files and constructing my website. Although some knowledge and time needs to be invested in understanding Pelican, the reward is that I have huge freedom to control how my blog behaves and feels.

Theme

To customise my blog I have created a theme called ajc2, which you can find and fork on github. (My initial attempt - ajc - was a failure!) A theme is a fairly simple thing. It contains a few HTML template files, one for each type of page in my blog. For example, there's a template for a blog post, like this page, and another for the index page. There is also a CSS stylesheet that tells the browser what fonts to use, and what colour to use for the background, and so on. Various icons are also stored within the theme, such as the RSS and twitter icons on the index page.

Managing the content

The content of the blog is absolutely the most important thing to me, and so I keep it backed up and version-controlled using owncloud. owncloud is a web application that you can run on your own webserver to provide your own cloud services, hence the name. It can do a number of things, but primarily I use it to store my files and make them available from any device that I own (laptop, phone and tablet) and indeed from any web browser. In addition to this it also saves versions of the files, so that if I mess up a file beyond the ability of my text editor's undo button, owncloud can offer up a number of previous versions of the file.

Simple, but not too simple

As I said, Pelican is simple, and in almost every way this suits me. But there are a few drawbacks. The biggest one is that it doesn't allow comments by default. This didn't bother me at first because when my blog was on blogger I didn't get that many comments. That was not blogger's fault - my blog was young back then and I had no presence on social media. More folk are reading my blog these days, and I think they might leave interesting comments (appreciative or critical) if they could. This was brought home to me during the Scottish referendum when my blog posts were being shared on facebook (I don't use facebook). I hadn't really appreciated what was going until one night I sat down in a local pub and the person next to me said they'd liked my blog post and it had helped make up their partner's mind on how to vote!

Anyway, please accept my apologies for not having comments on this blog as yet, and rest assured it is high on my list of things to do.

A wee sneak peak

Want to take a peak over my shoulder whilst I'm working on my blog? OK, you've twisted my arm, I'll include just one image in this post, and you can click to enlarge it if you like:

UK spending and the 2015 election

2015-02-20T12:01:00+00:00

Government spending? I don't know what it's all about. I don't know any more about this thing than an economist does, and, God knows, he doesn't know much. -- The Best of Will Rogers

The UK will have a general election in May 2015 and a crucial issue is how the different political parties will approach the economy, so I thought I'd have a close look at what's proposed. But let me be clear on one thing: achieving a strong economy is not an end in itself, but an inseparable part of what we need for a fair and stable society. If we do not understand the existing economic system, or if our understanding is clouded by ideology or populism, then whatever the intention, it is unlikely a just society for the majority will result.

Spending 2008 to 2013

Let's start with the easy bit (or at least easier bit): the amount spent by the UK public sector since 2008 when the recession hit, which is shown in the graph below. The larger of two expenditures shown is the total spent in each financial year, and is called the Total Managed Expenditure (TME). The other figure is the departmental spending, or to give it its official name, the departmental expenditure limit (DEL): it is the portion of TME that was predicted ahead of time and is used to set expenditure of different government departments. First, let's concentrate on TME.

Source: This is the total managed expenditure (TME) and departmental expenditure limit (DEL) published by HM Treasury in their Public Spending Statistics. Note that these data are not corrected for inflation.

At first sight the graph appears rather flat and uninteresting, but to appreciate the change in spending driven by the UK's coalition government, you need account for inflation. Using the official CPI measure to do this produces this graph:

Source: The % annual changes in real terms for TME and DEL shown in the previous graph. Note that these data are corrected for inflation. Each data point shows the change between the financial year shown on the axis and the previous financial year.

This graph shows negative values from 2010-11 onwards, which get several percentage points more negative each year - in other words, cuts in spending. However, notice that the first data point shows expenditure increased during the last two years of the Labour government which departed in May 2010.

Although these figures are corrected for inflation, they do not account for demographic growth, so that, even if spending were frozen in real terms, the fact that the number of people in the UK is increasing, and that people are living longer, means that spending on things such as health and pensions must increase above inflation to keep pace. Recent estimates of demographic growth are about 0.7% per year, and so such spending should at least increase at that rate in real terms to maintain current standards.

Three Expenditure TLAs

Before going further, it's necessary to distinguish between the two components of the Total Managed Expenditure (TME). The first, mentioned above, is the Departmental Expenditure Limit (DEL), and the other one is the Annually Managed Expenditure (AME). The relation between the three is (thankfully) simple enough:

TME = DEL + AME

The DEL is managed over three-year Spending Reviews by the government and it covers aspects of spending that can be more reliably predicted, such as salaries for government employees and running costs for buildings. The AME, as its name suggests, is managed each year and is much less predictable, involving social security, public pensions and other spending not immediately within Central Government control, e,g, that of local councils and also money going to Europe. With all this in mind, it makes some sense that plans for future spending are mainly concerned with the DEL component.

Future spending plans

The spending plans currently being proposed by the main political parties are based on how and when they would reduce the budget deficit to zero, also referred to as a balanced budget. A deficit arises because public sector spending exceeds what comes in from taxes and other revenues. Usually, the government chooses to cover the deficit by borrowing, and this means the public sector debt keeps rising each year. See my previous series of blog posts for more detail on this. The question of whether deficit reduction by cutting spending is a credible economic policy is something I'll leave to the end of this post, for now I want to concentrate on the plans and numerics of what is proposed.

The figures I quote below are all inflation-adjusted, being in 2014-15 prices, and come indirectly from the Office of Budget Responsibility (OBR) and HM Treasury via the Resolution Foundation, which provides more in-depth analysis and detail than I provide here.

The coalition government plan is to have the budget in surplus by 2017-18 - that is, the amount spent will be less than revenues received in the financial year 2017/2018. According to this plan, a reduction in DEL (i.e. a cut, sometimes eupemised as 'consolidation') of £45 billion between now and the end of that year will yield a surplus of £13m.

If Labour form the next government then they have committed to bringing the deficit to zero by the end of the next UK parliament, which will be in 2019/20. To do this will involve a much smaller reduction in DEL of about £7 billion over 4 years. That said, Labour have left themselves quite a bit of flexibility and could take a number of different routes to their end goal. You could interpret this as leaving the door open to making deeper cuts sooner, or equally, as an admission that there are many factors outwith a government's control, including many aspects of the wider economy.

In summary, if the Conservatives form the next UK government we'll see DEL cuts of the order of £15 bn per year, whereas if it's Labour then it's likely to be around £2 bn per year.

Assumptions

I think it's important to take these predictions with a pinch of salt. According to the coalition government's own plans, made at the start of the current parliament, the budget should already be in balance by now. The main reason that it isn't is because the economy did not grow at the rate predicted and, more pertinently, tax receipts, especially income tax receipts, failed to bring in the revenues they expected.

The same failure could well be repeated and render the above predictions invalid if assumptions made about growth and tax receipts are too optimistic. Or, it could be that unexpected growth in the economy means that the budget can be brought into balance by Labour's 2019-20 goal without the need for such deep cuts, or indeed any cuts at all, as Paul Mason pointed out on his blog.

So given the great uncertainty in the assumptions made, I'm not so interested in the absolute numbers that these predictions generate, but rather the difference between the numbers and the implications of how they would impact on our society.

Political comment

I'm very frustrated that we are being offered a choice between austerity and austerity-lite - in other words, a choice between two scales of cuts to public sector spending. Remember too that these cuts will have a much harsher effect on society than those to date because the "easy" savings, especially justifiable efficiency savings, will have been made first. That said, a common fallacy I've heard, mostly in social media and everyday conversation, is that there is no difference between the two big party's plans. To be clear, the Conservative's cuts are at least five times larger than those of Labour's under the above predictions.

In the UK we do of course have more than two parties. The Liberal Democrats also wish to zero the deficit by 2017/18, and propose something between the deep cuts of the Conservatives and the shallow cuts of Labour. Nicola Sturgeon - First Minster of Scotland and leader of the Scottish Nationalist Party - made an interesting speech in London recently. At first I was encouraged by her focus on social justice and welfare, which plays very well to the Scottish electorate at present, and her argument that these require an increase in spending. Except, she then lost me when she joined the other party leaders in saying that the deficit is "hugely important" and so increases should be limited to real terms to 0.5% in departmental spending (and she also confused spending and investment in that part of her speech). This will not keep pace with demographic growth, and as Douglas Fraser pointed out, it is austerity re-badged. The Resource Foundation showed that the difference between SNP and Labour plans are somewhat insignificant; a point also made, albeit in a different way, by Paul Mason. The figure that made the headlines was that the SNP were proposing £180 bn of extra spending, but that was compared to the Conservative's plan. I was left with an image of a person proudly standing tall, but in a five-foot hole in the ground.

But there's another recent historical fact that I touched on earlier that should make you stop and think - during the last two years of the previous Labour government, in the immediate aftermath of the 2008 crash, departmental spending grew at over 4% in real terms. Not only that, but the economy was showing a marked improvement at that time, experiencing five quarters of growth which ended when the coalition government came to power. In addition, as I showed in this post, the deficit post-2008 came about not simply because spending shot up, but because tax and other public sector revenues fell back. All this points to a solution: we reverse what happened in 2008. Public sector revenues will rise if the economy can be improved and that in turn will cause the deficit to drop. How should the economy be improved? The public sector should invest - not just spend - on things that the economy, and more importantly, society, needs, such as houses, transport, clean energy generation and so on. All these things will create jobs which will push wages up in real terms thus creating a demand for goods and services that will boost the economy. The distinction between spending and investment is crucial; the latter will bring you a return in the future.

Why does the public sector need to provide the investment - can't it be left to venture capitalists and businesses? Well, the private sector on its own has not found a way to do this after 7 years of economic doldrums, and despite the repeated claims, the austere act of cutting public spending has hindered rather than helped. Although the UK economy is growing, and employment rising from its low point, both GDP per capita and wages in real terms are still below where they were 7 years ago, and that hurts both private business (lower demand) and the public sector too (lower tax receipts). More importantly, the knock-on effect is to worsen the hardship of the people who already have the least in our society.

My view is therefore that we have trimmed public sector spending as far back as it can go, most likely too far, and we should instead increase public sector investment, even if borrowing is required. The UK public sector can borrow cheaply now and that investment - as the word suggests - will bring returns in future years in the form of a healthier economy and so higher tax receipts, which can balance a budget and even create a surplus if desired.

If you're willing to move further beyond prevailing dogma, then we don't even need to borrow to invest - the UK can, if there's political will, simply create the money. It could do something like quantitative easing, but invest more usefully, rather than just buying back its own bonds. The best proposal we have from mainstream political parties is the SNP's modest 0.5% increase in departmental spending, which is about £1.5 bn a year, and doesn't even keep pace with demographic growth. Comparing this with the severe £15 bn per year Conservative-planned cuts strikes me as a false dichotomy. Instead I'm thinking of the £200 bn that was created and then spent between March and November of 2009, followed by another £175 bn in subsequent years. The £375 bn created for QE is more than one year's worth of DEL, and a twentieth of this amount - yes, 1/20 - equates to a 5% real terms increase in departmental spending which, if you cast your eye back up to the last graph, is about equal to the increase in DEL in the last year of the previous Labour government, and roughly ten times what the SNP propose now.

I can't help thinking there is both a poverty of understanding and of ambition with our politicians, and that this is because they, and others, are too distracted by ideology and party political media battles to realise that their actions, with good intent or otherwise, will do exactly what no politician should ever do - make the poorest in a society poorer.

eBooks - HTML and EPUB

2015-01-16T10:47:00+00:00

I've written a novel and I'm in the final stages of revising it. Reading it on the computer screen was acceptable in the first few reviewing passes, but now I feel the need to read it as I read many books - on my phone. Not only does this give me a fresh perspective on what I've written, it also forces me to avoid editing the text directly as I go, which distracts greatly from reading it as a "reader", if you follow me. If I see something I don't like (and it's still alarmingly often!), I am restricted to leaving a short note in the phone app. I was using the Cool Reader app, but due it's weird user interface, I decided to switch to Moon Reader Pro.

So, to do this I had to convert the book from the format in which I wrote it to a format suitable for a mobile device. This blog post is based on the notes I kept whilst doing that.

I wrote the text using LibreOffice writer (though I did try the distraction-free application FocusWriter for a while) and I saved it in .odt format. LibreOffice is a free and open source alternative to Microsoft Office and .odt is the open file format that it prefers to use (though you can use .doc and .docx if you prefer). I used text formatting sparingly: a header style (large font, bold) for Chapter headings and italics in a few places in the text. I broke up chapters into sections using asterisks with blank lines above and below. Paragraphs were separated with blank lines (i.e. two newline characters) and no new-lines were used inside a paragraph, i.e. I let the text editor wrap lines for me. No other formatting was used.

I decided to use ePub format for reading on my phone as this standard was clear, well-documented and also widely supported. There were also good technical reasons for using it. I tried two ways to do the conversion. The first was to use an extension for LibreOffice called Writer2ePub and the other was to use a separate application called Calibre. I got both to work, but it seems now that Writer2ePub isn't working so well with LibreOffice version 4 - see this comparison page.

In both cases, the output was acceptable, in that I could read it on my phone, but the chapters weren't automatically recognised as such and so there was no useful table of contents. Also, the blank lines I'd used to separate paragraphs, which were helpful when I was writing, resulted in large, wasteful and distracting gaps when viewed on my phone. I'm sure both issues could be addressed in Calibre, and perhaps in Writer2ePub, but I decided on a different path after listening to a couple of podcasts by Jon Kulp on his experiences with eBooks - a music one and with Gutenberg eBooks. I recommend you listen to both, but if don't have time, then there are some good links in his show-notes.

What I learned from Jon was that instead of starting from a .odt file (or .doc or .docx file), I could make the conversion to eReader (and other) formats simpler by using HTML and some simple CSS. That said, if, unlike me, you like rich GUIs with oodles of buttons, then this way of writing may not be to your taste. But before you proclaim yourself a luddite and unlikely to be able to write HTML, do have a look at a file or two - I provide a link to an example below. The amount of HTML you need to learn is tiny, and it's much easier than writing a web-page.

The other good reason to go with HTML and CSS is that it is used inside the ePub format - this is the good technical reason I referred to above. This means that you could create the ePub files manually, but I chose to let Calibre to do the conversion work because some fiddly meta-data files are required by ePub. But understanding the HTML and CSS you created means that some post-conversion tweaking is made easier, and Calibre has features to help with that too.

I should also make clear that HTML is simple enough that I am comfortable editing it directly, i.e. I could ditch the need for .odt or .doc files and full-blown word processors such as LibreOffice Writer or Microsoft Word. A simple text editor will suffice, which in my case is Kate.

So, my first task was to get my text out of a .odt format and into HTML format. LibreOffice can export to an .html file, but when I used it, the HTML it produced was full of extraneous markup. I was after simplicity and having loads of <p superfluousAttribute="distractingValue> tags in my .html file was not on. No criticism of LibreOffice here - just that automated conversion tools inevitably generate output that is formatted to be syntactically correct, but not very human-readable.

Instead, I opted to save the file as text from LibreOffice and then do some tidying up on the command line. I've included the gorey details below if you want to see them. Details aside, the two main things were to enclose all paragraphs inside a pair of <p> and </p> tags, and chapter headings inside <h1> and </h1> tags. Next I created a simple CSS file based on the example provided by Jon Kulp and was able to get my first look at my eBook by opening it my web browser (firefox). I was quite pleased with the result. I'm not ready to share my novel yet, but you can look at this example HTML file (NOTE: it's readable but ugly in a browser, but if you right click on it you should see the option to read the source code) and also the CSS file that goes with it. The main difference between my CSS and the one I took from Jon Kulp is the removal of some styles I didn't need and the addition of the in-chapter section break:

.secbreak {
    text-align: center;
    text-indent: 0;
}

This means that this snippet of HTML will produce the asterisk that breaks sections within a chapter:

<p class="secbreak">*</p>

Once I had my .html file, I started up Calibre, clicked the big, red "Add books" button, selected the .html file and it loaded it and (without prompting me) turned it into what Calibre calls "ZIP" format. Next, I selected that book in the main pane and hit the "Convert books" button. By default it selected ePub as the output format. I then added as my name as the author, and at this point you can enter any meta-data you wish, and add a book cover graphic.

Next I went to the "Look & Feel" section (left panel of the convert window). I left all settings to defaults, but I pasted in the contents of my CSS file in the "Extra CSS" pane. There's much more you can do, as described here, but I just hit the OK button and after a few seconds it completed the conversion and a link to the "EPUB" version appeared on right of the main Calibre window. Clicking this opened a preview of my book complete with its table of contents which let me jump straight to any chapter. Job done!

Manual conversion of odt to html

Here are the sequence of steps that I took to convert an odt to an html file.

Open the .odt file in LibreOffice and go to File->Save As in the menu. In the dialog window choose File type as "Text - Choose Encoding (.txt)" and check the "Edit filter settings" option and click OK. In the filter, I selected LF (line feed) to be the newline character and UTF-8 encoding. The file is saved as book.txt.

The following steps were executed on the command line in linux. If you don't have access to the command line, or you don't like it, you may be able to perform some operations using a "Find and Replace" feature in a good text editor.

Replace Chapter * with <h1>Chapter *</h1>

sed 's/^Chapter.*/<h1>&<\/h1>/' book.txt > book_h1chapters.txt

Remove double newlines that I'd used to separate paragraphs

sed '/^$/d' book_h1chapters.txt > book_nodoublenewlines.txt

Insert <p></p> tags around each line:

sed 's/^.*$/<p>&<\/p>/' book_nodoublenewlines.txt > book_paratags.txt

Turn <p><h1> into <h1>

sed 's/^<p><h1>/<h1>/' book_paratags.txt > book_ph1remove.txt

Turn </h1></p> into </h1>

sed 's/<\/h1><\/p>/<\/h1>/' book_ph1remove.txt > book_h1premove.txt

Then I manually edited it in my text editor (loaded as UTF-8) to do the following:

wrapped whole lot in <html></html>
added <head>...</head> containing <title>...</title> at the start
added <body>...</body>
changed Prologue from p to h1 tags to make it a chapter
saved it as book_manual.html,

To make the HTML more readable whilst I edited it, I decided to insert blank lines after the HTML tags - these have no effect on the final formatting.

sed 's/<\/.*>/&\n/' book_manual.html > book_final.html

Scottish referendum polling - retrospective

2014-11-18T21:34:00+00:00

It's two months since Scotland held its referendum and so I thought I'd collect together some data I was looking at back then. I didn't write it up at the time because I'd become a little weary of the whole subject, but now the time feels right for a retrospective.

My aim here is to analyse all the polling data in light of the final result and place it in a longer historical context. I strive to prevent any bias from creeping in, but I feel that honest subjectivity is a more realistic goal than objectivity, so please bear in mind that my preference is for Scotland to remain a distinct country within the UK. You can read more on my reasons in this first blog post on the politics and economics and a follow up post on inequality.

Comparing like with like

It's common to hear statements such as "45% of people in Scotland voted Yes and 55% voted No". This is in fact not true because it ignores the fact that 15% of registered voters didn't vote. The correct statement is that "45% percent of people who cast votes voted Yes and the 55% voted No".

Now, don't get me wrong, 85% is a fantastic turnout, especially compared with the last Scottish election (50% turnout in 2011) and the previous UK election (65% turnout in 2010). But it still means that 85% of 55%, which is 47% of registered voters voted No, and 38% of registered voters voted Yes. I could insist on even more precision, in that it's estimated that 97% of those eligible actually registered to vote. However, that 3% gap is not only less significant, it's also more uncertain because some people wish to avoid making themselves known to the authorities.

Does this attention to detail matter? Definitely, because if you ignore it, you can end up either estimating double the swing to Yes, or else concluding that there was no swing at all. There are many examples to choose from, but let's take one of the most prominent: a Guardian piece by Irvine Welsh that appeared on the front of the Observer on the Sunday after the referendum. It contained this:

The process and the subsequent debate, which they won handsomely, took support for independence from around 30% to 45% and heading north.

The first half of that sentence is a matter of opinion, and I'm comfortable disagreeing with Mr Welsh on that, but the second half is a statement of numerical fact - and it's wrong. As we'll see below, the 30% refers to the Yes figure in a Yes/No/Don't Know trio from opinion polls, so the correct figure to compare it against is not 45%, but 38%.

You can judge for yourself how public opinion changed by looking at this graph of polling results from January 2012 up until the referendum on 18 Sept 2014:

Source: A plot of all polls from 2012 to 2014 prior to the referendum. Don't Knows are kept separate. Data after 31 January 2013 is from a CSV download from whatscotlandthinks.org, and data before that is obtained via the list on Wikipedia which I've checked back to original publications. The horizontal lines show the result of the actual referendum, though bear in mind that the green line represents "Did not vote" which doesn't necessarily mean the same as "Don't know".

The scatter is very large, but it's clear that the Yes vote slowly increased during 2014. There's a weaker indication that Don't Knows increased during the same period. In contrast, it's hard to see a trend for the No vote amongst the scatter. In the weeks before the referendum, the Don't Know percentage drops, and both the Yes and No percentages rise as people make up their minds. What's striking is that those final polls overestimate the Yes and the No vote percentages, and their Don't Know percentages underestimate the 15% of people that did not vote.

Much of the scatter can be explained by the different questions and approaches used by different polling companies. For example, those that use online panels where people sign up to take part are probably biased in favour of people who have made up their minds, and so produce lower Don't know figures than random-dialled telephoning or in-person interviews. Such non-online polls were less common, but make up most of the Don't Know points plotted above 20%.

So, my version of Irvine Welsh's 30% to 45% statement is the more verbose but carefully qualified:

Opinion polls from 15 October 2012 (the Edinburgh agreement) until the end of 2013 show a support for Yes in the range 25% to 45%, No in 40% to 60%, and Don't Knows mostly sitting in 10% to 20%. The final result was that of all registered voters, 38% voted Yes, 47% No, and 15% did not vote.

So why did Mr Welsh compare the 30% figure with 45%? Because it makes for more dramatic sounding news and exaggerates the swing to Yes. By making selective comparisons, it's just as possible to argue - disingenuously - that there was no swing to Yes at all. Take Panelbase, who consistently found a higher Yes vote than other polling companies: their 12-20 November 2013 poll found 38% Yes; 47% No; 15% Don't know - identical to the final result. So, on the basis of that one poll I could claim (but I do not) that the Yes campaign made zero progress in the last year of the campaign. I could go back another year, and claim no progress was made in the two years from Panelbase's 9-19 Oct 2012 poll (37%/45%/17%), and a similar result can also be found in the 27-29 Jan 2012 Ipsos Mori poll. However, exaggerated statements such as these are seldom made in public because "there was little change" does not make for interesting news. I would also hope that reputable journalists would know better than to misrepresent facts and figures in this way. I'll not mince my words - Irvine Welsh is guilty of such a misrepresentation, but then his talent is for writing fiction, not data-driven journalism.

Real figures for headlines

Let's take the 45% figure for Yes, because it is most quoted in the media, and imprinted on many an avatar in social media, and estimate a reasonable pre-campaign figure to compare it against.

By taking the average of all eight polls in 2012 - the year of the Edinburgh agreement - I find the averages for Yes/No/Don't know were 32%/53%/15%, which can be compared to real result of 38%/47%/15%. If we now exclude the Don't Knows then the eight polls of 2012 average to 38%/62%, which we can compare with the real result of 45%/55%. (Note: All of these averages were weighted by the sample size, but this made little difference because almost all polls have a sample size close to 1000. The exception was a Lord Ashcroft poll ending 9 May 2013 which had a sample size of 10,007, but I excluded this from the dataset because its quoted percentages added to 108%.)

The table below shows the averages for 2012, 2013 and 2014, using data from a CSV download from whatscotlandthinks.org.

Year	No. Polls	Yes	No
2012	8	38%	62%
2013	31	39%	61%
2014	71	45%	55%
Actual	-	45%	55%

It's clear from this there was hardly any change during 2012 and 2013, but that the Yes vote increased by 6pp during 2014. However, bear in mind that polls were held more frequently as the referendum day approached, so the 2014 average is not evenly weighted over time. In particular, I wouldn't attach too much significance to the fact that the 2014 average is the same as the actual result.

You can better appreciate the data behind these averages in this graph:

Source: A plot of all polls from 2012 to 2014 prior to the referendum. Don't Knows are excluded. Data from 1 Feb 2013 is from a CSV download from whatscotlandthinks.org, and data before that is obtained via the list on Wikipedia.

Notice that this graph is perfectly symmetric - you can fold it along the 50% line and each red dot will land exactly where there's a blue dot. This is because we've discarded information by excluding the Don't Knows leaving us with two percentages that must add to 100%. So, unlike the graph above, which shows the Yes increase with little decline in No, this one portrays a false symmetry. The advantage is that it's a bit simpler and clearer, but it conceals information on when, and if, people came to a decision.

With that in mind, there are two things to note about this graph.

Firstly, the drift towards Yes began before the official campaigns got underway on 30th May 2014. This can be intepreted as an effect of pre-campaign activity in local meetings and in social media such as Facebook and Twitter, but also, as you'll see below, there's historical evidence that shows a rise in support for independence whenever Scottish constitutional issues are raised.

Secondly, there is little correlation with high profile announcements and events in the mass media, notably the high profile TV debates that took place in August and the hurried and desperate-seeming visits and vow of the Westminster leaders. I wondered if the scatter hid any such correlation, but I found none even when plotting graphs of data from a single pollster, for which I used both the BBC's poll tracker and the tools on whatscotlandthinks.org. In fact, the only sudden change that can be seen in the above graphs is the narrowing of the gap between Yes and No at the start of September 2014, which took most people by surprise precisely because it wasn't obviously prompted by any event or disclosure of information. With hindsight, the change seems more likely to be a change in how people responded to polls than a simple shift in voting intention. It may be related to the Spiral of Silence (or Silent No) effect that I discussed in a previous blog post.

The long view

We can look at data going back to 1978 thanks to Ipsos Mori (formerly called just Mori). The cavaet they state is that the question being asked and the way it has been presented has changed over time, but, of course, that's also true of the previous two graphs.

Source: Data from Ipsos Mori, taken from Figure 2 in the House of Lords library note LLN-2014-027.

This data shows percentage support for independence that's most directly comparable with the 38% of registered voters who voted Yes in the 2014 referendum. Please do not compare it with either the 45% Yes vote figure, or the above graph that excludes Don't Knows. To double check this point, I verified that the last two data points shown in this graph are for the Ipsos Mori polls for STV which gave Yes/No/Don't trios of 34%/57%/10% in December 2013 and 32%/57%/11% in February 2014.

As you can see, the majority of polls since the mid-1980s show support for independence has been between 30% and 40%, though there was a pronounced dip to 20% in the wake of the financial crisis of 2008. It reached an all-time peak of 47% in 1998 just after the Scottish people approved devolution in the 1997 referendum, and it only matched this level again in the STV/Ipsos Mori poll a few days before the 2014 referendum (not shown on this graph).

So, taking the long view, the referendum result of 38% for Yes, or 45% excluding Don't Knows, is not at all unprecedented, but consistent with the run of Scottish public opinion over the last three decades.

Final thoughts

Rolling issues up into a single number is always a gross oversimplification, and although 38% (or 45% if you prefer) is not unprecedented, the circumstances in society, politics and the economy are far from normal. What's different now to the previous peak periods in support for independence - specifically, devolution in 1998 and the poll tax in the early 1990s - is that dissatisfaction with traditional politics and inequality is evident throughout the Western world and is not just confined to Scotland. Low voter turnouts, upsets to traditionally dominant political parties, social unrest and even riots are occurring across Europe, especially in Italy, Greece and Spain where the financial crisis hit hardest, but also over the Atlantic in the USA. An understanding of the changes occurring within the UK has to be understood in the context of this more global situation.

UK in Recession part 3 - borrowing

2014-11-12T21:01:00+00:00

This is the third post in this series on the UK in recession. This time I'll look at how the deficit (described in my last post) leads to borrowing (PSNB ex) and debt (PSND ex). Unfortunately this means I have to start off with rather dry and possibly boring definitions, but be assured this is so I can enter more interesting territory later. As before, I'll concentrate on the data first and confine all political commentary to the last section.

Full sources are given below, but you can also check the spreadsheet I've constructed to produce all the graphs in this post: download ods or download xls.

Previous posts in this series:

The connected definitions of public sector deficit, net borrowing, net cash requirement and net debt that I discuss below can be seen in the diagram on page 10 of the Public Sector Finances, August 2014 document.

Public Sector Net Borrowing (PSNB ex)

The PSNB ex is the official figure for what the UK public sector borrows in a given time period. (How it is borrowed and why it has to be borrowed will be discussed later.) It is the headline figure published by the ONS and it is the one most often used by both journalists and politicians, even though they still prefer to use the word 'deficit' when discussing it. Remember that the "ex" means that it excludes temporary effects due to bailing out of banks and most aspects of quantitative easing (see the last post's discussion of APF's interest payments).

The PSNB ex for a given time period (usually month, quarter or year) is given by

PSNB ex = Deficit + Net Investment

Investment is the amount spent by the UK public sector on buying capital assets. This is not counted in the expenditure used to calculate the deficit because capital assets continue to be useful in the future. For example, money spent on building a hospital in a given year will create a capital asset that will be used to improve the health of people in subsequent years. In this sense it is an investment in capital, but unlike the usual meaning of that term, the return on this investment is not monetary, but societal. In contrast, paying a nurse to work in a hospital for a given year would come under government expenditure because that money only "buys" the nurse's work in that year. Similarly, the purchase of medicines, bandages and food is all included in expenditure because these items are consumed (quite literally for food!) in a one-off use and not available to be used again.

So the reason investment is treated separately is because the money is not consumed but transformed into an asset. It's possible that a public asset may be sold, and such a sale, or disinvestment, or divestment, would be subtracted from the total, hence "net" investment. You may recall from my last post that in addition to expenditure, a deficit can be increased by depreciation; this accounts for the fact that assets lose value over time.

The graph of PSNBex is shown below along with its split into its deficit and net investment components.

Source: Office of National Statistics (ONS) Public Sector Finances August 2014 release. The data plotted are the labelled -JW2T, -JW2Z, -J5II on the "psa1" tab of the data tables spreadsheet. Note that these data are not corrected for inflation.

Notice that, by definition, if you stacked each blue column (deficit) on the adjacent red column (net investment) it would exactly equal the the yellow column (PSNB ex). Also, before 2008, net investment generally made up the larger share of PSNBex, but thereafter deficit dominated.

Cash vs accrual accounting

If life were simple, Public Sector Net Debt (PSND ex) for a year would be calculated by adding PSNB ex to the previous year's PSND ex. But life isn't so simple and some understanding is needed of basic accounting practices.

In cash accounting the transaction is deemed to take place at the time when the cash was transferred. Consider the example of a company having their windows cleaned on the 1st of October. The cleaning firm does the work and issues an invoice to the company with that date on it. If the company pays the window cleaning invoice on the 30th October, meaning that money left their bank account on that day, then that is the date that will appear in the company's cash accounts. However, if the accounts were done on an accruals basis then the date of the transaction is when the liability came into existence, which would be the invoice date of 1st October.

Public Sector Net Cash Requirement (PSNCR ex)

PSNCR ex is calculated on a cash basis whereas PSNB ex is on an accruals basis. As such, there's no simple formula relating the two, but the differences are caused by two distinct effects:

timing differences - as described above, the time between when the liability arises and the cash changes hands.
financial transactions - purchase of an asset does not affect PSNB ex if its value is equal to the amount of money paid for it, but the flow of cash might need to be factored into the PSNCR ex

In both cases, if you look over long enough time periods you'll find these differences are much less pronounced. For example, for the financial year to end of March 2013, PSNB ex was £119.4m and PSNCR ex was £95.5m, but if we add the preceding three years to these figures we find they are £518.8m and £524.3m. (These data can be found on the "psa3" tab of the data tables spreadsheet.)

Public Sector Net Debt PSND ex)

If you take PSND ex at the end of last year and add this year's PSNCR ex then you will get this year's PSND ex. Well, almost. The following effects mean that changes in PSND ex don't just come from adding PSNCR ex:

variations to do with gilts (UK government bonds) and their value and when changes accrue
fluctuations in exchange rates that alter the sterling value of liquid assets in PSND ex
reclassification of bodies into or out of public sector

Taken together these usually amount to no more than a few percent of PSND ex prior to 2008, i.e. most of the difference from year to year does come from simply adding on PSNCR ex. Since 2008 the above three effects have mostly been larger. You can see this is the "psa4" tab of the data tables spreadsheet

The graph below shows the PSNDex (blue) and with PSNCRex (red) stacked on top of it.

Source: Office of National Statistics (ONS) Public Sector Finances August 2014 release. The data plotted are the "psa3" and "psa4" tabs of the data tables spreadsheet. Each column shows PSND ex for the year to end of March and stacked on top of it is the PSNCR ex for the following year (from April 1st). Note that these data are not corrected for inflation. The PSNCR ex for 2014/15 is of course not known, so there is no red portion for 2014.

The stacked plots lets you see that PSND ex increases by about the PSNCR ex each year. For example, the top of the whole column for 2008 (PSND ex plus PSNCR ex) is at about £700 bn, which is level with the top of the blue column (PSND ex) for 2009. Notice that the top of the whole column for 2013 falls short of the 2014's blue column. In this case there is a £39 bn increase in PSNDex that is not accounted for by PSNCR ex. Looking at the row for 2013/14 in the "pas8d_1" tab, it's clear that there is a high entry in the column labelled "Bank of England contribution to PSND". The footnote indicates that most of this is due to APF, i.e. buying of gilts, but I'm not clear why this amount effects the PSND ex but not PSNCR ex.

How is the borrowing done?

Most of UK public sector borrowing - £1300 bn at time of writing (tab "ps8a", column labelled BKPM) - is in the form of gilts, which are UK government bonds. Rather than going to a bank, as you or I might do, and asking for a single big loan, the UK public sector carves its debt up into small chunks, called government bonds. Each bond can be bought for £100 and represents that amount of debt. When you buy a bond, you become the owner of £100 of public sector debt and in return for this you will receive a fixed payment twice each year (called the coupon for a UK gilt) and after a specified number of years the bond expires and you'll get your £100 back.

New gilts are issued by the UK Debt Management Office (DMO) on a regular basis and are usually bought by pension funds, investment companies, private banks and even by members of the public. You can buy gilts in the open market but you will find that the price is not necessarily £100. The amount you pay for a gilt will determine its yield. For example, if you buy a bond direct from the DMO for £100 with a stated return of 2% then you will receive £2 per year, and the yield will be the same, i.e. 2%. However if you bought that gilt in the open market later on for, say, £200, then you would still only receive £2 per year for it, and so the yield would be £2 out of £200, or 1%. In other words, if the price of gilts goes up, the yield goes down, and vice versa.

So, if the UK public sector wishes to increase its debt it issues more bonds, and if it wishes to reduce its debts, it simply has to let existing bonds expire and not issue new ones. The UK has never defaulted on its debts and for that reason (and because a state with its own currency can't go bust, of which, more in the next post) gilts are seen as one of the safest and lowest risk of all investments. There is no shortage of people or firms that wish to buy UK gilts, and pension funds in particular own a significant number of them.

Why does the UK public sector borrow?

You may know, possibly from reading my previous post, that Quantitative Easing (QE) is the process of creating money to buy government bonds. In the UK, the Bank of England has been creating money and lending it at a low interest rate to its wholly owned subsidiary, the Asset Purchase Facility (or APF), which then uses it to buy gilts. To date, £375 billion has been created and spent doing this, and as a result 30% of UK public sector debt is currently owned by the APF, which is owned by the Bank of England, which is part of the UK public sector. In short, the UK public sector owes itself about £375 bn of its own debt.

Why, you may ask, doesn't the UK public sector just write-off this debt? After all, it's silly to owe yourself money. The answer is that it cannot do so without violating the Maastricht Treaty by which the European Union was established. In short, countries in the EU are forbidden from creating money to fund spending or pay off their own debts. QE is permissible, as long as the gilts are purchased in a particular way, hence the APF and the low interest "loan" from the Bank of England, and there is an intention to sell the gilts back into the market at some point.

Political comment

I'm pretty sure the detailed cash-flow of the UK public sector shows that created money plays a significant role in government spendinng: in fact, QE is just a big and obvious example of that. By this I mean that the UK public sector does not need to wait for money to come in from tax, other income or borrowing before it can spend. It can spend first, and reconcile its accounts in retrospect, as long as it does so according to the rules set out in the Maastricht Treaty.

Political statements often make it sounds like the UK public sector, and particularly the government, needs to balance its books in the same way as a household. But it should be clear from all that's said above, and the wealth of evidence contained in the ONS's Public Sector Finance spreadsheets, that the UK public sector is fundamentally different from a household. For starters, there is absolutely no equivalent of QE in a household. Furthermore, a household with a zero bank balance cannot spend money without first securing a loan, credit agreement or some kind of overdraft with a bank or other lender. The UK public sector has much more freedom and is limited only by its voluntary agreement to the Maastricht Treaty.

This leads to the interesting question of why the Maastricht Treaty places such restrictions on EU states? The simplest response is that money creation, or to be more pejorative, printing money, has very negative connotations, such as the hyper-inflation in Germany after the first world war. But QE in the years after 2008 created hundreds of billions of pounds and this money is still in the economy, yet inflation is currently hovering around 1%, which is not only historically low, but also worryingly low. My next blog post - the final one in this series - will look at why many of the doom-laden predictions of QE's money creation turned out to be completely false, and what implications that has for promoting economic recovery.

Cadfael

2014-10-27T11:56:00+00:00

rother Cadfael is a fictional Benedictine monk created by the author Ellis Peters who served as a Herbalist at Shrewsbury Abbey in 12th century England. Against the historical backdrop of "The Anarchy" - a civil war between King Stephen and the Empress Maude - Cadfael operated as a detective who solved many murders.

I've been watching the TV adaptations in which Cadfael is played by Derek Jacobi. He, as always, is superb and matched by an excellent supporting cast of regulars and guests. The locations and sets are suitably authentic and, as far as my limited knowledge permits me to judge, it doesn't skimp on historical accuracy. Although the basic structure is classic "whodunnit", there is a unique atmosphere created by aspects unfamiliar to a modern audience: the austere life of monks; the power tensions of lords, church and king; the brutality and lack of modern law-keeping and justice.

Cadfael is portrayed as being both devout and pragmatic in his application of his religious rule and teachings. Before becoming a monk he was a soldier in the crusades, and he has retained his skill sufficiently to defend himself and vulnerable others. But it his knowledge of the real world and everyday people that sets him apart from other monks. Many monks began their vocation before they had become adults and are completely ignorant of the outside world. Cadfael is often mistaken by others as being such a monk, which allows him to move unregarded and extract information with apparently innocent questioning.

Some aspects of the stories are familiar. The motives of the murderers are various combinations of love, lust and greed. And it's interesting to see that humans almost a millennium ago were much the same as we are now, even though they don't have cars, electricity or even metal spoons. Cadfael's ally, Lord Hugh Berringer, the under-sheriff, provides extra muscle and sword-power on many occasions. What makes these stories immediate and engaging is that we see the plot evolve from the perspective of Cadfael and Berringer - the goodies - who are in fact characters with contemporary values of justice and fairness: Cadfael the detective, and Berringer the policeman.

There's something very appealing about monastic life to me. The simplicity of it; the lack of choice; the order. But I wouldn't last a day as a monk, as I'm utterly unable to conform and have no religious belief at all. So why does it appeal? Probably because it is the antithesis of my current existence and a part of me yearns for such simplicity and order.

There is one aspect of monastic life from which we might benefit: the drive to do deeds for their own sake, or for the sake of others' or for some greater purpose, and not simply for selfish reasons. Fulfillment in life does not come from sitting around relaxing or spoiling yourself with material luxuries, but from hard work and mental discipline. Thankfully, my sense of fulfillment doesn't involve solving murders in 12th century England.

UK in Recession part 2 - deficit, income and spending

2014-10-21T15:24:00+01:00

This is the second post in the series. I'll pick up where I left of at the end of the last post and investigate the deficit and how it relates to public sector spending and income in the time period of interest, from 1997 to 2014. As before, I'll concentrate on the data first and confine all political commentary to the last section.

Full sources are given below, but you can also check the spreadsheet I've constructed to produce all the graphs in this post: download ods or download xls.

Previous posts in this series:

UK in Recession part 1 - GDP and debt

Deficit

The UK public sector spends money on goods, services, paying staff (including civil servants, doctors, nurses, teachers etc), paying state pensions and welfare, paying interest on its debt and many other things besides. The public sector receives income from VAT, income tax, corporation tax and interest payments and dividends, amongst other things. If what the public sector spends - its expenditure - is larger than its income - its receipts - then there is a budget deficit. Conversely, if receipts exceed expenditure there is a budget surplus. Confusingly, the budget surplus/deficit is sometimes referred to as just "the current budget" or "the budget value". A more recent term being used in Europe is SOCB - Surplus On Current Budget - which will be a negative value for a deficit.

The deficit is calculated as

Deficit = Expenditure + Depreciation - Receipts

as defined on page 10 of the Public Sector Finances, August 2014 document.

Here is the graph of the deficit and its three components.

Source: Office of National Statistics (ONS) Public Sector Finances August 2014 release. The data plotted are the receipts total labelled ANBV on the "psa6b_1" tab of the data tables spreadsheet. The Expenditure, Depreciation, Deficit are labelled as ANLP, NSRN, ANLV on the "psa6b_2" tab. Note that these data are not corrected for inflation. The depreciation is clearly a relatively small and slowly varying so we needn't concern ourselves with it.

What's almost immediately obvious from this graph is that the expenditure follows much the same increasing trend from 1997 to 2014, even through the recession itself. The increased deficit post-2008 is caused by the drop in receipts. In other words, government spending did not cause the deficit to grow; it was caused by a drop in income.

One problem with looking at these graphs is that there are strong quarter-to-quarter variations. These are mainly from receipts, and in particular from timings of when corporation tax is paid. To avoid being distracted by these details it's better to look at the annual graphs:

We can clearly see that receipts dropped well below expenditure from 2009 onwards and that expenditure increased every year, although only slightly between 2012 and 2013. It's also apparent that the reduced deficit in 2013 is mostly explained by receipts rising. To understand why, it's instructive to break the receipts down into three groups "Taxes on production", which includes VAT and corporation tax; "Taxes on income and wealth", which includes income tax and capital gains tax; and "Other income", which includes National Insurance Contributions (NICs), interest received and dividends due on publicly owned assets.

Source: Office of National Statistics (ONS) Public Sector Finances August 2014 release. The data plotted are all on the "psa6b_1" tab of the data tables spreadsheet. The "Taxes on production" data are labelled NMBY; the "Taxes on income and wealth" data are labelled NMCU and "Other income" is the sum of columns labelled LIQR, AIIH, LIQP, LIQQ. All three of these groups of data total to give "Total receipts" labelled ANBV, as plotted in previous graphs. Note that "Asset Purchase Facility" which is given in the column labelled L6BD is included via the LIQP "Interest and dividends" column. These data are not corrected for inflation.

We can see from this graph that receipts fell in all three groups between 2008 and 2009, but particularly taxes from production, income and wealth. This can be plausibly explained by the immediate effects of the recession. Unemployment rose, which meant less from income tax and NICs; consumers spent less, so VAT income decreased; and depressed spending throughout the economy meant companies made lower profits which resulted in lower corporation tax income.

From 2009 onwards we can see the recovery is most evident in the "Taxes on production", but weaker in terms of "Taxes on income and wealth", which has hardly changed between 2011 and 2013. It's worth noting that the increase in Receipts from 2012 to 2013 is mostly due to "Other income" which jumped up by about £20 billion from 2012 to 2013 - roughly the same increase as over the previous four years combined. Most of the this increase was due to £18.6 billion of income that appears for 2013 under a column called "Asset Purchase Facility" (APF) - see spreadsheet tab psa6b_1 column labelled L6BD. This was zero for all previous years, and is worth a bit of discussion.

Self interest, Quantitative Easing and the APF

From March 2009, the Bank of England has been creating money and using it to buy gilts, which are UK government bonds, i.e. shares in UK public sector debt. This process is known as Quantitative Easing (QE). The result of this is that as of 2014Q2 the Bank of England, or rather its wholly owned subsidiary, the Bank of England Asset Facility Fund (BEAPFF or APF for short), has spent £375 billion buying UK public sector debt (see spreadsheet tab psa9 column labelled MDE2). And, since the Bank of England is part of the UK pubic sector, it is correct to say that the UK public sector now owns about £326 billion of UK public sector debt (see spreadsheet tab psa9 column labelled MEX2).

This is clearly a bizarre situation. But it gets more bizarre. In return for holding gilts, the UK government (specifically HM Treasury) will pay interest (often referred to as coupons). This means that the Bank of England's APF has received tens of billions of pounds of payments. The intention is that one day, QE will be reversed and the gilts will be sold, and for that reason they are not counted as an asset of the UK public sector, but a temporary effect of dealing with the recession. If they were counted as assets, we would have to lower the UK public sector debt by about £326 billion, but this is expressly forbidden by the Maastricht Treaty because it amounts to the UK paying off its debt by creating money (see Bank of England Handbook 5). Until 2013, it was also true that the interest payments on these gilts that the APF received were excluded from the Public Sector Finance reports from the ONS. However, it was decided that the interest payments were a permanent effect of QE, and so the APF started transferring interest payments it accumulated to HM Treasury. So from 2013 onwards this flow of money was included in deficit and Public Sector Net Borrowing (PSNB ex) figures from the ONS.

That said, it seems that not all of the payments were included, and that the actual total transferred from the APF to HM Treasury in 2013 was around £40 billion - see spreadsheet tab psa9 column labelled MT6A). The existence of a limit of cash transfers included in PSNB ex is mentioned in this ONS document, but not the reason for such a limit.

The result of including the transfer caused by the build up of gilt interest payments boosted UK public sector income by £18.6 billion, which in turn decreased the deficit by this same amount. This means that about 60% of the £30 billion deficit reduction, from £94.9 billion in 2012 to £64.5 billion in 2013, was due to the new accounting for these interest payments. In other words, more than half of the deficit reduction was not due to spending cuts, nor was it due to a rise in income because the economy had recovered and generated tax receipts.

Deficit, Borrowing and Debt

These days the headline figures published by the ONS are not for the deficit, but for Public Sector Net Borrowing, specifically PSNB ex, and it's those figures that are most often quoted in media reports. I'll explore the relationship between the deficit and PSNB ex and PSND ex (i.e. debt) in my next post and tackle an important question I touched on above when discussing QE: why does the UK government borrow to cover the deficit when it can just create the money?

Political comment

It is astonishing that the UK public sector holds around 30% of its own debt, and that the interest it pays itself for doing so can have such a large impact on the budget - responsible for 60% of the deficit reduction in 2013. But, to me at least, the rationale is sound for reclassifying the monies accumulated from gilt interest payments in this way. It's reasonable to ask why they weren't included in Public Sector Finance reporting from the outset of QE. The answer is that no one expected that as much as £375 billion would be spent on buying gilts, and so the interest payments were not expected to be so significant. When it became clear that they were significant, it was correct for the ONS to include them. I'd therefore say that this is not an example of political trickery, but a case of accounting conventions that became awkward and in need of revision due to unusual economic circumstances.

However, any claim that's made by the current coalition government that it's reducing the deficit should be viewed with profound scepticism. Furthermore, the reason that expenditure continued to climb through the recession is that there was no sensible way to cut it. In fact there were quite legitimate reasons for it increasing: as unemployment rose, the number of people entitled to state benefits rose too. This was not due to shirkers or benefit cheats, as has sometimes been portrayed, but because around 900,000 people who had been working and contributing to the growing economy became unemployed during the first year of the recession. Likewise, health spending could not be cut without hurting those in need.

As you can see from the graphs above, the austerity in the form of spending cuts planned by the coalition government have done no more than to slow the growth in expenditure. and most of the claimed deficit reduction has come from a distracting accounting reclassification. I'm not going to get into the arguments for a government spending its way out of recession in this blog post, and nothing in this blog post depends on such an argument (though I have written about that point before). For now, I'll just point out that the coalition government has not even been able to meet its own targets of reducing the deficit.

Let me conclude by letting Prime Minister David Cameron's words speak for themselves. In January 2013, he said the government was "paying down Britain's debts". At no point has the debt decreased under the coalition government, and so either he was deliberately trying to mislead the public, or he does not understand the difference between 'debt' and 'deficit'. That may be an understandable mistake for an ordinary person, but it's nothing but shocking from the Prime Minister.

UK in Recession part 1 - GDP and debt

2014-10-14T14:02:00+01:00

Poor analogies, partial information, misunderstandings, oversimplifications and outright lies seem to be too common in any political discussion. I'm not just talking about partisan politicians and the media, but also discussion in online social media and in real life. Such misconceptions are especially troubling concerning the great recession of 2008/9 and its aftermath in the UK. If we do not understand what has happened, there is little chance we can improve matters and prevent it happening again. So I decided to investigate the data for myself and write a series of blog posts about it. The big questions I hope to answer are: what effects have government actions already had, and what future governments can do to create a true economic recovery, by which I mean increase well-being, employment and wealth, whilst reducing inequality.

I've pulled all the data directly from the published government statistics and accounts and spent much time making sure I understand the accompanying documentation. I'll give precise links to all this information and you can also check the spreadsheet I've constructed to produce all the graphs in this post: download ods or download xls.

As far as possible I've confined political comment to the final section of each post, so that all other sections describe the data itself. The data used in this post is the quarterly time series from 1997 to 2014Q2, which is the full dataset available at the time of writing.

UK GDP

GDP or Gross Domestic Product is the total value added in producing goods and services produced in the economy. For example, for a company making steel girders, its contribution to GDP would be the amount made from selling the girders less the amount paid for the raw steel. Notice that this contribution to GDP will therefore include the company's profit, what it pays its employees and what it pays in tax and rent.

Here is the graph of GDP.

Source: Office of National Statistics (ONS) National Accounts Q2 2014. The GDP CVM data are the seasonally adjusted, chain volume measure (SA CVM) labelled as ABMI in the "A2 Aggregates" tab of the data tables spreadsheet. The GDP CP data are labelled YBHA on the same tab of that spreadsheet.

CP stands for current prices and refers to the fact that the values are not adjusted for inflation. The CVM is adjusted for inflation and expressed in terms of the value of prices in 2011. The GDP CP curve is steeper because it is showing the effect of both inflation and growth together. Most quoted GDP figures are CVM.

You can see that GDP increases at a steady rate until 2008 when it dipped sharply. After 2009 it began to grow again. The CVM pre-recession peak value of 2008Q1 was reached again in 2013Q3.

I'll calculate the growth rate of GDP as follows:

growth rate for quarter = (this quarter's GDP - last quarter's GDP) / this quarter's GDP

Note that this a quarter-to-quarter growth rate. It's more common to see annual growth rates quoted - these will be larger than quarterly ones. Here is the graph of GDP CVM growth.

This graph shows that the growth rate is always positive (i.e. the economy grew) until 2008Q2 when it went negative for five quarters in a row. Two consecutive quarters or more of negative growth (i.e. decreasing GDP) is commonly taken as being the definition of a recession. GDP growth returned to being positive in 2009Q3. From 2011Q4 to 2012Q4, the growth was mostly close to zero or negative, which prompted people to speculate the economy had entered recession again making for a so-called double-dip recession. Technically it isn't according to this data because there aren't two consecutive quarters with negative growth in 2011/12. But, to me, that's beside the point - the recovery was weak and faltering. To put some numbers on it, pre-recession, from 1997Q1 to 2008Q1, GDP growth averaged 0.75% from one quarter to the next. From 2009Q3 to 2014Q2 it averaged 0.44%.

In fact, the 0.44% is generous because it ignores the five quarters of the recession itself. If we include them, the quarter-to-quarter growth (i.e. from 2008Q2 onwards) averages to 0.1%. If growth had carried on as it had done from 1997 to 2008 for another six years, then we should have seen GDP at around £480 billion in 2014Q1, but in reality it was £420 billion. So although we can say that GDP growth has recovered to a degree, the effect of the recession is to leave GDP £60 billion below where we could have expected it. Whether that expectation is reasonable is a separate question.

The Public Sector

GDP is a measure that applies to the whole economy, but the next couple of sections address the debt and borrowing of the public sector, so it's necessary to define what is meant by that.

The UK public sector includes the central government, local government, the Bank of England and all organisations in public ownership. This not only includes constructing and running facilities such as hospitals and schools, but also the employment of doctors, nurses, teachers and other admin staff. The public sector also includes formerly private business such as banks and railway operators brought into public ownership for a variety of reasons, but mainly because they were failing in private ownership. Network rail was the most recent private company to be taken into public ownership.

I will use the "ex" figures of public debt and borrowing, that is ones that exclude "temporary" effects due to special measures dealing with the recession. In particular, this excludes assets associated with Quantitative Easing (QE) and with banks taken into public ownership. There is some sense in doing this because it is expected that QE will be reversed at some point and the banks will be returned to private ownership. Using the ex figures also allows a more meaningful comparison with the economy prior to the recession. I'll deal with the non-ex values of debt and borrowing in a future blog post.

Public Sector Debt

Here is the graph of the UK's debt.

Source: Office of National Statistics (ONS) Public Sector Finances August 2014 release. The data plotted are "Public Sector Net Debt excluding public sector banks" (PSND ex) labelled as HF6W in the "psa1" tab of the data tables spreadsheet. The debt value is in "current prices" and so is not adjusted for inflation. Also plotted is the debt as a percentage of GDP CP. This is more informative because, for example, a £1 billion debt is a much more pressing concern for an economy with GDP £1 billion than it is for an economy with GDP £10 billion. As a convenient side-effect, expressing debt as a percentage of GDP CP also corrects for the effect of inflation.

The debt/GDP falls from 1997 until 2001, then increases slightly until 2008 after which it rises much more rapidly. Clearly, the majority of the debt was added after the recession hit in 2008.

Public Sector Borrowing

Each time an amount is borrowed, the debt will rise by that amount. In this sense you can think of debt as being the accumulation of all past borrowing. Conversely, you can think of borrowing as being the growth rate of debt. Here's the graph of borrowing for each quarter.

Source: Office of National Statistics (ONS) Public Sector Finances August 2014 release. The data plotted are "Public Sector Net Borrowing excluding public sector banks" (PSNB ex) labelled as J5II in the "psa1" tab of the data tables spreadsheet. Note that these data are not corrected for inflation.

This graph shows that borrowing in a given year tends to be larger in Q2 and Q4 than it is in Q1 and Q3. To smooth such seasonal variations and bring out the trend a four quarter running mean is plotted. The running mean data point for 2012Q2 is the average of the values of the four values 2012Q1, 2012Q2, 2012Q3, 2012Q4, and the data point for 2012Q3 is the average of 2012Q2, 2012Q3, 2012Q4, 2013Q1 and so on.

Bearing in mind that borrowing is the growth rate of debt, the longer term trends are just the ones we saw before: borrowing goes negative (money is paid back, debt falls), then in 2001 borrowing becomes positive (debt rises) and then from 2008 borrowing is significantly larger (debt rises faster). This graph shows that borrowing starts falling in 2009 - something which isn't clear from the debt graph.

If you look closely at the data you might see that the increase in debt each quarter is not equal to the borrowing for that quarter. This isn't a flaw in my graphs, nor is it in an error in the data from the ONS. It is quite real. If you want to investigate this further you can start your journey of discovery on page 10 of the Public Sector Finances, August 2014 document. I'll come back to this in a future blog post.

Political comment

The UK had a Labour government from 1997 until mid-2010 when a coalition of Conservatives and Liberal Democrats came to power. There are two points I'd like to address that are relevant to these governments: the recovery in the year or so after the main recession dip, and the timing of the debt build-up.

Gordon Brown was the UK chancellor from 1997 until 2007, and prime minister from 2007 to 2010 during which time Alistair Darling was chancellor. Brown and Darling were credited by world leaders and respected economists for acting decisively as soon as the financial crisis hit in 2007/8 and limiting the worst of the damage, setting an example that others followed. Their actions were even endorsed at the time by Conservative George Osborne who became chancellor in 2010. If you look at the GDP growth graph above you will see that the economy recovered and had five consecutive quarters of respectable growth from 2009 into 2010. The growth faltered after the first quarter in which the coalition government was in power. It's not yet clear to me whether the new government did anything to disrupt the recovery, or if it was a coincidence of timing - that's a question for another blog post.

Many people believe that the Labour goverment was responsible for leaving the UK with a large debt before the recession because it spent too freely. This is the usual story told by the Conservatives, such as prime minister David Cameron, who came into coalition government from 2010. But it is clear from both the debt and borrowing graphs above that the first part of the claim is not true: the rise in debt was a consequence of the recession. I've even heard it said the Labour's overspending caused the recession. Not only does this put effect before cause, but a claim that a global recession could be caused by a UK government's spending policy is ridiculous. In fact, the debt graph shows that Labour had slightly reduced the debt/GDP percentage between 1997 and 2007.

I know many threads are left dangling in this blog post, but I'll tie a few of them up in the next one which looks at the UK budget deficit and the associated public sector spending and income.

Mid-life neuroplasticity

2014-10-06T21:56:00+01:00

Eyebrows. They need trimming? Why did the hair dresser ask me that, and why never before? I was told the answer at the Troon real ale festival about a week later. The following morning I'm fixing an amplifier that has increased the sounds levels of music for me for over thirty years. It has blown a slow-blow fuse - that's one very slow blow. The amp is older than the people I was drinking with the evening before. When I open it up it I discover I have become part of it - a thick layer of dust coats its circuit board, fat capacitors and the impressive heat sink. The volume control causes a crackle when I use it because my skin has got under its contacts.

They were a lovely bunch of people - bouncy, lively, intelligent, well-educated and irreverent. I ended up in their company thanks to a good friend; he is older than them but younger than me and described hanging out with them as liberating. It certainly is. It didn't make me feel old, just different. I didn't mind when one of them mentioned a swimmer who was competitionally "past it" at 13 years my junior. To a student-days' story about a pub whose name I couldn't recall, one laughed and said "Dunno, I was six then." I was surprised to be told the eyebrows thing was age-related, but was comfortable accepting it. I doth protest too little you think. Yes, but genuinely so. If anything I felt pleasantly surprised to see my reflection in these people.

I feel a little like I did as a teenager. My mind feels like it's opening up, and certain things are unsettling me and getting under my skin, but at the same time the world evokes more visceral feelings in me, and music punches deeper into my heart than it has in a good while. In short, I feel more alive.

I've recently moved to a new home. But has that caused this change in feelings? Or perhaps it was a symptom of it? Neither feels right. The referendum on Scottish independence certainly unsettled me, but it was not the sole cause of how I now feel; my reaction to it highlighted the symptoms of a change in the receptive state of my mind. The most I can say is that something has left me feeling more affected by, and more open to, external influences. The shake in society caused by the referendum would not have affected me so much if I was the person I was five or ten years ago. I feel sure that something has changed in me. Not necessarily for the worse. Feeling more alive is always better, isn't it?

Perhaps I'm having a mid-life crisis, but I prefer to think of it as an experience in mid-life neuroplasticity? I feel a stronger urge to express myself and make some impact, however small (though preferably large), on society around me. Being passive isn't an option. I want to do something for others, and give them the chance to do better, and be better than me. Making myself heard is important now - it wasn't before.

There's a tiny bit of me that wants the simplicity of life in a rut. A regular job, with regular hours, with a boss who gets paid more than me, but who shoulders responsibility that I don't want, and who I can rail against when excess work or blame gets loaded on to me. I could be more like others and play (or just talk about) football or golf, or be interested in cars. I could declare for a particular political party. Supporting the Yes vote for independence seemed like great fun, so much so that the SNP and Green parties have had a surge in membership. Or I could sign up for the Labour party - they're less fun, but seem to understand the economy better, even if they feel unable to stand up for what they're supposed to believe in.

But I'm as likely to join the Conservative party as to do anything of those things. Or, in other words, there's no chance of any of that happening. It all seems repellent and ridiculous to me. All reeking of lazy crowd-think. I'm conscious I may have insulted a number of people there. To them I say that's not my intention (but nevertheless, go fuck yourself). I'm not saying anyone is wrong, just saying what makes sense to me.

The path ahead of me doesn't seem easy, lacks any obvious destination, with few tangible rewards along the way. But it will be a path I beat out for myself, largely invisible to others, with many compromised twists and turns to accommodate close family and friends who barely comprehend what I'm trying to say. I don't feel I have much option but to express my free will so as to ensure the inevitable will happen.

But Dad, you might be wrong

2014-09-30T15:47:00+01:00

"But Dad, you might be wrong."

I couldn't disagree with my nine year-old son, though I said I'd given the matter much thought. We were walking down the hill from the polling station where I had just cast my vote in the Scottish independence referendum. Two weeks before that, sat on the ferry from the Isle of Arran, I had asked him about his thoughts on the referendum, and he said either choice would be fine. That pleased me. He knew both his parents would vote No yet he felt comfortable to think for himself and speak his mind.

Now he was asking me to explain why we were voting, and why Scotland didn't just do the "right thing". My response was that some things, such as the result of adding 2 and 2, do have a right answer (and infinitely many wrong ones), but no amount of voting for 5 will make it the correct answer. Insisting it is 5 will lead to a contradiction which, by the principle of explosion, will destroy all of arithmetic (formally known as number theory). Other things, such as how a country should be governed, do not have a right or wrong answer in that sense, so we resort to democracy and voting.

The act of explaining a long-held belief to someone with an honest and open mind often makes you realise that it's actually not at as obvious as you first thought. To my son's challenge, I could not be sure that No was the right option on the ballot paper, yet some part of my mind did regard it as being correct. Certainly, if an ardent Yes campaigner had challenged me, I would not have given any hint of the answer I'd just given my son, specifically, that I might be wrong. Instead I would've gone into hyper-rational certainty mode and argued a number of points to make the case. And I'd be caught in contradiction: I cannot be rational and certain at the same time, because I can easily rationalise my uncertainty.

When I heard the referendum result I breathed a sigh of relief. Not only because the vote went the way I wanted, but because people - including me - could now calm down and return to being more rational. I can admit my uncertainty again and think more critically. Yet I feel uneasy. Not with the result, but that for a good while I worried that emotion and irrational optimism seemed to be trampling over rational argument, even in my own head.

I would like to reach out to those who voted Yes and now feel deeply disappointed that the bright future they saw has been denied to them. We may disagree on whether that vision of the future was real, but I do not doubt the reality of your feelings, and so I do feel for you.

And to all who voted, whether Yes or No, perhaps we should let the fog of emotion clear and see what new ideas emerge, particularly ones which we might share.

Voxelands

2014-09-25T10:43:00+01:00

Voxelands is a fork of Minetest which itself is a Free and Open Source (FOSS) reworking of the more famous Minecraft. I enjoyed Minecraft's immersive blocky world and there were lots of interesting mods to enhance and extend the game, but I found the process of installing them was far too troublesome. I even had a go at writing mods, but that was even more arduous. Part of the reason for this is Minecraft is closed source and wasn't originally designed for modding.

Then along rolled Minetest which was FOSS (GNU LGPL) and had a much better modding engine powered by a scripting language called Lua (Portuguese for Moon). My first experience playing Minetest left me with the impression of a relatively bare and empty world compared to Minecraft, but I soon realised that was because you really have to add in a few mods, such as one for farming or one with various creatures (known as mobs). Even better, I wrote my own mod which provided for a sort of magic tunneling machine. This may sound great, but in all honesty, I just wasn't enjoying the game. The reason was that Minetest lacked Minecraft's well thought-out and balanced gameplay that's squarely directed at one thing: letting the player have fun in doing whatever they want to do. That said, I remain open to the suggestion that I've not yet found the right combo of Minetest mods.

I wasn't alone in thinking this and some of Minetest's developers split off a fork of the game called Minetest-classic, which later got the much better name Voxelands. It dropped the modding engine of Minetest in favour of creating a game that is more fun. It's still got a way to go to reach Minecraft's maturity - most noticeably, Mobs aren't much fun yet. I understand these kind of issues will be addressed soon and there is certainly active development at present.

I also appreciate that it's trying to be its own game and not just copy Minecraft. One thing that irked me about Minecraft is that I had to keep leaving the game to search the wikis to learn the recipes (or else invest in a lot of guess-work-driven discovery that I simply don't have time for). But in Voxelands that's become part of the game: you arrange a few sticks into a crude question mark shape on the 3 by 3 crafting grid and voila, you've got a book with all the recipes. "Spoilers!" you may say, but then just don't look at the book.

After I compiled and installed Voxelands (and you don't have to, as pre-built downloads are available, including one for windows), I fired up a server on my laptop and posted a message online for folk to come and join me. The first person that did was Darkrose, the lead developer. I thought that was really nice of her to visit.

So, my thanks to Darkrose and others for creating it, and I recommend giving Voxelands a try - it's fun!

Warning: What follows is some technical stuff which you can safely ignore unless you're a bit geeky like me.

Compiling on Slackware

This was pretty straightforward, though I did hit a snag which I found the solution to immediately in the Voxelands forum. I wrote a slackbuild script which was you can get at Slackbuilds.org.

Compiling a server-only build for Centos

I'm running an old eee PC 1000H as a home server with 32 bit Centos 6.5. First, I needed to install generic stuff needed to build from source on Centos (equivalent of build-essential on apt systems):

yum groupinstall 'Development Tools'

and also cmake

yum install cmake

then install stuff needed for Voxelands:

yum install zlib bzip2-devel sqlite-devel libpng-devel libjpeg-turbo-devel
irrlicht-devel libXxf86vm-devel

Determining the details in the above command took a bit of experimentation as the yum install command in the README.txt in the Voxelands source specified slightly different package names, e.g. libsqlite3x-devel instead of sqlite-devel. This could be due to differences between Fedora and Centos.

I then set up the build with this cmake command:

cmake . -DRUN_IN_PLACE=1 -DBUILD_CLIENT=0

And then started the build with just

make

but it failed to compile with this error related to type 'u64' (Note: This happened in build 1408, but did not in 1409.)

[ 20%] Building CXX object src/CMakeFiles/voxelands-server.dir/content_sao.cpp.o                                                                        
In file included from voxelands/src/serverobject.h:25,
             from voxelands/src/content_sao.h:23,
             from voxelands/src/content_sao.cpp:20:
voxelands/src/utility.h:48: error: reference to 'u64' is ambiguous
voxelands/src/common_irrlicht.h:72: error: candidates are: typedef uint64_t u64
/usr/include/irrlicht/irrTypes.h:100: error: typedef long long unsigned int irr::u64

I found online that this can be caused by compiling with an older version of Irrlicht. Indeed, on Centos I had 1.7.3 so I downloaded the 1.8.1 headers and source and tried compiling again, starting with

cmake . -DRUN_IN_PLACE=1 -DIRRLICHT_INCLUDE_DIR=../irrlicht-1.8/include -DBUILD_CLIENT=0

but I got exactly the same error. After a bit more investigation I realised that Irrlicht was defining u64 to be one type, but Voxelands was defining it to be something different. The above fix probably solves the problem on a 64 bit system, but not on my 32 bit system (the clue was WORDSIZE in irrTypes.h). To solve the problem, I just commented out this line in src/common_irrlicht.h in the Voxelands source code:

//typedef uint64_t u64;

and compilation worked fine.

The server ran fine, but I found I couldn't connect to it from other machines on the same network. I created an iptables rule to allow access to voxelands port, but that didn't solve the problem either. I started to look at SElinux, which always makes my head hurt, but even going into permissible mode didn't allow me to connect. At that point I realised that Voxelands, like Minetest, uses UDP not TCP. Doh! I deleted my previous firewall rule and added a new one with:

iptables -I INPUT 3 -p udp --dport 30000 -j ACCEPT

and I was able to connect!

Scottish independence polling - update

2014-09-17T15:32:00+01:00

I've updated the analysis I performed at the weekend to include the five polls released since then: ICM for The Sunday Telegraph; ICM for The Scotsman; Panelbase for The Sunday Times; Opinium for The Daily Telegraph; and Survation for The Daily Mail. All of these were online polls.

Here are the updated graphs for these new online polls:

And here are all the September polls. Note that the polls in each group are ordered in time, so that, for example, the rightmost 'Survation - online' poll is the most recent one.

Very little has changed, except there is now good agreement across polls. The obvious exception to this is the ICM online poll for The Sunday Telegraph which stands out as giving Yes a significant lead - a first in this campaign. However, it has an unusually small sample size compared to others and was conducted quite differently - see John Curtice's blog post for more on this poll where he also explains why it receives a weighting of 0.7 in his Poll of Polls. The other change is that there is evidence of a slight boost to the Yes vote, but at 1%, it's less than statistical sampling uncertainties involved.

The No lead (including Don't knows) averaged across the fourteen September polls (applying a 0.7 weighting to ICM Sunday Telegraph Poll) is 2.9pp. Is this statistically significant? The binomial distribution tells us that if we are looking at a sample of 1000 people where there's an equal chance of getting a Yes or a No, then we can expect that 95% of samples will fall in the range 47% to 53%. This is the basis of the +/-3pp uncertainty, and why it is stated as being for 95% confidence. If we take N samples of 1000 people and average the results, as I have done here, then the uncertainty is reduced by the square root of N. Therefore we can say the No lead is 2.9pp +/-0.8pp. If we restrict the average to the most recent poll for each pollster, and for each type (i.e. so we included two polls for Survation and ICM each, as they've recently done phone and online polls), then the No lead averaged over those 7 polls is 3.9pp+/-1.1pp.

Although these No leads are statistically significant, no account is taken of anything but statistical sampling uncertainty. Most crucially, there's no bias being accounted for here, other than whatever is being corrected for in the weightings of the various polls. In my previous post I devoted most attention to the Spiral of silence bias in the No direction, but here I want to see if there is any evidence of a bias in the Yes direction arising from failure to weight properly by social class. The more affluent class is ABC1 and poorer one is C2DE. To investigate this I've looked through the tables of 5 recent online and phone polls to see what biases might be left over from the weighting.

There's actually quite a lot of weighting performed in the polls, and mostly it has the effect of raising the Yes vote relative to the No vote. Generally, this is because more affluent people are responding to the polls (i.e. there is more of social class ABC1 than C2DE), but the effect of this seems to be properly corrected for in the weighting. The only exception to this was the Survation phone poll for which no weighting by social class was performed explicitly, but in that case it's plausible that heavier down-weighting in other categories, e.g. Conservative voters, at least partially addressed the problem.

The recurring feature that I noticed is that although the voting recall of the 2011 Scottish elections is correctly weighted by party, this is not true for the group of people that did not vote in this election. This is precisely the problem that John Curtice addressed in his blog post on the missing million. For the five recent polls, I've peformed a similar analysis. The table below shows the results.

End date	Pollster	Total	DNV	Y	N	DK	Reference
16/09/14	Survation - online	885	20%	61	93	23	Table 21, p24
16/09/14	ICM - online	964	22%	60	94	57	Table 3, p6
15/09/14	Opinium - online	1156	16%	67	87	36	Q3, p11
12/09/14	Survation - phone	893	24%	83	106	29	Table 4, p6
11/09/14	ICM - phone	803	15%	46	51	23	Table 3, p6

The Total column gives the total number of respondents that were included in producing the headline Yes, No, and DK figures, often reduced below the total number of respondents because of turnout weighting (i.e. don't include people who might not vote). DNV is the percentage of that total that said they did not vote in 2011 (unlike John Curtice I didn't include those who couldn't remember). Then Y, N, DK give the Yes, No and DK numbers (not percentages!) for referendum voting intention. Note that Y+N+DK divided by Total is equal to DNV. The Reference column tells you where to find these numbers in the PDF downloads from the pollster's website. I did try to include Panelbase's most recent poll in this study, but the pages in its PDF were truncated so data was missing for the DNVs.

What's immediately striking is that whilst 50% of people did not vote in the 2011 election, this table tells us that only between 15% and 24% of poll respondents say they did not vote. It should be noted that DNV is the value after weighting - the unweighted values were significantly lower in some cases.

The second feature is that the No vote is consistently stronger amongst the DNVs in all cases, being about 3:2 in favour of No in Survation and ICM's online polls. The two phone polls show smaller differences though. So, as John Curtice found in previous polls, people who did not vote in 2011 seem more likely to vote No. If we were to weight poll results so that DNV was 50% then the No lead would be increased by a pp or more for all but the ICM - phone poll.

The 'Missing million' issue was originally phrased as being due to less well off voters, but who are the DNVs? To see if the DNVs reflected any particular class, I looked closely at the results for Survation's Daily Mail poll. Its DNVs were 2:1 female, less likely to be 65+, not obviously from any particular part of Scotland, and almost as likely to be in social group ABC1 as in C2DE. In short, these DNVs with a preference for No seem to be mostly woman under 65.

Of course, if there's a class of people that are completely hidden from the polls, then there's a bias we won't know about until the result is announced on Friday morning.

Scottish independence polling

2014-09-13T17:00:00+01:00

Poll companies are finding the Scottish referendum to be a real challenge. It's unlike any general election: there are only two options (yes and no) and the turnout is expected to be high, perhaps 90%, compared to 50% for the 2011 Scottish elections and 65% for the UK elections in 2010.

My previous posts explain why I will be voting against independence; this post is about my analysis of the polls. Can I be trusted not to produce a biased analysis? That's for you to judge, all I can say is that I want to be as honest with the data as possible and that if any bias does slip through it will be subconscious.

A few notes on conventions. I use % to mean an actual percentage and pp if I mean a change in a percentage (percentage points). For example, I might say that "the undecided voters went from 23% to 20%, a drop of 3 pp". All percentages I quote will be poll results for Yes, No and DK (Don't know). In some circumstances it's better to exclude Don't knows, but as they're an interesting part of interpreting the data, I prefer to keep them separate.

The data I'm working with was from a CSV download from the whatscotlandthinks.org site on the evening of Saturday 13 September 2014. I have also examined the published tables of all pollsters directly, which are available as PDFs from their websites. Only polls with tables published at time of writing are considered in this post. All polls I refer to were conducted according to the guidelines of the British Polling Council and had approximately 1000 respondents (ranging from 927 to 1268).

The graphs below show the time series evolution from the start of June until 12th Sept. On the left are the time series of the Yes votes for each company (click a graph to enlarge) and on the right are the time series of the No votes. Don't knows are "included", in the sense that the Yes vote for a given poll plus its corresponding No vote will add to less than 100%, the remainder being the Don't knows. The top two graphs are for online polls and the lower two are for the others. I've separated them because, as I'll discuss below, their approaches to polling are fundamentally different.

Looking at the left graphs, you can see that two pollsters - Survation and Panelbase - put the Yes vote consistently higher than the others - YouGov, TNS-BMRB, Ipsos Mori and ICM. To put some numbers on it, for the months June, July and August 2014, Survation did five polls showing a No lead at 5-6pp (5 to 6 percentage points) and Panelbase's three polls put it at 3-7pp. In contrast, YouGov did four polls in these months and put the No lead at 13-20pp; TNS-BMRB did three polls putting it at 9-14pp; ICM's four polls put it at 7-14pp; and Ipsos Mori's two polls put the No lead at 14pp and 18pp.

This didn't go unnoticed, in fact the president of YouGov Peter Kellner wrote about it on YouGov's blog and in the Times newspaper, and even commissioned a poll to try and show why Survation's method was flawed. Survation wrote this rebuttal. Given that I wanted to believe a high Yes vote was indeed wrong, I hoped to find Kellner's case was the more convincing. I didn't. Survation's response was well-argued, and Kellner's argument relied on using unpublished, proprietary data used to perform weighting, so I couldn't check his findings. Also, Kellner's criticism of a rival pollster in this fashion was quite unprecedented. Bottom line: it remained a mystery why Survation and Panelbase kept putting the Yes vote high. See John Curtice's post for more details on this story.

Panelbase and Survation polls in June to August were all conducted online, but this doesn't offer an obvious explanation because so were the YouGov and ICM polls. Ipsos Mori used the telephone and TNS-BMRB interviewed people in-person in their homes. It's certainly the case that the ordering and wording of questions can influence a poll's result, and that criticism has been levelled at Panelbase, but not as far as I am aware at Survation.

In early September 2014 there was a change. The top left graph shows that YouGov's Yes vote had risen. Soon after TNS-BMRB released a poll also showing a rise in the Yes vote. But, the disagreement between the pollsters persisted - both Panelbase and Survation polls showed no change from before, i.e. No leads of 4pp and 5pp respectively. Also, a look at the No vote graphs (right) show no decline in the No vote, other than for YouGov, which also showed the biggest rise in Yes. This leads to me to think that there was a real increase in the reported Yes vote, but for some reason this was exaggerated in the YouGov polls.

An acknowledged problem with online polls is that they have to draw results from a recruited 'panel'. These are people who agree to take part in future polling, usually in return for a small payment each time. As you might expect this will lead to a bias in demographics, e.g. biasing against folk who do not like to use the internet (older folk), and also biasing towards people who like to take part in polls (political activists). Some of these biases can be corrected by keeping detailed profiles of panel members and by weighting poll results to known population data, e.g. ONS census data. For example, if a poll finds there to be 400 women and 600 men responding, then the answers given by each woman will be weighted up. The best online polls are those where little weighting is required, but even then there are almost certainly other factors that cannot be weighted away, e.g. that people motivated to join online panels are probably more likely to have firmly held views.

One explanation of the disagreement across polls is that Panelbase and Survation's panel was from the outset a population with a disproportionate number of people who had made up their minds on how they intended to vote. So as we near the day of the actual referendum what we see is that their more settled panels produce polls with unchanging results. If so, the other polls appear more volatile because they contain people who hadn't thought about it much until recently and who started the process of making up their minds in early Sept, and moving between Yes-Don't know-No, with a preference for the Yes direction. Two non-online polls are consistent with this explanation, showing high levels of Don't knows - TNS-BMRB at 23% and ICM at 17%.

But this explanation is not satisfactory by itself. For one thing, why does YouGov, an online poll, not have such a settled panel too? Some volatility may be attributed to the fact that The Sun and The Times have been commissioning YouGov polls much more frequently of late: three polls from June to Mid-August, then four in as many weeks thereafter.

Let's turn our attention from the time series and look only at September polls, shown in the graph below. Online polls are all on the right so we can more easily compare like with like. All but one poll shows a lead for No, and the lead varies from 1-7pp. It's clear that the online polls are all now in good agreement, certainly to within statistical uncertainties. A poll's percentage has a 95% probability of being correct to within +/-3% based on the statistics of sampling; though this, of course, doesn't address other uncertainties and biases.

Martin Boon, director of ICM noted on BBC Radio 4's More or Less (12th Sept, around 6 minutes in - download here) that some people are shy to give answers to certain questions, even in the apparent privacy of a poll interview. This effect is known as the 'spiral of silence'. He suggested that there is shyness to say No because it might feel "unpatriotic". For UK general elections after 1992, when ICM first discovered the effect, they would attempt to correct it by returning some Don't knows back to what they voted for in the previous election. This cannot be done for the Scottish referendum because it has no precedent.

So, is there any evidence for the spiral of silence here? TNS-BMRB and ICM stand apart from the online polls in that their No leads are smaller - 1pp and 2pp. Both show a high level of Don't Knows compared to the online polls. For ICM the Don't Knows have increased since their previous poll, 14% up to 17%, and for TNS-BMRB the Don't knows stayed the same at 23%. The deficit of No voters and excess Don't knows compared to the online polls is consistent with the spiral of silence. But consistency is not proof. Arguing against the spiral of silence effect, is the fact that the Yes vote is also slightly lower than all the online polls. So if there is a spiral of silence in this data, it's mixed in with a sizeable minority of voters who are genuinely undecided.

For the spiral of silence to operate, there needs to be a tendency for a class of voters to be less willing to express themselves. My firsthand experience in Glasgow and Stirling during the first half of September is that people will put up Yes stickers and posters and wear badges far more frequently than "No, thanks" ones. This is true even in affluent areas where the polls tell us there should be a greater No vote. I'd estimate the ratio of "Yes" posters, stickers and badges to "No, thanks" ones as being at least 5 to 1. I understand from others (I don't use mainstream social media) that Yes voters are also more vocal on Twitter and Facebook, and they will display Yes badges on their profiles much more frequently than No voters do. It seems clear that the circumstances are right for the spiral of silence to be in effect, depressing the No vote in polls. But again, this is not proof that it's present, merely an argument that it's plausible.

The TNS-BMRB and ICM non-online polls were followed by the Survation phone poll of 10-12 September which gave results consistent with the online polls. This is probably because Survation asked respondents how they would vote now, whereas TNS-BMRB and ICM asked how the respondent would vote on the 18th, the latter leading to more Don't knows.

There's also direct evidence in recent polls that many people have made up their minds. For example, Opinium's poll tells us that over 90% of people saying 'Yes' or 'No' are highly unlikely to change their mind.

There's also the question of what the 50% who didn't vote in the last Holyrood election in 2011 will do in the referendum. Many polls on the referendum ask people how they voted in the 2011 election, and this was examined by John Curtice's blog post on the so-called "missing million"; his conclusion was that they were more likely to vote No. The last TNS-BMRB poll weakly agrees with this too: second questioning of "Don't know"s yielded a preference in favour of No of about 4:3, though the majority of Don't knows said 'Don't know' even at that second question.

Taking all the above into consideration I'm inclined to believe that the No lead is small but real and if anything underestimated. It's still possible for Yes to win, but I think it unlikely.

EDIT 15 Sept 2014: after I wrote this post, I found this piece by Stephen Fisher which makes many of the same points, and also looks back at other comparable votes.

Steam, linux and wine

2014-09-11T09:23:00+01:00

This isn't about my worshiping of Bacchus whilst using linux in a sauna, but instead about getting a windows-only steam game to work on a recent 64 bit linux distro. I'm using slackware, but I suspect the pitfalls and solutions I encountered would be similar on other distros.

This article was originally written on 1 August 2014, and an audio version was released on Hacker Public Radio on 11 September 2014. Be warned, it gets a little technical!

My 9 year old son loves the game Ace of Spades (AoS). His computer runs slackware linux 14.1 with KDE and he uses it for you tube, games, writing documents with libre office and listening to music, and he's quite comfortable with it. But, since Ace of Spades is only available for windows, I kept his computer dual booting with Windows XP. On the boot menu that selects between operating systems the label reads "FailXP", and without any irony, that's what he calls it. I don't usually approve of such childish name-calling, but my excuse here is that my son is actually a child. Anyway, post-April 2014, when XP finally stopped being supported by Microsoft, I said no more FailXP. My son accepted my reasons, but we agreed that I'd find a way to let him play Ace of Spades again.

Despite AoS being exclusive to steam, and steam going all linuxy of late, there's no linux version of Ace of Spades (boo to you Jagex, the developer company), and I didn't want to have to install Fail7, sorry, Windows 7, and definitely not 8, so my remaining option was to get steam and so AoS working in wine. A look at the steam page on wineHQ encouraged me.

My first step was to install wine which allows windows programs to run in linux (in a way that's perfect legal). It was straightforward enough. I just downloaded a slackware package provided by slackware guru Alien Bob and installed it. Non-slackers can do yum or apt-get install wine and such like, or use a fancy-pants GUI installer. The computer's running 64 bit Slackware, but I'd previously added multilib capabilities (thanks to that generous Alien Bob chap), i.e. installed essential 32 bit libraries in the system so 32 bit applications could be run. Now, here's a slightly confusing point. I installed wine compiled for a 64 bit system, but wine itself will run 32 bit windows binaries, which is what I want because Steam is only available in 32 bit. This fact is worth bearing in mind, as we'll see.

Next I installed steam using the "Download" link given in the aforementioned wineHQ page, and then, as my normal user, I ran

wine msiexec /i SteamInstall.msi

and steam installed just fine after a few clicks on some ugly dialog windows. I then found the executable hiding away in my home dir: .wine/drive_C/Program Files/Steam/Steam.exe. I cd'd to that dir and then did

wine Steam.exe

and it started up, but there was no text in the pretty window that popped up. Hmmm, a quick scan of the wineHQ page led me to the solution:

wine Steam.exe -no-dwrite

and now I could see the text, but it seemed to know that my user name was mcnalu. Eek! How did it know that? The only thing I could think of is that steam recognised the mac address of my network card. Creepy. Anyway, the steam client seemed to work just fine. I then tried to launch of Ace of Spades. It seemed to install it OK, but any attempt to run it ended in failure with no GUI message at all and the most generic of inscrutable error messages in the terminal where I'd launched it.

After a bit of reading around I came across winetricks and decided that I should 've used that to install steam. It's actually just a perl script that automates various irritating wine configuration tasks, and it has a special steam option. Before doing this, I ran the SteamInstall.msi using the same command as above, but this time clicked the button to remove the steam installation. Next, I downloaded and installed winetricks (in my case using sbopkg, but your linux's package manager might have it, and if not, it's just a script which you can download and run anywhere you like). Next, as a regular user, I did

winetricks steam

and it downloaded a steam installer but then stopped and complained that the sha1sum didn't match. A quick look at the winetricks script revealed that an sha1sum from March 2011 is hardcoded into it (very odd, IMHO), so I used the sha1sum command to generate the sha1sum of the installer I'd downloaded and replaced the sha1sum in script with that (a bit naughty). Then winetricks steam worked perfectly. To my surprise the installer was not the same as the one I had run previously. Amongst other things, it had different dialog windows. But this encouraged me - perhaps the previous installer was old or perhaps misconfigured.

Once steam was installed I had to log in (no creepy auto-username detection this time!) and, as before, the client worked well. I then tried to launch AoS and... it didn't work. I was slightly heartened to see the terminal window was full of much more verbose errors than before, and after a few minutes poring over it, I decided the problem was that steam wanted access to the audio library OpenAL and wine was trying to oblige but couldn't find libopenal.so. Strangely, OpenAL was actually installed, but only the 64 bit version, so I guessed wine was after the 32 bit version so it could fool AoS into using it disguised as a windows DLL.

OpenAL is not part of stock slackware, and so libopenal does not come with multilib 32 bit stuff I had previously installed. It wasn't hard to create the necessary package though. I downloaded the 32 bit OpenAL package from AlienBob's site and, as root, ran

convertpkg-compat32 -i OpenAL-1.15.1-i486-2alien.tgz

and this generated the package OpenAL-compat32-1.15.1-x86_64-2aliencompat32.txz in the /tmp directory. I then did

installpkg /tmp/OpenAL-compat32-1.15.1-x86_64-2aliencompat32.txz

and 32 bit OpenAL was installed in my system, specifically libopenal.so was now present in /usr/lib, whereas before it was only in /usr/lib64.

I then started AoS again from Steam and it worked! Happy Ending. Well, almost, after happily playing it for a while, my son was kicked from the server by other players. It's not the first time he's been accused of cheating when, in fact, he's just very good at the game!

Scottish independence and inequality

2014-09-04T12:03:00+01:00

Since I wrote my last blog post on independence back in July, I've thought about the matter more and spent more time reading about the issues involved and discussing them with other people. I'm not going to talk about the high profile debates, nor am I going to comment on the campaigning other than to say I'll be very glad when it's over. This post is going to concentrate on the issue of inequality and how that relates to Scottish independence and the UK. This, as I highlighted in my last post, is important, because although the majority of folk voting for independence are nationalist, there is a significant minority that are more interested in socially progressive change.

Scotland, it is often stated, is a prosperous country with a bounty of natural resources, both renewable and non-renewable. No argument there. But those arguing for independence go further, it could be more prosperous and keep more of its wealth if it left the UK. The Better Together campaign make the same claim in the opposite direction - Scotland would be better off remaining in the Union.

There's no way to be sure which of these claims is true, because no one can predict the future. But let's take another claim made by supporters of independence, that, under current arrangements, Scotland is shortchanged by the way that UK public money is shared with Scotland by the Barnett formula. This leads to the question: is Scotland already more prosperous, but not benefiting as it should because it's being shared with the rest of the UK? Even though that does not involve predicting the future, it's still hard to decide objectively because it relies on agreeing on a fair allocation of resources and wealth. I have not looked at all the details of the Barnett formula, but I have looked into health spending and it's clear that at present more money is spent per head on the NHS in Scotland than in England, and the hospitals seem better resourced in Scotland too. I saw this firsthand earlier this year when I spent time in similar departments of Scottish and English hospitals. The NHS in Scotland is completely devolved already, so the only way the Barnett formula could threaten spending on it in Scotland is if public health spending in the whole UK fell, which has never happened. Such a cut would be a sure-fire election loser for any political party. This BBC article by Eleanor Bradford accords very closely with what I've come to understand.

But, let's put such details aside and suppose, for the sake of argument, that Scotland is intrinsically more wealthy and that it has historically shared its greater wealth to benefit the rest of the UK. Let's further suppose that its relative wealth will be boosted if it becomes independent or receives further devolved powers. Unless stated otherwise, I'm using 'wealth' to refer to a measure of wealth per individual. This is of course a poor metric, as more wealth does not necessarily equate to better quality of life, but it'll do for this argument.

It follows then that if Scotland gains independence, with all else being equal, the rest of the UK will be deprived of the wealth benefit of having Scotland. When making this point to a friend, he immediately responded "I'll be in an independent Scotland and it's not my business how the rest of the UK fares." I could've pointed out that Scotland exports £47.6 billion to the rest of the UK but only £26.0 billion to the rest of the world. These are Scottish Government 2012 figures both excluding oil and gas. If our biggest customer is made poorer, it must have some effect on Scotland. This is not the argument that's most important to me though, there's another ethical one which I'll explain by way of something I experienced myself.

Eleven years ago I moved from rural Cambridgeshire to a working class (or lower middle class, if you prefer) area of Glasgow. I was rather shocked to discover that my council tax bill had doubled despite my new house being of similar size and value. Why was this? The explanation was that Cambridgeshire has a much more affluent population than Glasgow, partly because of Glasgow's post-industrial past, but also because the better-off folk who work in Glasgow reside just outside the city in the suburbs. The result is that a much greater proportion of the population of the City of Glasgow pays little or no council tax (whether legally exempt or otherwise) as compared to Cambridgeshire. And I saw the same phenomenon again when I recently moved a very short distance out of Glasgow into one of those wealthier suburbs. My house is about twice the size, and valued at much more than the previous house in Glasgow, but the council tax I pay remains much the same. Same explanation - I now live in a more affluent area where a greater proportion of households pay council tax.

What's that got to do with independence? Well, once I realised why I was paying more tax in Glasgow, it seemed fair to me. The extra money I was paying in tax - which I could afford - was going to subsidise the less wealth off, of which there are too many in Glasgow. And this is the ethical point I'm making. If I earn more than others, I think it is fair that I also contribute more in tax towards public services: roads, schooling, cleansing, public swimming pools etc. Almost all of us in the UK agree with that principle, so much so it's enshrined in law that we pay various taxes.

So my point is that, on this ethical principle, I think that it is fair that Scotland shares its extra wealth with the rest of the UK. I can emphasise the point by applying the same principle to a different situation. If a wealthy section of Glasgow - say its West End - said "We're fed up subsidising the poorer parts of Glasgow by paying all this extra council tax" and took a community vote to split with the City of Glasgow council, I couldn't disagree with the democratic outcome, but I, like many Glaswegians, would see them as extraordinarily selfish. The wealthy people of the West End would be wealthier and the poor of Glasgow would be poorer. In fact, you may have already have noticed that this is not a hypothetical situation - it happened when Glasgow's wealthier population moved out into the suburbs. The wealthy pay less council tax due to historical and geographical boundaries.

I therefore reject any argument that Scotland should be independent simply because it is, or could be more wealthy than the rest of the UK. And conversely, if Scotland needed financial assistance from the UK, I'd welcome it, as happened when the Royal Bank of Scotland needed bailed out in recent years. The difference in wealth between England and Scotland is certainly not as dramatic as that between rich and poor areas around Glasgow, but that's not the point, my objection is ethical, not economic.

Intimately related to wealth, or the lack of it, is the issue of job creation. Let's consider a scenario that has plenty of precedent in reality. A politician in a deprived area of Glasgow is keen to get a big multi-national company to move into their area and set up a big factory (or call centre, or some such large workplace). It will create jobs and the local population, having endured years of deprivation and high unemployment, are right behind her. The company agrees to move in, but only if given a generous tax break, and the politician, keen to serve the interests of her community, manages to get government agreement on that. At face value, it seems like a good deal. But you have to look at the bigger picture.

If the company is expanding then this job creation is real. However, it's often the case that a company is not expanding but just relocating production, and in so doing closing down a factory in another hard-up community elsewhere, blighting that community with sudden unemployment. If this other community were in Glasgow, I'm pretty sure there would be sufficient outrage to flag the whole deal as a scandal. But, it might well go through if the other community were in a distant town, and it will almost certainly be overlooked if the hard-up town was in another country. Why? Because the local population and the politicians have their own problems and have no interest, or perhaps no knowledge, of what's happening in that other country. Overall, no jobs are created and some may have been lost. There's also a precedent set for tax breaks that could drain the public purse in the future. The only certainty is that the big company has driven down costs to enlarge profits and, in all probability, it will be the already-wealthy shareholders that benefit most. This kind of scenario is one reason why inequality has risen in the Western world since the 1970s.

So if a country tries to create jobs with a simple tax cut for business, which is what the Scottish National Party (SNP) are currently proposing for an independent Scotland, then this is effectively a subsidy from public money or, in other words, a gift from the shared money of the populous. Not only can jobs be lost elsewhere, but a drive to cut costs might also mean the move is motivated because the new workers are willing to accept less pay than the existing workers. Politicians and the public will agree to it because there is the illusion of progress in reducing local unemployment, but overall this exacerbates global inequality.

If you do not find my argument convincing, or feel that I'm not qualified to comment, then listen to Nobel laureate in economics Prof. Joseph Stiglitz - a member of the SNP government's Fiscal Commission working group - who describes it as “just a gift to the corporations increasing inequality in our society.”. A tax cut is more typical of a right wing party, but even the Tories in Westminster realise that a tax cut to big business is a bad idea (and I don't agree with the Tory policy either).

There is increasing recognition that growing inequality is a serious problem. The cause is that governments are not able to deal with big international businesses. The policies of the UK and those proposed for an independent Scotland are misguided and are likely to increase inequality and, as I have argued, they also have a dubious ethical basis. Solutions do exist to the problem, and the fact that inequality fell throughout the middle of the 20th century shows that, somewhat incredibly, we found solutions, but then forgot them! One example is FDR's New Deal in the 1930s that created real jobs in the USA without borrowing. Another quite different solution came from John Maynard Keynes's negotiation of a huge loan in 1946 from the USA to the UK. As a consequence of that loan, which was just paid off in 2006, the UK not only escaped post-war bankruptcy but its employment and wealth rose, and inequality fell. Massive public spending by the UK government, with control of its own currency, created the world's best public health care system - the NHS.

I'm not saying we simply repeat these brave and innovative schemes, nor that the UK must remain as it was then, but we should be inspired and learn from these examples. If proposals like those were being made for an independent Scotland, I'd back it enthusiastically. But as it is I'm getting angry with politicians, and some others, for making promises on social progress that they clearly will not be able to keep because they are based on vague, vote-winning initiatives or poorly thought through policies, such as corporation tax cuts. The people they have misled, and will likely let down, are already the ones that society has shortchanged. I say that we can deliver on those promises if we plan properly and recognise that inequality is a global problem that needs global solutions. We can create a fairer society for Scotland, a country of about 5 million people, but would it not be better if we did it for a country of 60 million, or 300 million, or more? We've done it before.

Chalkahlom pilau

2014-08-25T21:23:00+01:00

"Good evening dear. Would you like to eat a meal cooked with this red powder sent from Hungary by a chap I've only ever met on the internet?"

I didn't introduce the meal like that in actual fact, and if I did I might have been sent out for pizza. Hmmm, there's an idea. Anyway, here's my recipe for what I call Chalkahlom pilau.

2 cups of rice
2 cups of vegetable stock (or water plus a stock cube)
1 can of kidney beans in water
4 rashers of smoked bacon, finely chopped
1 medium onion, finely chopped
2 tablespoons of olive oil
1 teaspoon of paprika (preferably from someone you've never met on the internet)

Put the rice in a rice cooker and add the two cups of stock, or add the two cups of water and crumble in the stock cube. (If you use a pot rather than a rice cooker then you might need to adjust quantities.)
While the rice is heating up, fry the onion in a frying pan or wok on high heat for several minutes, then add the bacon.
While the bacon is frying, open the can of kidney beans and add its contents (including water) to the rice cooker.
Return to stirring the frying onion and bacon and keep it on high heat until it just starts to brown and caramelise, then turn to lowest heat.
Wait until the rice has absorbed almost all the water then sprinkle the teaspoon of paprika over the rice and add the fried onion and bacon.
Once the rice cooker is done, serve, and keep the emergency services on speed dial in case that chap in Hungary isn't as nice as he seems.

Scottish independence

2014-07-22T16:03:00+01:00

On September 18th 2014 people resident in Scotland will be asked "Should Scotland be an independent country?". I am not in favour of independence as currently proposed, partly because I am concerned that it is really driven by nationalism, but also because I think the possible benefits are not (yet) worth the risks. This post sets out my thinking around two key questions: what are the main benefits and risks; and who and what will shape a newly independent Scotland?

Before going further, let me first explain my background. I was born in Glasgow, moved away, and later chose to return to the city so my wife and I could raise our children here. However, my mother was from Iran and my father was born in London and his parents were Irish immigrants. Perhaps this mixed ancestry is why I have almost zero sense of national identity. Not only that, I also struggle to see how national identity adds much to progress of humanity through history. In fact, I see it as a limitation that needs to be overcome.

The onus is on the people pushing for independence to set out their plans, and, for the most part, this has been done by the SNP - the Scottish Nationalist Party - who currently govern in the Scottish parliament. They published their white paper in November 2013.

This brings me to my first problem: if Scotland does become independent, it will mainly be shaped by people with nationalist sentiments. (Please note 'mainly', I'll address this qualification later.) Many of my yes-voting friends are clearly motivated by their sense of national identity, and some will openly admit that they are going to vote yes with their hearts and not their minds. I don't dismiss it as the "Braveheart" vote, but instead respect such honest and acknowledged irrationality, because, like it or not, we are not rational beings and it's hard, sometimes impossible, to overcome subconscious influences. For myself, I make no claim other than I attempt to be rational in certain arenas, and root out my own contradictory beliefs. But, to return to the point: since I'm not the least bit nationalist, there's very little for me to engage with here, even with those who claim not to be motivated by nationalism. Their arguments are designed to justify what they already believe: Scotland is great, and would be greater if independent.

But it's hard to ignore nationalism in Scotland. Over the last decade or so there's been a surge in support for the SNP. The most striking result of this is that the SNP won enough Scottish parliamentary seats (MSPs) for a majority government in the 2011 election. If you don't know Scottish politics, you might be tempted to conclude that this means that the Scottish population now favoured independence. Numerous polls show this not to be the case though, with support for independence going from its historical 30% average to peak at about 40%. Recent polls show it sitting between these two values. [EDIT 12 Sept 2014: the poll data in those links have been updated since this post was written.]

The SNP should be given credit for running a tight ship under its leader, Alex Salmond, who is a good orator and an experienced player in the party political mainstream. But, their success was also due to the Labour party's support reaching a natural nadir after 13 years of government, and because the Liberal Democrats, to whom some Labour voters might turn, went into coalition with the Conservative (aka Tory) party after the 2010 Westminster elections. Since the Tories are unpopular in Scotland, this meant Labour and Lib Dem votes went to the SNP, even from people who don't favour an independent Scotland.

The key point here is that many people in Scotland feel that the UK parliament in Westminster is more irrelevant to them than ever before. I feel that too, and baulk at the policies and rhetoric that come from the coalition ruling in Westminster, especially from the larger Tory contingent led by Prime Minister David Cameron. However, the Scottish Nationalist Party government led by First Minister Alex Salmond over in Edinburgh is equally unappealing to me. I know that the issue of independence goes beyond just the SNP, but it's important to take a close look at them because they are clearly dominant in proposing what an independent Scotland will be like.

The Scottish Nationalist Party have historically been what their name suggests: Nationalist and Scottish, and a mainstream political party. They present themselves as centre-left just now, but go back a few years and they were cheering on the banks escape from state regulation into the land of mythical success (specifically the Royal Bank of Scotland and their ex-CEO not-Sir Fred Goodwin), basing their independence economics on big oil money (renewables came later) and saying that Scotland could be a "Celtic Lion" economy like Ireland's Celtic Tiger. If you think I exaggerate please read this 2008 speech by Alex Salmond at Harvard. The mainstream media obsession with "now" mean that views of various political leaders tend to be forgotten all too quickly and the SNP have, like other mainstream parties, used this to their advantage. Notice that the link I provide is to web.archive.org because that speech has disappeared at least once from the official Scottish Government website. It's also worth noting that prior to his successful political career, Alex Salmond was an "oil economist" for the Royal Bank of Scotland.

It seems to me that the SNP only started promoting socially progressive policies after the economic crisis had begun and they realised that the Celtic Lion vision was no longer popular with people struggling with the consequences of the credit crunch. What the SNP currently offer on education and health is positive, and does set them apart from the current Westminster government, but they are also keen on pro-big-business tax cuts and are largely silent on reducing inequality through fairer taxation. Alex Salmond may be a good orator, but I find him inconsistent and shifting in the substance of what he says. I would not trust him nor senior SNP figures to usher in an independent Scotland for the better, either in a social or economic sense. I believe their primary agenda has always been nationalism: what's best for Scotland as a nation, irrespective of what's best for the rest of the UK, or indeed the people of Scotland. History has so many examples where action in the interests of a nation served to degrade the welfare of its own people; such is the paradox of nationalism.

However, as I emphasised above, the call for independence is not just from the SNP and other nationalists. The extra support for independence, and it's somewhere between 5 and 10 percentage points, sending it from about 30% to up towards 40%, comes from some people making arguments I can engage with. Specifically, these people do not want to be ruled by the current coalition of Tories and Lib-Dems in Westminster; they don't want the likes of Tony Blair leading us into futile and/or illegal wars; they want a fairer society in which overpaid fools who run financial institutions can't undermine the wealth and wellbeing of the general population; they want to protect the NHS from privatisation; and so on. The late Scottish author Iain Banks expressed such views very eloquently. I'm concerned about all of these things, but I disagree that it's likely that independence as proposed will improve matters. Why? Because the people that want this kind of change are in the minority (sadly), even amongst pro-independence supporters.

If independence offered a reasonable chance for such positive change, even with a less than 50% probability, I might be persuaded to make the leap of faith. But there are serious concerns that must be properly addressed, and of these my biggest concern is on the currency issue. But understand that my concern is not the superficial one put forth by the likes of the "Better together" campaign. Let me explain.

An independent Scotland as proposed will be using a foreign currency, the Great British Pound (GBP). There's a whole other blog post in this, but, in short, a country that uses a foreign currency is not truly independent. Even a small country can steer itself through an economic storm if it has control over its currency. Look at the last economic crisis. Ireland and Iceland were both crippled by bloated finance industries, but it was Ireland that had to suffer much more post-crisis because it was tied to the strong economies such as Germany via the Euro. In contrast, Iceland fought against the worst effects of the crisis quite ingeniously, and a key part of this was in devaluing its currency which, in effect, made all Icelanders take an equal pay cut. Ireland could not do this and was compelled by Europe to implement austerity which hits the poorest disproportionately. Spain and Greece too suffered the same fate as economies without control over their currency. In contrast, the UK used its control over the pound to create money (quantitative easing), which helped to a degree, but then the Tories tried to implement austerity, even though they didn't have to - there's another blog post/rant in that.

The defences I get against the currency point from yes-voting friends are: a) that it doesn't matter; b) let's not be restrained by fear; c) we can fix it down the road. My answers: a) OK, you're voting with your heart then; b) I'm not, I'm saying let's do independence properly and avoid an unnecessary risk because the root causes of the last economic crisis have not been resolved and will re-emerge in years ahead; c) no, this is something that can be, and definitely should be decided ahead of time - look before you cross the road.

But there's another issue that doesn't get discussed very much, but is of some importance to me. If Scotland votes for independence, I lose my vote for a representative (MP) in a big and influential parliament down in Westminster. It's true that the parliament doesn't currently represent me much, and that mainstream party politics fails to engage me, but that's an argument I can equally make about Scottish politics, including the referendum itself. Both the Yes campaign and Better Together are disingenuous and tedious bandwagons using the traditional and social media to broadcast sound-bite rhetoric with very little logic. The party political system and big business may mean we have a compromised democracy right now, but that does not mean I'm willing to give up on it completely and let go of my vote.

I'll make the same point in a different way. The political and business classes in the US certainly like to meddle in foreign affairs, e.g. dragging the UK into wars in the middle east. So, can I please have a vote for the Leader of the Free World..., sorry, US president? And how about giving it to the people of Iraq whilst US agencies and soldiers do their "work" there? Isn't that democratic? I'm serious, I'd take a vote in US elections if it was ever offered to me. International voters like me could make a difference - I'm pretty sure George W Bush wouldn't have won with an international vote.

So, my position is this: the referendum is to decide independence, not anything else. I want to see a change for a fairer and more equal society, and I wish for a cleaner environment, and want a healthier NHS, amongst other things. Those are causes to fight for in themselves, and I don't want them just for me, and my neighbours who happen to live in Scotland. That line between England and Scotland has little meaning to me and I don't want to be distracted from more important issues by expending too much thought and effort on national and historical divisions. As cellist Pablo Casals said "The love of one's country is a splendid thing. But why should love stop at the border?"

Life - rocks, gravel and sand

2014-05-18T15:02:00+01:00

Many years ago (about 13 if you must know) I was standing waiting in a colleague's office when I happened to notice a sheet of paper pinned to the wall. On it there was a picture of a pint glass filled with small rocks, gravel and sand. You might know the story that goes with it. If you put the rocks in first, then the gravel and then the sand, you can fill all visible space in the jar. But, if you start with the sand, then you cannot. A neat analogy is made if the rocks represent the important things in your life - family, friends, home and perhaps fulfilling work - and the sand represents the minutae of life that doesn't matter but fills up your time.

I was very struck by this analogy and have found that it fits well with my experiences in life: if you sort out the fundamental aspects of your life, the smaller bits will often just fall into place.

It applies more generally too, I think. In running a company, if you get the right structure, hire right people into key roles, then you don't have to worry about, say, the website being filled with nonsense and the accounts descending into a shoddy mess.

Of course, life is full of compromises and also there's an element of luck. Through no fault of your own, you may find that you started off with a dollop of wet sand at the bottom of your glass, or that without realising it you've allowed too much gravel to build up. I could give examples, but I suspect most people can identify the sand and gravel in their lives. In my life, a lot of it pours in through my smartphone.

If I said I spent the morning loading the washing machine, hanging up washing, taking the rubbish out and weeding the garden, what would you class that as? Gravel? Sand even? It was definitely rock. Why? Because I did it all with my two year old daughter, and we filled a glass with rocks, gravel and sand too.

Mortgages and mathematics

2014-01-08T22:02:00+00:00

I've had to think about mortgages lately. At first this seemed like a boring chore, but it generated a bit more interest (pun alert!) when I noticed that most mortgages are based on a mathematical series. What's more, I found it hard to find a clear explanation of the mathematics involved - hence this post.

A mortgage is a long term loan usually used to fund the purchase of a property. There are two types of mortgage. The simplest kind is interest only, and that really is rental of money. You don't have to pay anything more than the interest during the term of the loan, but at the end you must pay the loan amount back in full. You could do that by selling the house, or by using some savings or other investment. In real terms, you don't pay back the amount you borrow, because inflation will have substantially eroded it: £100,000 in 1994 bought a lot more than would £100,000 in 2014.

The more common type of mortgage is the repayment mortgage, also referred to as capital and interest. In this scheme, you pay off some of the loan, often referred to as 'capital', as well as the interest at regular intervals over the term of the mortgage so that by the end of mortgage term you owe nothing and own the property.

In this blog post, I want to consider how payments are split between capital and interest. If you appreciate that interest payments will decrease on every payment of capital, you'll soon see that this split can significantly affect the total amount of interest paid over the whole mortgage.

I'll use the example of a £100,000 mortgage for a 20 year term at a 6% annual interest rate, where the payment is made, and interest calculated, at the end of each year. In reality, most mortgage payments are made more frequently, most usually monthly. This can also affect the overall cost of the mortgage, but isn't what I want to highlight in this post. I'll give a link at the end to a good online book that deals with that and other details.

At the end of the first year, you pay interest on all of £100,000, so that £6000 of interest is due. Let's say that you choose to pay £10,000 in capital. The first thing to realise is that you will not pay any interest on that £10,000 for the next 19 years. This saves on paying £600 per year for the next 19 years, which totals to £600×19=£11,400.

Clearly, the borrower will benefit from paying as much capital as possible early on, and conversely, the lender benefits most if little capital is paid early on. But, do lenders force borrowers to pay little capital up front?

To answer the question, let's start with what I consider to be the most straightforward approach. Each year, the borrower pays back the same amount of capital £100,000/20=£5000 and pays interest on the amount outstanding. So the schedule will look like this:

Year	Amount owed	Interest
0	£100,000
1	£95,000	£6000
2	£90,000	£5700
3	£85,000	£5400
...
19	£10,000	£600
20	£5000	£300
21	£0	£0

The combined repayment plus interest in year 1 is £5000+£6000=£11,000 and reduces by £300 each year until in the last year it is only £5300. This is what makes most sense to me because it is the simplest mathematically, but this is not the standard schedule used by most lenders.

The standard schedule involves keeping the total payment of capital plus interest constant. I see some sense in this, in that it is easier to explain to a typical customer during the sales pitch: "You pay this amount per year, every year for the next 20 years". But, it leads to mathematics that most mortgage customers would find far from simple.

The formula from the standard schedule would give a constant payment for the above example as £8718 per annum. This is less than the starting £11,000 in my schedule but more than the final £5300. Surprise, surprise... lenders discourage paying more capital up front. Over the life of the loan, my schedule's total interest is £63,000 and the standard schedule's total is £71,708.

But, before you start finger-pointing and baying at the evil, greedy capitalist (and interestist) lenders, do bear in mind that it could be much worse. For example, the lender could just zero capital payments in, say, the first five years. Or lenders could only offer interest-only loans that would result in a total of £120,000 in interest payments. They don't do these things (normally) because they are mindful of the risk of the loan never being paid back, and because there is competition amongst lenders.

So, what is the standard way, and what is the magic formula that produces the £8718 figure? Time for a bit of maths...

Let the constant payment $p$ be made at the end of each year $n$, where

\begin{equation} p=C_n + I_n \label{payment} \end{equation}

where $I_n$ is the interest and $C_n$ is the capital paid at the end of year $n$. Our goal is to find a formula to calculate $p$ from the loan amount $L$, the mortgage term $N$ in years and the interest rate $i$, expressed as a fraction, e.g. 6% would be $i=0.06$.

The interest paid at the end of a year is on the loan amount less all capital payments in previous years, so

$$ I_n=i(L- \sum_{k=1}^{n-1} C_k) $$

So for $n=1$, the interest is just calculated on the entire loan, i.e. $I_n=iL$. And $I_2=i(L-C_1)$, and $I_3=i(L-C_1-C_2)$ and so on.

Next, let's look at equation (\ref{payment}) for $n=1$:

$$ p = C_1 + i L $$

which we can re-arrange to give:

$$ C_1 = p - iL $$

and then $n=2$ gives

$$ p = C_2 + i (L - C_1) $$

and we can substitute for $C_1$ and re-arrange this to give

$$ C_2 = (p-iL)(1+i) $$

and if we do this again for $C_3$ we get

$$ C_3 = (p-iL)(1+i)^2 $$

You can probably see the pattern emerging, so it shouldn't be a surprise to learn that it can be proved (by induction) that

\begin{equation} C_n = (p-iL)(1+i)^{n-1} \label{capitalPayment} \end{equation}

The sum of all capital payments must total to the loan amount, so

\begin{equation} L=\sum_{n=1}^N C_n \label{capitalSum} \end{equation}

Let's put $r=(1+i)$ and substitute equation (\ref{capitalPayment}) into (\ref{capitalSum}) to give

$$ L=(p-iL)\sum_{n=1}^N r^{n-1} $$

The sum is that of a geometric series and so we can write this as

$$ L=(p-iL)\frac{r^N-1}{r-1} $$

Now we can re-arrange to give the formula for $p$ we've been seeking

$$ p = \frac{Li}{1-\frac{1}{(1+i)^N}} $$

This is the formula that gives the annual payment of £8718 quoted earlier, but it is not very enlightening by itself. To understand why this payment schedule is significant enough to have become the standard, it's worth reviewing how we just derived it.

First, we asserted we wanted a constant overall payment, and that for each year we could write it as a sum of a capital repayment $C_n$ and an interest payment $I_n$. Then we said that the interest payment $I_n$ is determined by the interest rate $i$ and the capital payments to date. This gives us $N$ equations - equation (\ref{payment}) for each year - for $N$ unknowns - the $C_n$s. We then solved this system of equations for the unknowns which let us calculate the constant payment $p$ in terms of $L$, $i$ and $N$.

So, to go back to the question posed earlier: do lenders force borrowers to pay little capital up front? The answer is no, based solely on the mathematics. If the simplicity of a constant payment is desired, then the standard formula used by lenders is mathematically justified. Further, if the discerning borrower wishes to minimise the total interest they pay out over the whole term, they can choose a mortgage that allows them to decide the schedule of capital repayment, and abandon the constant payment restriction.

There is another way to understand the standard schedule: imagine that no interest nor capital is repaid on the initial loan, and that it grows and that interest is compounded. That is, interest is added to the loan total each year and so interest is subsequently paid on previous interest. Also, imagine that the capital repayments are paid into a savings account and they earn interest which is similarly compounded. Eventually the savings catch up with the debt, resulting in the loan being paid off. You can see this approach detailed on this page, but beware, the notation used is totally different to what I've used in this post.

If you'd like to know more about the real-world details of mortgages, then the online book Mortgages Exposed is well worth consulting. I'm not sure it's been updated in recent years, but most of it is still relevant and the spreadsheet tools are very useful (and worked fine in LibreOffice).

One final note. In the UK at present, there are mortgages with interest rates under 2% and savings accounts with interest over 2%. If you are fortunate enough to have enough savings to pay off your mortgage, you'd be financially worse off to do it, because interest on your savings can exceed those of your mortgage. If the mortgage is for your home, that means a bank will be paying you rent whilst you live in your home. Some other thoughts in a similar vein can be found in my previous post Lending is good and bad.

Mazes - Minecraft on the Raspberry Pi

2014-01-05T19:14:00+00:00

Minecraft for the Raspberry Pi has been with us for about a year now. It's cut down quite a bit from the normal version, being much closer to the pocket edition available for phones and tablets. I wasn't particularly interested in it until I read recently that it comes with a python API. This makes it easy to write scripts to do a great many things, for example, rapidly building structures like castles, bridges and towers. Not only that, but you can make things appear and disappear, or change their structure - think staircases in Hogwarts!

I'm not going to go into the detail of how to do it, because that's already been done very well. See this blog post about a free book written on the subject. I've not read it all, but from the bits I dipped into, it seems like an excellent resource for teaching children (and possibly adults) who are into minecraft and curious about coding in python. The author of the book, Craig Richardson, has a blog full of interesting tips too. I also found this API reference by Martin O'Hanlon to be very handy.

So, on a rainy, winter's Saturday evening, I decided to dabble with this python API and write some code to generate mazes in minecraft. This is a rough sketch of what my quickly hacked-together code does:

Sets a number of rooms, a room size and a room height.
Creates a big cuboid of stone using setBlocks() to house the entire maze.
Creates an entrance and an exit at opposite corners.
Moves an imaginary player to the centre of the first room.
Hollows out the room.
Randomly chooses a wall to make a door in.
If the door coincides with the exit, then STOP - our job is done!
If the wall is exterior, go back to 6.
Move through the door to the centre of the next room, go back to step 5.

It's not particularly efficient, because the imaginary player will often visit the same rooms over and over again, until, by chance, it enters the room adjacent to the exit and chooses to make a door there. But, on the upside, the algorithm does create a viable maze in a few seconds, without any risk of getting stuck in a never-ending loop. To speed things along, and reduce the load on my pi, I stopped it hollowing rooms after the first visit.

To make the maze a little more interesting, and not completely dark, there is a 50% probability of a torch being planted in the centre of a room. Also, to add a little distinctiveness and colour, there is a lower probability that the block under the torch might be made of iron, gold or diamond.

Another thing I added to the script was the ability to erase the previous maze. If any command line argument is given (super hacky!) then the script performs the setBlocks command with block.AIR instead of block.STONE and then exits.

The final thing worth mentioning is how I set up the development environment. I tried using IDLE which came bundled with Raspbian, but it wasn't to my taste. Geany is my weapon of choice for python coding, but instead of rubbing a lamp, I did this:

apt-get install geany

But another problem soon presented itself: the minecraft window was a little messed up and it wouldn't go behind other windows, including geany. Also, when it came time to look up stuff in the web, I found midori on the pi frustratingly slow.

So, to speed things along, I switched to working on my laptop and installed sshfs and mounted a directory with a command similar to this:

sshfs raspberrypi:/home/pi/minecraftpimaze minecraftpimaze

which meant that I could access all the python files on the raspberry pi from my laptop. I fired up geany, ssh'd into the pi in geany's terminal window and was soon coding away in a very sweet development environment with my laptop sitting on the table next to the pi. I should mention that the pi was connected to an old TV, and it took a bit of fettling to get a decent resolution - for sage advice on this point see this blog post.

Of course, I spent almost no time actually trying to solve the mazes I'd created, because, as I'm sure you will agree, the real fun was in coding their creation.

Robot arm, Raspberry pi and wiimote

2013-12-08T21:58:00+00:00

Last week my 8 year old son was tasked with making a robot for a school project, but one made from toilet roll tubes, aluminium foil and possibly an LED or a wee speaker to make a noise. I said unto him, "No, you shalt make a real robot!" and he replied "But, how wise pater?" and I said "Lo! Let us go forth to Maplin and searcheth the webs and employ Raspberry of Pi."*

So we went to Maplin and bought this robot, and spent a companionable Sunday afternoon putting the robot arm together. Gear boxes needed constructing, motors needed wired to the control board and numerous small screws, fixings and nuts needed careful manipulation to bring the black and yellow mechanical beastie into this world. It certainly wasn't an easy undertaking, but the instructions were good and we worked methodically and carefully for about 2 hours with a couple of short breaks.

When we finished I installed the Windows-only software on my desktop PC running XP, got my son to plug it in and switch it on and... it worked first time! My son played about with it for a bit and we were particularly tickled by his attempts at getting the arm to dip down to the keyboard and press the key that would make it reverse its movement and move back up - yes, we'd made a self-controlling robot. Scarey.

But, he certainly wasn't going to take this PC into school, nor was it feasible to install the (closed-source) software on the teacher's PC because it'd probably require a call to some off-site IT support, incurring a cost to the school of more than the robot arm itself.

So I suggested to my son that we a) use my raspberry pi and b) use a wiimote to control the robot arm. This was met with some enthusiasm.

A quick search of the web led me to Poly's Python blog, which, although not updated in almost a year, contained an interesting record of someone hacking python on GNU/linux until it controlled the robot arm.

The last update on that blog tells of Matt Dyson's further work on developing a python class and getting an XBox controller to control the robot. I checked his python code out of his SVN repository, connected the robot arm up to my laptop (a Dell XPS 13 with Ubuntu linux 12.04 LTS) and tried to run his testRobotArm.py script:

sudo python testRobotArm.py

This failed with a message about usb.core because I'd skipped the step of installing PyUSB 1.0, which was clearly described in his blog post - RTFM! After that was corrected, the robot arm waved around in its programmed test dance.

Next up, getting the wiimote to talk to linux and python. A quick web search led me to Wii Controller + Raspberry Pi + Python = Awesome!! and all I had to do to get the python classes to give me this awesome python-powered wiimotiness was:

sudo apt-get install python-cwiid

Bluetooth seemed to be working fine on my laptop, so to prove my set-up was working, I downloaded and ran the script wiimotetest.py on the command line:

sudo python wiimotetest.py

and was able to see output in response to pressing buttons on the wiimote. This was too easy, but the work was not yet done.

Next, I procured a cheapo, never-heard-of-before-brand (signalex, I think), USB bluetooth dongle from a local electronics shop. I took it home, stuck the latest Raspbian on my pi, updated it, which took an hour or two, then shoved the bluetooth dongle in, and followed the instructions on the above-mentioned website and once again had wiimotetest.py proving the wiimote connectionworked.

I should say that my son wasn't very involved with this bit, but I did explain what was going on: he was the one that would have to take it into school, set it up and operate it in front of his class.

At this point I had all the information I needed to start hacking away in python to get the wiimote to control the robot arm. But, being very tired, overloaded with a sudden deluge of work and having various family members succumb to a flu/cold, I turned once more to a web search, and found some code by a chap called Keith Jefferies on the official Raspberry Pi forums. Once again, I just downloaded the code, ran it and bingo! I was controlling the robot arm with a wiimote. My son was immediately delighted.

But, some difficulties now presented themselves. I was using the raspberry pi connected to an old TV via HDMI. My son would have no display to use at school, so the raspberry pi would have to boot up, start the script and then he'd have to press buttons to initiate the connection between the wiimote and robot arm.

The second problem was that, despite my initial successes, the connection to the wiimote and robot arm from the pi sometimes didn't work and required two or three attempts, and sometimes both the robot arm and the pi needed to be powered on and off. He'd have to learn to do this guided only by flashing LEDs on the wiimote. After I hacked the code a wee bit and added a line to /etc/rc.local, my son learned the tricks to getting it all connected by pressing the various buttons at the right time. In fact, he quickly became more adept at it than me.

My tweaks to the code of Keith Jefferies is in this script robotarmwii.py (he didn't specify a license, but I'll get in touch with him) and here's my /etc/rc.local that gets the script running on boot.

Bottom line: one happy geeky dad, one happy proto-geek son and an impressed teacher and class, who all got a glimpse of the power of Free and Open Source software, Python and the Raspberry Pi. I do have to report that my son and his pals did use the robot to terrorise some hapless lego mini-figures, and trash the pi's lego case, as evidenced in the above photo.

*This dialogue isn't 100% accurate.

Space debris

2013-11-18T13:48:00+00:00

I was recently asked to speak on the radio about space debris. This was prompted by the release of the film Gravity (1), and the re-entry of the GOCE satellite (2).

You can listen to the interview here (Copyright: BBC, broadcast on Radio Scotland, 16 November 2013). At the beginning of the audio clip is the tail-end of a report by my friend Pauline McLean who, unlike me, is a professional. I include it not only because she has a lovely voice, but there's a great wee clip from Mrs Browns' Boys.

Below are some more in-depths thoughts on the subject of space debris, drawing heavily on many excellent sources of information which are referenced below.

What is it?

Space was already fairly well populated with debris by the time the human race put Sputnik into orbit in 1957. This debris was natural, mostly being left over from the formation of the solar system 4.6 billion years ago, with some pieces coming from subsequent collisions and the detritus left behind by comets. This natural debris takes the form of small grains of dust, pebbles, boulders, some being rocky, others containing ice too. The smaller objects are far more numerous than the larger ones.

When one of these objects happens to encounter the Earth, it will likely burn up in the atmosphere due to air friction, leaving only a fleeting streak of light for the human eye to catch. These meteors, also known as shooting stars, are quite frequent: stand under a decent night sky and look up and you'll see one within an hour, unless you're unlucky. Larger objects may create a bright fireball in the sky and fragments of it may reach the ground; this is called a meteorite. Very large objects have the potential to cause serious damage and even loss of life, but that's a subject for another blog post.

In recent decades, natural space debris is being joined by debris that we've put there. Most of our satellites are unlikely to collide with each other because they are of value to us and we have been careful to place them in distinct orbits. So they present no problem. However, in launching these, we have discarded stages of rockets, released smaller objects when stages separate and there's a myriad of tiny objects such as paint flecks, droplets of coolant and ice crystals now orbiting the Earth. Accidents and explosions have occurred, contributing further to the orbital debris.

The larger objects, down to about 5 cm in size, can be tracked and so avoided. However, smaller objects are not only more numerous and difficult to track, but they still present a danger because of the large speeds involved.

A little physics helps make the point. The kinetic energy of an object is the energy associated with motion, and the greater the energy, the greater the damage caused by an impact. The kinetic energy is proportional to mass and the square of the speed: double the mass and you double the kinetic energy; double the speed and the energy quadruples. For this reason, even a fleck of paint can cause serious damage to a satellite.

The Kessler effect

But there's another process which amplifies the space debris problem. When two objects collide they fragment into many more smaller ones. Many objects in low Earth orbit will, in weeks, months or years, experience atmospheric drag and suffer orbital decay which will cause them to burn up, or, if large enough, fall to Earth. But, if the number of objects in low earth orbit exceeds a certain critical value, the frequency of collisions will mean that small objects will be created faster than they are removed. In other words, even if humans don't launch anything else into orbit, the number of objects will continue to increase for decades to come. This is called the Kessler effect. It is worrisome that many recent estimates tell us that we are already at or above the critical number.

A low Earth orbit (LEO) is any orbit with an altitude between 160 km and 2000 km. These include polar orbits which, as the name suggests, take the satellite over both poles. It is LEOs where the space debris problem is now acute. The geostationary orbits (GEOs), in which satellites can remain stationed above a particular point on the Earth's surface, are at a much higher altitude of 35,786 km. Space debris is less of an issue for GEOs just now, but since there is virtually no atmospheric drag at that high altitude, space debris can take centuries or millenia to be naturally removed, and so it's important that LEO space debris mess is not repeated for GEOs.

Notable events

On January 11 2007, the Chinese intentionally destroyed a weather satellite called Fengyun 1C to test an anti-satellite weapon. This immediately created several thousand new pieces of space debris. A little over six years later, a piece of Fengyun 1C hit and severely damaged the Russian BLITS satellite, leaving it spinning and in two or more pieces. (3)

In 2009 the still-operational Iridium 33 satellite collided with the defunct Russian Kosmos 2251 satellite at a speed of approximately 42,000 km/h, which is about 12 km/s. This collision created more than a thousand pieces of 10 cm or larger according NASA estimates. (4)

What can be done?

The US Space Surveillance Network currently tracks about 500,000 pieces of space debris, and of these, more than 20,000 are 5-10 cm or larger in size. There are of course many more smaller ones that cannot be tracked, but that still present a considerable danger. Much of the space debris in orbit can be traced back to the Fengyun 1C satellite and the 2009 satellite collision. (5)

If a tracked piece of debris is predicted to approach the International Space Station (ISS) within less than 50 km, then a debris avoidance maneuver will be performed. Source: (6) Up to the end of 2012, 15 ISS debris avoidance maneuvers were performed. (7)

The other part of the solution is of course to clear up the debris before the worst predictions of the Kessler effect becomes reality. Sending "janitor" spacecraft into orbit to capture debris and bring in to the atmosphere is feasible, but costly. Another alternative, more attractive for smaller objects, is to fire a powerful laser at an object, slightly reduce its speed, which will cause it to lower its orbit with a resulting increase in atmospheric drag hurrying its re-entry into the atmosphere. (8). Even more fanciful still is to use a giant net! (9).

References

Welcome to my new blog

2013-11-15T11:46:00+00:00

Welcome to my new blog. All the content of the old blog, which was hosted on Google's Blogger, is now here (except for the comments). The old blog can still be found at oldblog.mcnalu.net until Google see fit to make me remove it, which may be never.

This new blog is powered by Pelican, which I will no doubt describe in a future blog post.

Lending is good and bad

2013-09-09T21:55:00+01:00

I was once told by a person I very much respect that I had an aversion to taking out loans. I suppose in the climate of free loaning (free as in loving, not as in beer) that persisted until about 2007 I did seem loan-averse to most. I thought it imprudent to view the rise in the value of a house that served as a home as a real increase in wealth. I thought it doubly imprudent to remortgage on such an increase. I warned a friend in 2002 not to take out a mortgage for nine times his salary. I didn't like the fact that a new generation of students who came after me viewed it as normal to start out one's working life in debt.

But of course I have taken out loans. I've taken out mortgages for houses. I took out a loan fifteen years ago to buy my first car. Taking out these loans made sense. The mortgage helped me get on the housing ladder in 1995, which in the long run has allowed me to spend a surprisingly small amount to live in a decent home. The car was different - it was a necessity to allow me to get to work and so the loan and depreciation were unavoidable at the time.

But I would have preferred to avoid these loans because if I had the money to buy the house or car outright I'd have saved tens of thousands of pounds over the years.

So now, with the UK's economy still bumping along the bottom, as it has for several years, it might surprise some to hear me say, "yes, borrowing money to spend money is a good way to get the economy going again". The immediate retort of many is to say, "if you're in debt, it's stupid to take out more debt to pay it off." It may or may not be stupid, depending on what you do. Let's take a familiar example of debt - that associated with buying a house.

A person wishes to take out a loan to buy a house to be the family home. They borrow £200,000 at a 5% interest rate and use that to buy the house. This loan is perfectly sensible (aside for possibly inflated house values) because a home is a necessary thing. The person is essentially renting the £200,000 required to purchase the home by paying £10,000 in interest per year.

Some years later they decide to remortgage for the same amount but at a 4% interest rate offered by a different lender. This means they take out the loan from the new lender and the money from that goes to pay off the original loan. This is a clear example in which taking out a new loan to pay off an existing loan is a good idea because now the cost of the £200,000 loan is only £8000 per year.

But let's imagine when it comes time to remortgage that house prices have increased and the house is now worth £250,000. If they decide to remortgage at 4% then they will be left with £50,000 to spend and still be paying £10,000 per year. This seems like a good deal, the cavaet being that if house prices ever drop and they spend their £50,000, say on a luxury car, then they might get stuck in a negative equity trap one day, i.e. their house can't be sold to pay off the £250,000 loan.

But a person who is prudent with money would take that £50,000 and invest it. In that way, they will get a return on that new money and they'll safeguard against the negative equity trap. Let's say this prudent person buys some government bonds (a safe investment which is in fact like lending money to the government) which lock the money up for 5 years but give a return of 5%. This person has increased their mortgage but is making money on the extra £50,000 they borrowed: it was borrowed at 4% and invested again at 5%. Again, another example where borrowing is a sensible course of action.

Now let's consider a different example, a business. Let's say this business has a factory that produces chairs. Suppose each chair costs £50 in material and labour to make and can be sold on for £100, making a profit of £50 on each one. If they make 100 chairs in the month, the profits are £5000. The chairs are good quality and in high demand and they have no trouble selling them and are sure they could sell more. But, there's a problem. The factory's fixed costs are £5000 a month (rent, administration etc) so they're only breaking even and can't save up any money to expand their operation and actually become profitable. The obvious solution is to take out a loan and any bank that's half-way decent would be willing to give them that loan, all else being equal.

But it will only make sense to take out this loan as long as the interest payments it incurs are less than profits of the newly expanded business. So let's say £200,000 is borrowed at 5%, i.e. £10,000 of interest needs to be paid per year. Clearly, this will only be worthwhile if overall profits are £10,000, at the very least, which means they'll have to sell at least 200 more chairs per year, and probably a bit more because the fixed costs will rise. If this investment increased their production to, say, 150 chairs per month, then the business will almost certainly become profitable.

The business example again demonstrates that taking out a loan can be a very prudent thing to do, in the right circumstances. This example is of course very well known and is sometimes referred to as gearing, by analogy with changing the gear to keep an engine working in an efficient range of revs: if the factory drops below its current output it will lose money and "stall".
The common element that exists in both the household loan (mortgage) example and business example is that the loan is only worthwhile when the interest that you need to pay on it is less than the return from investing it.

So in the UK economy, if you can borrow money cheaply at the moment (which the UK certainly can) then it is not a stupid thing to do if it is wisely invested into the productive economy. To put it simply, the UK can borrow now and borrow cheaply because interest rates are low: the Bank of England base rate is 0.5% at the time of writing, and it has been at that rock bottom for years now. It is not hard to envisage that this borrowed money can be invested to bring back a return above that base rate of 0.5%. If done sensibly, it can create jobs, fund affordable houses and more.

There's plenty more to say here, such as on the control of currency, money creation and the issue of inflation, but I'll leave that for another blog post. My main point is that borrowing is an economically sensible strategy in the right circumstances. Also, as a wee corollary, it's surprising how much of the unproductive world of banking starts to make sense once you realise there are profits to be made by borrowing money so you can lend it out again at a higher interest rate.

Firefox OS ZTE phone

2013-08-26T20:58:00+01:00

The other day I received one of the limited number of ZTE Open phones running the relatively new Firefox OS. The specifications of the hardware are limited, especially next to my Nexus 4. But, at £59 compared to the Nexus 4's £290 (and that's cheap for a top-end smart phone), I think it is excellent value. Yes, Firefox OS is a little rough round the edges from a user's point of view, but my first impressions are that it's perfectly usable. And, I have more trust in Mozilla than Google, especially after the NSA/Snowden revelations of recent weeks.

To me, the main attraction of Firefox OS is the fact it is based on HTML5 and CSS and javascript. To those who don't know what that means: those are standards that keep the web open, i.e. you can rely on being able to view and interact with content on a web page on almost any type of PC, tablet or phone, made by any manufacturer, or under an operating system from any vendor. In contrast, mobile phone apps are not like this: android apps work only on devices with Google's android; iOS apps only work on Apple's iThings. But, if you write an app in Firefox OS, it's a simple matter to deploy it as a web app which may be used from any device with a web browser. There are of courses cons against that pro, but that's a discusson for another blog post.
About the time I received my phone, I noticed a message from a well- respected-techy-geeky-sort-of-a-chap, called Jezra, who had also recently acquired the same phone, referring to it as "shitty". Upon asking why, he told me it was because he thought it was hard to get his software on to the phone. Despite my dewy-eyed enthusiasm for this phone driven by open standards, I was soon to discover he had a point.

To start my adventure, I read this article and set about trying to install the Firefox OS Simulator. It's an Add-on to Firefox and should've been a simple matter to install, but I discovered that it's not available for Firefox 17 ESR, which is the default version in slackware 14.0, which I have on my laptop (Patrick Volkerding did provide prompt upgrades to recent Firefoxes, but at 22 he couldn't work around a troubling bug, and decided to go back to 17 ESR for slackware stable). Not wishing to mess with a very stable operating system, I downloaded the linux binary tar ball of Firefox 23.0.1, closed Firefox 17 and ran the newer Firefox binary without installing it - it just sits under my home directory. I was then able to install the Firefox OS simulator version 4.0 as I would any other Add-on, and it seemed to work fine. Soon I had this Hello World app and the Boiler-plate app working without issue.
Next up was the potential "shitty" bit - getting the app onto my phone. Now there are numerous ways of doing this, including via a website, but I wanted to get it done via USB, because that, in my experience, mainly with android, is probably best when developing and debugging an app.

According to the aforementioned article, connecting my phone to my laptop involved creating the file /etc/udev/rules.d/51-android.rules containing the following:
SUBSYSTEM=="usb", ATTR{idVendor}=="19d2", MODE="0666", GROUP="plugdev"

The bit that will vary from phone to phone is idVendor; 19d2 means ZTE. You can look these up online (there's a table somewhere in the android adb documentation), or, you can do what I did under linux, and hunt around in the output of dmesg | tail immediately after plugging the phone in. A restart of udev is recommended after you create the rules file, and the udev folk recommend restarting your system.

Luckily, I happened to have the android adb tool already installed, and from the output of running adb devices, I could see that "roamer2" was listed, which, despite the misleading name, was indeed my ZTE phone.

Now when I fired up the Firefox OS simulator it said "Device connected" and "Push" buttons appeared to the right of my apps. This all looked very encouraging. But, clicking the "Push" button did nothing. Nothing happened in the simulator, not even an error message, and the phone didn't display the connection prompt depicted on the page of instructions (mentioned above). Further down that page, a command line method of pushing apps on to the device was described. OK, I thought, if I try that, even if it doesn't work, at least I'll get some clueful error messages, such is the advantage of the CLI.

The CLI tool is called make-fxos-install, and it needs both the android adb, which I had installed, and xpcshell from Mozilla's xulrunner, which I did not. Luckily, I found xulrunner listed on slackbuilds.org and so within a few seconds I had sbopkg downloading and then building xulrunner.

The source downloaded was 80MB, so I was expecting the compilation to take a while on my rather slow laptop. To pass the time, I decided to play around editing the various demo apps and testing the results out in the Firefox OS simulator. I immediately hit problems. After even the tiniest change to an app's manifest, the simulator kept complaining it couldn't update the manifest cache and something about an NS_ERROR was mentioned amongst other gobbledegook. Frustrated, I decided to launch VirtualBox and try out the simulator under Windows to see if the trouble was caused by some linux-related issue (which I thought very odd and rare). To my astonishment, Virtual box failed to start and also gave an NS_ERROR. I thought for a moment that my dual use of Firefox had screwed up the system. I rebooted the system and found X Windows wouldn't start.

I was just about to hurl myself out the nearest window (OK, I was on the ground floor), when I noticed a little snippet that mentioned a write to /tmp/something had failed. I quickly realised that the disk had filled because of the xulrunner compilation.

I then set xulrunner building on my more powerful desktop machine which has a much larger and less full hard disk. Unfortunately, since I had done this whilst sat at my laptop downstairs, my wife inadvertently shut-down the desktop machine upstairs once our son had gone to bed. Grumble. Once my third attempt at building xulrunner had finished without incident, I scp'd the .txz slackware package over to my laptop and installed it.

I then needed to edit the Makefile that comes with make-fxos-install to specify the path to the xpcshell executable. But it didn't work. After a fair bit of pulling of what little hair I had left, I noticed that the xulrunner version I had built was 15. Guessing this was rather old, on the basis that version numbers track firefox, and unable to face any more compiling, I went to mozilla ftp site and found linux 64bit binaries for xulrunner version 23, which matched the firefox I was using for the simulator. I downloaded the xulrunner-sdk (which is where xpcshell is to be found these days) and stuck it in a directory underneath the Makefile.

But still xpcshell refused to work, complaining about a missing library, but this line in the Makefile got it to play ball:
LD_LIBRARY_PATH=xulrunner-sdk/lib xulrunner-sdk/bin/xpcshell
After a little RTFM I found the command to push the helloworld app to my phone was:

make FOLDER=/home/MyUserName/dev/firefoxos/examples/firefoxos-helloworld packaged install

Did I finally get to rejoice and dance about the room at the sight of this little app installing on my phone? No. Grrrrrr! It complained that it couldn't bind to port 6000. After looking at the Makefile I understood that the android adb tool was trying to forward port 6000 on my laptop to 6000 on my phone. Right. So if local port 6000 was unavailable, for reasons unknown, perhaps I could just tell the Makefile to use 6001 and forward that to port 6000 on the phone, which simply involved a tiny tweak to the thoughtfully constructed Makefile, namely:

PORT_LOCAL = 6001
And now executing the make command above worked! The phone prompted me to accept the connection and the app installed and ran. Phew. But, I wasn't quite satisfied; I had to know what port 6000 was being used for on my system. Executing the command

lsof -i tcp:6000
gave me no output, from which I concluded that no process was actually using the port. But this command

nmap localhost
told me that port 6000 was in some way being claimed by X windows. A bit of web research told me that it was needed for remote sessions, which I never use. To stop it using this port, I shut down X windows and started it again with

startx -- -nolisten tcp

I put PORT_LOCAL back to 6000 in the Makefile and found I could now push to the phone. More than that, I found that the "Push" button in the Firefox OS simulator now worked too.

In summary, to get apps to push to a phone from the FireFox OS simulator, in linux:

use a recent version of Firefox (23.0.1 worked for me)
create the udev rule file
make sure port 6000 is open

and to get it to work from the command line, in addition you need to:

get a recent version of xulrunner sdk which includes xpcshell (23.0.1 worked for me)
get the android SDK which includes the adb tool
edit the Makefile to specify paths to xpcshell and adb, and libraries they need

So, is FireFox OS "shitty"? No, I'd say not. Is FireFox OS currently difficult to push apps to via USB? Yes, definitely. But to be fair, the FireFox OS simulator is clearly marked alpha and some of my difficulties were freak (wife turning off desktop) and others were linux distro specific (port 6000). That said, without drivers for the ZTE Open phone, it's currently impossible to do it on Windows.
I hope the details above prove useful to someone, and if not, I hope you had a geeky chuckle or two at my FireFox OS misadventure. Hopefully I'll soon write some posts about how easy it is (stifles laugh) to write web apps.

Writing

2013-06-13T14:44:00+01:00

I've enjoyed writing since I was a small boy. At school I remember writing Dr Who stories on old computer paper from my Dad's work. I also wrote a series of Goon Show-like parodies of Star Trek which seemed to go down well with folk at my school. Later on writing became a central part of my job when I worked at the Open University, writing science courses. I learned a huge amount doing that, especially from the diligent and patient science editors that would pore over my work, and sit with me, talking me through their comments sentence by sentence.

These editors made sure the writing was clear, correct in grammar and free of typos whilst also taking care not to disturb the style of writing. This surprised me because the text books we wrote involved several authors and I expected part of an editor's job was to pull it together and bring more consistency of style between chapters (writing style, not font sizes etc which was strictly controlled). I distinctly remember how one of them explained his role to me: "We do not want to make the book homogenized. Students generally like a change of tone and change of pace. Besides, you can't edit those things anyway."

It was gruelling at times. I remember being sat in one editor's office as she patiently explained when I should use 'which' rather than 'that', e.g. "This is the train which takes you to Newcastle." versus "This is the train that takes you to Newcastle." Which one is correct, dear reader?*

But, there's no other way to learn other than to write and rewrite and seek reactions from readers, then write something else and revise, and so on. I find it very hard to progress a piece of writing working by myself. Very often it would start well enough but then I would get stuck during the revisions, often abandoning the piece. So I consciously decided to write my first drafts with more care, rationalising that if I could do that, then I'd have less revising to do. But this made matters worse. My writing now slowed to a snail's pace ~~crawl~~, was thoroughly unenjoyable and the turgid result could not be rectified by any amount of disciplined editing because the style and tone were compromised. Had I stopped to think about it back then, I might have realised that that's exactly what the editor at the OU had meant.

So, for many years I hardly wrote at all. It was even a welcome relief when the rise of mobile phones and twitter meant that emails became shorter and more sloppy: I could spend less time trying to write them well, or so I thought for a while.

But I missed writing. I wanted to return to it, even though it seemed like a futile waste of my time. I had ideas that needed expressed, but repressed them because I couldn't think how to get them out. After writing nothing for years, the urge strengthened sufficiently for me to bang out a few short stories, but I didn't feel happy enough with them to let others read them. And I was slow. One story took me more than two years to complete.

I remember sitting in my car outside a supermarket listening to BBC Radio 4. The author Frederick Forsyth was being questioned about how he wrote. To my surprise he typically wrote six pages a day, but stressed that it was very important to get into the discipline of writing every day. That didn't seem like a lot to me. What did he do for the rest of the day? Presumably gathering the experiences and knowledge that gave him something to write about.

I investigated the habits of other authors and I found many did something similar. One author, I forget who, had a strict rule of writing only two pages per day and never any more. But, as Forsyth had also emphasised, even at this rate you can yield a novel-length work in a few months.

The details of how authors work varies tremendously, but I'm not aware of one that advocates binge writing. Little and often yet disciplined and regular was the recurring advice. Approaches to revising vary greatly, but most recommended that you should not aim for perfection on a first draft.

Then I happened across NanoWriMo.org, which encourages people to produce 50,000 words from scratch in the month of November (continuing an existing work is not allowed). At first I was skeptical. Isn't this just going to produce 50,000 words of drivel and probably verbose drivel at that, so as to produce the requisite 1667 words per day?

The answer is sometimes, perhaps even in most cases. But that's missing the point. What it achieves is to encourage a person to write with discipline and work towards a goal. In my case it helped prove to myself that even with two children, a day job and many other calls on my time, I could get those 1667 words out almost every day.

The quality of my writing varied tremendously of course, with some days being awful, most days being OK and a few being good. But I realised it didn't matter. As long as I was relaxed when I wrote so that my writing was natural rather than forced, what I produced was generally acceptable in that it expressed my ideas well enough so that someone else might understand them. It took me another year to get the word count to 80,000 and since then seven months have elapsed during which I have performed two editing passes over the whole thing. It's still not finished. On the second editing pass I decided to add several new sections and remove a few that weren't working. Once I've done that, I hope, it'll be ready to be read by someone else.

It's certainly the case that my writing has improved over this time and with it I'm probably being more economical with words, though I still struggle with repetition (She did that... she did this) and banishing superfluous words, especially adverbs (He just stood there, impassively). I'm also somewhat blind to typos and rely on others to help me find them. But I'm realising that staying relaxed and writing a little too much and editing out is better, for me at least, than trying to write precisely under conscious invigiliation. Also, most readers, at least ones who are not authors, will skate over pointless adverbs and other minor flaws as long they want to know what happens next.

If I had to summarise it in one pithy phrase, then I'd settle for "be as simple as possible, but no simpler".
You can read some of my writing here at http://www.mcnalu.net/writing

*That one. :)

Faster USB sticks

2013-01-03T16:49:00+00:00

I've been trying to set up my Linutop 2 box as a small server for a while now, but keep hitting a variety of annoying problems. The latest was that an intenso 32GB USB stick seemed to be the cause of achingly slow performance.
I read in Linux Format recently that it was wise to make sure partitions and filesystems were aligned with the structures found on the flash drives. Flash devices come set up in this way, but invariably with archaic FAT32 filesystems and I wanted to use a more up-to-date linux filesystem. I didn't know how to ensure alignment, so I embarked on a course of reading across the internet.
Unfortunately the information I found was generally very confusing, some articles told me what to do, but not why, and others went into bamboozling levels of gritty detail, and one or two even told me aligning wasn't worth the effort on USB2.0 drives. I'll provide links of the sites I found most helpful at the end, but here's my account of what I did, starting with the USB stick formatted in a completely misaligned way with three partitions - the result of my fettling before I knew any better.

First I plugged the drive into my laptop, then, as root:

mount /dev/sdb3 /mnt/hd

I tested the read speed with:

hdparm -t /dev/sdb

and this told me the speed was 23.8 MB/s - I did a few tests and found this varied by no more that +/- 0.05 MB/s. I then mounted the third partition, which had an ext2 filesystem on it, with:

mount -o sync /dev/sdb3 /mnt/hd

The sync option means that the writing isn't buffered - if it is buffered then the test will give you a misleadingly high speed because it will only include the time take to write to the buffer in memory. I then tested the write speed with:

dd count=100 bs=1M if=/dev/zero of=/mnt/hd/test

This copies one hundred 1 megabyte blocks of zeroes to a file called test. The result was a paltry 893 kB/s.
I then set about deciding how I could reformat the drive to get better performance, particularly for writing. I wish I'd kept a note of the original partitioning of the stick as that would give me vital clues about how to align partitions and filesystems. After searching the web, the best I could do was guess that the USB stick was likely to have a 4kB or 8kB page size and a 512kB erase block size.

To be honest, I haven't really had time to get to grips with the details, so please excuse all the uncertainty, but I'll try and set out my patchy understanding. The erase block size is significant because if the filesystem requires, say, 100kB to change, then the flash drive electronics will have to wipe at least a 512kB block and rewrite it all. I say "at least" because if that 100kB happens to cross the boundary between two adjacent 512kB blocks then both blocks will have to be erased. These erase blocks are grouped together to form erase block groups which, I understand, are typically 4 MB in size. This means that partitions boundaries should be on multiples of 4MB on the device.

The page size represents the smallest unit that can be addressed on the flash drive. I'm not sure what impact this has, except that clearly you don't want the partition or filesystem constantly straddling the "pages". But, if you're using a block size of 4kB in an ext filesystem and you've aligned the partition to 4MB there should not be a problem. With modern multi-GB drives 4kB will usually be the default block size used by the mkfs command and it's also the maximum permissible block size on typical systems.

From this you can get some idea of how aligning data with these blocks can help with performance. Also, because it reduces the number of writes to the disk, alignment can also help increase the life of a flash drive. One big problem in aligning is that there is no easy way to find out the page or erase block size for a particular stick because such information is often not published by manufacturers. The best you can do is look at the original factory formatting of the stick or search around on the internet to see if someone has measured it. (There are tools to do it yourself, and I tried flashbench which was intriguing but unfortunately it gave inconclusive results for my USB stick.)

I side stepped the issue of partition alignment by not creating any partitions at all (zero is a multiple of all numbers!) In linux it is quite permissible to create a file system directly on the device. I did this with the command:

mkfs.ext4 /dev/sdb -b 4096 -E stride=128,stripe- width=128

This creates an ext4 filesystem on device sdb (sda is the hard drive of my laptop). The -b 4096 specifies the use of a 4kB block size (I had tried 8192 but that is possible only on a few specialised systems. Next, I tried to make sure the filesystem was likely to be aligned with 512kB blocks by setting the stride and stripe-width both to 128 because 128 times 4kB is 512kB. These two parameters are more often used to make sure filesystems work efficiently with RAID arrays, but as we're only dealing with one drive here I understand that they should be set to be equal to each other (though I confess I don't know why).

Finally, to save on disk writes, I turned the journal off:

tune2fs -O ^has_journal /dev/sdb

The downside of this is that the disk will be more likely to suffer from data loss if powered down or removed incorrectly. (This option can also be set when creating the filesystem I believe so it should be possible to combine the above two commands.)

With this done, I performed the above tests again and found that the read speed was almost the same at 22.8 MB/s. The write speed was now 4.7 MB/s - a five-fold increase. Job well done! That wasn't all, if I repeated the test, the write speed seemed to go up reaching a little over 10 MB/s after 4 runs of the test. I'm not sure why this might be, but it could be to do with some optimisation in the onboard electronics in the flash drive combined with the artifice of the test in writing long stretches of zeroes.
But, I'm not quite comparing like with like here. In the "before" tests I was writing to an ext2 partition, so is it possible that some of the speed-up is due to inefficiencies of ext2 compared to ext4? To test this I created an ext2 partition directly on the disk with this command:

mkfs.ext2 /dev/sdb -b 4096

(The stripe and stride width options are not available for ext2.) The first thing that struck was how slow this was compared to the ext4 creation - 30 minutes compared to a few minutes. The tests revealed that performance was back to almost exactly what I found on the original ext2 partition with its completely misaligned partitions and filesystems.

Finally, I wanted to see if it was ext4 that was behind the speed gain, or if it was the setting of the stripe and stride option that brought about the improvement. To test for this I issued this command:

mkfs.ext4 /dev/sdb -b 4096

and did not disable the journal. The write speed was now 3.6 MB/s which seems to suggest that ext4 is responsible for most of the speed gain. The continued flashing of the stick's LED after the test had ended suggested that the journal was keeping the stick busy at least to some extent and this was further confirmed when I umounted the stick and the flashing ceased (this ruled out the activity as the flash doing wear-leveling maintenance. Interestingly, there was a slightly improvement in read speed, but no more than a few percent.
So, in conclusion, the most important improvement seemed to be just to use ext4 instead of ext2. Disabling the journal helped a little too, as did setting the stripe and stride options on ext4. But, I didn't find any evidence that alignment helped improve the performance of this drive.

Here are some links:

Good starting point: http://linux-howto-guide.blogspot.co.uk/2009/10/increase- usb-flash-drive-write-speed.html
A very thorough and detailed technical account: http://lwn.net/Articles/428584/
Somewhere between the above too in technical detail - some good diagrams: http://blog.nuclex-games.com/2009/12/aligning-an-ssd-on- linux/

Reading Keynes part 3

2012-12-26T19:27:00+00:00

In my previous post in this series I described some thoughts that came from reading book III of Keynes's General Theory. Before moving on to discussing subsequent books, I'd like to make some notes regarding the simple model I discussed in that post.
The first note regards how a lack of consumer spending (in the absence of saving and investment) could drive an economy to disaster. A step in the argument was that employees who feared for their jobs will cut their consumer spending. This is plausible, but there are really two types of such spending. The one that can be most readily cut is the spending on luxuries, but spending on essentials, such as food, clothing, rent and repairs cannot be cut below a certain level. This enforced minimal level of consumption can stop a descent into complete economic collapse, but the word 'disaster' is still applicable as this bottomed-out economy will doubtless have high unemployment and a class of the employed that are living on the breadline.

My second note on the last post concerns the statement at the end on how effectively the 1% can syphon away the wealth of the 99%. This jars with the fact that the 1% own the companies that pay the wage bill of the 99%, and that they are therefore reliant on the consumers (the 99%) recycling the money back to them by spending and/or saving. I think the answer here is two- fold. A minority of companies do successfully syphon money to themselves (e.g. Apple) but it's at the expense of their competitors (e.g. Microsoft) not their customers. But I was really meaning sums of money summed up across the whole economy, and I believe that aggregate syphoning from consumers isn't significant, and that it can actually occur towards the poorer 99%. However, it is clear that the 1% start off with disproportionate ownership of wealth, including the capital to generate and control it, and they use that control to make sure they continue to own their unfair share.

Book IV of The General Theory is entitled The inducement to invest and its first of eight chapters is called The marginal efficiency of capital. In very loose terms, the marginal efficiency of capital is to capital what interest is to money (despite the popular misconception, capital is not synonymous with money). If we give money to another party, we can expect them to pay us a sum of money over time - the interest - for allowing them to hold and use the money. Likewise, if we invest in purchasing some capital equipment, say machines for making mobile phones in a factory, then we will expect our ownership of those machines to bring us a net income over time from the sale of produced goods. Keynes makes a rigorous definition of the marginal efficiency of capital so he can go on to compare such income with interest rates. The main conceptual difference between the two is that for the latter the owner of the money is not directly concerned with how the asset (i.e. the money) is used to generate a return.

Much of the subsequent chapters are concerned with discussing the interplay between the marginal efficiency of capital and interest rates. If interest rates are low then there is a greater incentive to invest in purchasing capital assets and expect a return from them, and vice versa for high interest rates. But, of course, it is not so simple. There are psychological factors at work. How do people perceive the future? If there is optimism that consumer spending will remain strong, then investment in capital will be more attractive. But if not, people may have a preference for liquidity and be less inclined to lock up money in more difficult to release capital equipment, even if interest rates are not very high.

Keynes discusses what sets money apart from other repositories of wealth and makes the point that interest is not actually unique to money. One example that occurred to me was carrots. A crop of carrots is perishable, unlike money, but a given number of carrots can be planted and grown - that is "invested" - so that year- on-year, assuming you didn't sell them, you would add to the total number of carrots in your possession (even though these wouldn't be the same carrots). You could quite legitimately call the extra number of carrots each year your interest. If you instead, as most farmers do, sold those carrots to receive a return in terms of money then that return would feed into your calculation of your marginal efficiency of capital (along with the initial stock of carrots, land, machinery etc).

The carrot example illustrates a couple of points, most obviously that money is preferential to carrots for holding wealth because it doesn't necessarily incur land, labour and storage costs. But it also demonstrates what Keynes calls "own- interest", i.e. you can define the interest rate of carrots in terms of carrots. This helps explain what he means in this key quote from Chapter 17 Section III:

No further increase in the rate of investment is possible when the greatest amongst the own-rates of own-interest of all available assets is equal to the greatest amongst the marginal efficiencies of all assets, measured in terms of the asset whose own-rate of own-interest is greatest.

He argues that the thing with the greatest own-rate of own-interest is money. So, in plainer English, he is saying that people will put their wealth into a form that they see as most likely to increase their wealth. The convoluted digression into interest defined in terms of other assets (such as carrots) strikes me as possibly a distracting abstraction, but it shows the rigour that Keynes wished to apply to his arguments.

To return to the key point, the halting of investment caused by a high money rate of interest can (and usually does) stop an economy proceeding to full employment. To put it more starkly: even if companies are wanting to find a way to increase profits, and even if willing and able unemployed workers are wanting employment, the economy might not be able to respond because of a blockage caused by a shortage of the very thing that is supposed to facilitate the economy: money. Keynes highlights this absurdity in his delightfully idiosyncratic style:

Unemployment develops...because people want the moon;- men cannot be employed when the object of desire (i.e. money) is something which cannot be produced and the demand for which cannot readily be choked off. There is no remedy but to persuade the public that green cheese is practically the same thing and to have a green cheese factory (i.e. a central bank) under public control.

He goes on to highlight the folly of attempting to anchor the value of money against a rare substance extracted from the Earth:

It is interesting to notice that the characteristic which has been traditionally supposed to render gold especially suitable for use as the standard of value, namely, its inelasticity of supply, turns out to be precisely the characteristic which is at the bottom of the trouble.

In other words, if you tie money to something that appears to have tangible value, then you actually make the situation worse because you lose all control over the supply of money to the economy and are then completely helpless in dealing with a money shortage.

But, putting gold standards, interest rates and capital aside, the problem is very simple: we have become too preoccupied with money itself, to the extent it distracts us from the essentials of the production it is supposed to facilitate, namely, securing food, shelter, employment and health.

Fesenjan

2012-11-18T22:35:00+00:00

Fesenjun - pestle & mortar and saffron to the left.

When I was a wee boy, probably about the age my son is now (7), I remember being confronted by a dish that my aunt had made at her home in London. "That's mud, isn't it?" I said, without meaning to be funny, but everyone laughed and they still laugh about it. The mud was an ancient dish from Iran called Fesenjan. It was by far the most delicious mud I had ever tasted.

I don't recall eating Fesenjan again until a few years ago when I spotted it on the menu of an excellent restaurant called Paradise in Glasgow. That was very different from the one my aunt had made, much more sweet and sour, so much so that I imagine that it would over- challenge too many people's palettes. Sadly it probably did, as it doesn't appear on their menu any more.

Some months back I started cooking recipes I found in a Persian cookbook my wife had bought on a trip to California (where there are many Iranians). At first I didn't spot the recipe for Fesenjan, but when I did I had a go at cooking it. The first try was good, but not quite as dramatic and intense as I had wanted, so I tried again and ended up with the recipe I described below. I've cooked it for quite a few friends now and it seems to go down very well and it really is quite different from anything they've tried before.

Fesenjan is really made distinctive by the combination of walnuts and pomegranate. It is most commonly served with chicken, but duck, beef or vegetables can all be substituted. I don't recall what my aunt put in it, but there were no large lumps of meat, so I wonder if she used minced beef or even lamb. I intend to experiment with that for the true muddy consistency.

Fesejan - serves 6

Ingredients

2 finely sliced medium onions
1kg of chicken, legs and thighs preferred
1 litre of pomegranate juice
300g of walnuts, ground in food processor
1 tsp saffron, ground in a pestle and mortar
1 butternut squash cut into 1 inch cubes
1 tbsp of muscavado sugar
juice of one lime or lemon
some pomegranate seeds and some unground walnuts for garnish

Method

In a large deep-sided pot, fry the onion in a few tablespoons of olive oil, and add the chicken and butternut squash after a few minutes. After a few more minutes of stir frying, turn the heat down and cover it while you prepare the sauce.
Put the ground walnuts in a bowl with the pomegranate juice and lime or lemon juice and sugar.
Dissolve the saffron in a tablespoon or two of hot water (as much as it will dissolve) and pour into the bowl and mix it.
Pour the contents of the bowl into the pan with the onion, squash and chicken.
Cook on a low heat, uncovered for 2 hours. Stir every 15 minutes or so to avoid the walnuts burning. The sauce should reduce and thicken quite a bit, but if you like it a bit more runny and less intense in flavour, you can put a lid on it for some of the time.
Garnish with pomegranate seeds and unground walnuts and serve with rice.

Reading Keynes part 2

2012-11-14T21:04:00+00:00

I've now completed reading book III of The General Theory by John Maynard Keynes. I previously wrote about my thoughts on books I and II. The title of book III is "The propensity to consume" and it is composed of three chapters.

Probably the most important statement made in this part is in Chapter 3, section II, paragraph 2:
Thus, to justify any given amount of employment there must be an amount of current investment sufficient to absorb the excess of total output over what the community chooses to consume when employment is at a given level. Out of context the meaning may not be clear, so I'll try and explain in my own words, using a very simple model of my own devising.

Consider a simple, closed economy which comprises a number of companies and a community of people - the consumers - who will buy products from those companies. The economy is closed in the sense that it does no trade with any other companies or consumers elsewhere, i.e. there are no imports or exports. Let's also suppose there is no taxation or spending by a government.

So each one of the consumers are either employed by a company or else they are unemployed and have zero income (the government doesn't spend, so there is no welfare state). The ones who are employed receive a monthly wage from their company. Let's add up all the wages across this community and call it W. Also, each consumer will spend a certain amount each month. Let's add up all that is spent and call it C.

Each company will have an income, and if we add up the income from selling to consumers across all companies we must find that it comes to C - the total the consumers have spent. One company can sell to another company and so we can add up the business-to-business income across all companies and call it B.

The costs of each company will be the amount it pays out in wages plus the amount it spends on buying from other companies. Adding up wages across all companies must give us W. Because this is a closed economy, the total spent on companies buying products must be B.

So the total income of all companies is C+B and their total costs are W+B. The total profit across all companies is therefore C-W (the Bs cancel out).

And you may have spotted the problem with this simple economy. People will generally not spend more than they earn (and even if they do, they can not do so indefinitely). This means that the total amount spent on buying products from companies C will have to be less than the total wage bill W, or, in other words, the companies in this economy must make an overall loss of W-C. Of course, there will be winners and losers - profitable companies and loss-making companies - but overall companies are loss making in this economy.

A company that finds it is making a loss will try to drive down its costs. It can do this by reducing what it pays to other companies, and it will probably also reduce its wage bill, either by cutting wages or by firing employees. The overall result will of course be to drive down B and W, though B has no impact on the overall profitability of companies. But, as we argued above, C cannot be greater than W, so if W is decreased, C will decrease too. In fact, employees who are starting to fear for their jobs are likely to start saving and so the total amount they spend C will probably fall by even more than W and so the overall loss W-C may even increase. In any event though, W will remain greater than C.
This effect is an example of positive feedback in that the fact that W is greater than C will cause companies to act in a way that will cause W-C to increase. In this way, without any other stabilising factors, the economy will collapse: all companies will go bust and everyone will become unemployed.

And this is where the quote from Keynes above becomes relevant. To stop this happening, the companies, as a whole, need to receive monies from another source to offset the loss they collectively make, i.e. W-C. This extra source of monies Keynes calls "investment". And where does it come from? Well, in this simple economy it can only come from what the consumers don't spend on goods. Instead they either use their savings to buy shares in a company or save the money by putting it in banks that then invest or lend the money to the companies. And, of course, this excess of consumers income over spending is exactly equal to W-C, which means this economy can be made stable in the sense that companies can break even overall and so the level of employment can be sustained.
The essential point that Keynes is making here is that investments and savings in an economy are necessarily equal. I had previously only thought of investment as being necessary for companies that want to grow or establish themselves as start-ups. Although that's true, I can now see that continued investment is a necessity to sustain a stable economy because of the gap between C and W.

The other point which Keynes makes is that an economy that has reached a stable equilibrium may have done so at less than full employment. It may well be, and in fact generally is the case, that the willingness to invest and the propensity to consume are not sufficient to motivate companies to increase their production and take on more employees. The unemployed who have zero income (there is no welfare state in my model) would clearly prefer to work, but the companies will not employ them because they cannot see how this would maintain or increase their profits. This conclusion, that an economy can be stable at less than full employment, distinguishes Keynes's theory from that of classical economics (at least of his own time, but I bet there are still many who do not get this point today).

There are a number of other interesting implications that can be drawn from this simple model. One is to consider what happens if hoarding rather than saving occurs in the economy. For example, an old miser who distrusts banks, stuffs all excess money under a mattress. Likewise, a bank that takes deposits but refuses to lend is doing much the same thing. The result in both cases is that money enters a stagnant pool and the economy as a whole will suffer because it is deprived of investment. If money could be released from that stagnant pool, then employment could rise because companies could access a new potential source of revenue. Of course, in my simple model there is no government to spend, but in reality, government spending can achieve the same result in raising employment.

But, if there's one lesson to take away from this simple model, it is that "one person's spending is another's income", or more specifically, "consumer spending is company revenue". Companies as a whole are completely beholden to their consumers. If consumers do not part with their money to buy products, then it is their savings transformed into investments which keeps the companies in business. This raises an interesting question: how does this fit with the current situation in which the richest 1% of the population, who presumably own companies, are often supposed to be syphoning wealth from the 99%? I'm still pondering this question, but will save my attempts at an answer for a future blog post.

Reading Keynes part 1

2012-10-16T21:37:00+01:00

It was my fortieth birthday recently and I was glad to be presented with many books. Amongst them, courtesy of my father, were The Wealth of Nations by Adam Smith and The General Theory of Employment, Interest and Money by John Maynard Keynes. I decided to start reading the latter book first. This is the first in a series of posts in which I'll try and explain what I've got out of the experience.
First off, I'm not an economist by training, but the problems with economies around the world in recent years have kindled my interest, especially when I began to appreciate that so-called "Keynesian" solutions, as described by the likes of Paul Krugman, seemed to not only stand up to skeptical, rational scrutiny but also had a very solid basis in historical precedent. Before going further, the very first thing I learned is that Keynes is pronounced canes, not keens.
Not being an economist of the early 20th century makes reading Keynes's classic book a bit of a challenge. My first stumble was when I encountered the word "disutility". I tried and failed to understand it from the context and a search of the web kept leading back to extracts from the book itself. It turned out that "utility" is a term in economics used to quantify human satisfaction and that the word "disutility" was either peculiar to Keynes himself, or else is now arcane. The term appears in the context of Keynes disputing the received economist wisdom of his time that workers settle for a real- wage (i.e. in real terms, not in terms of an amount of money) that is just sufficient to make their employed state of greater utility (or lesser disutility) than their unemployed state. If you are struggling with that last sentence, then you have a sense of what it feels like to read Keynes's book.

The point is valid though. An employee will immediately seek redress with their employer if forced to take a pay cut, i.e. the money-wage is reduced, but may not even notice if the real-wage is reduced, say, because prices of everyday items have increased. Even if noticed, the employer is unlikely to see rebargaining the wage as their problem; they too may be faced with increased business costs due to increased prices. This situation is playing out just now across Europe: in Spain and Greece people are understandably furious at having to take money-wage cuts (or else be made unemployed) to solve economic problems that were not of their creation. Meanwhile, outside the land of the Euro, in Iceland, most people implicitly accepted real-wage cuts when the Icelandic currency was devalued.

The next piece of received wisdom at which Keynes takes aim is the notion that "Supply creates its own demand". That strikes me as immediately strange, but nevertheless there are still people who believe it today. One implication of it is that there should be no involuntary unemployment, on the basis that everyone who wishes to work can do, because when they produce something, there will be a demand from someone, somewhere to buy it. This seems so obviously absurd that you may wonder why it would need refuted, either in 1936 by Keynes, or by anyone today. Of course, the absurdity of it is not often as plainly stated as this, but it is implicit in much economic wrong-thinking. In fact, as Keynes sets out, if you believe in "supply creates its own demand" or one of many equivalent variants of that statement then you are actually adopting an economic model that assumes there is no real barrier to full employment. In boom times that might not be so bad a model, but during a depression, like the current one, it's simply the wrong model to adopt. To use an analogy: if a car fails to start, no amount of tinkering with the engine will help if you've incorrectly assumed that it has sufficient fuel in the tank.

But, that said, after only having read books I and II of Keynes' classic text, I have not yet encountered anything else that you would recognise as "Keynesian", e.g. that governments should spend their way out of a recession. Instead, Keynes has merely sketched his ideas in broad brush and then painstakingly set out a series of definitions of net income, investment, saving and something called "user cost" (akin to depreciation, except that no cost is incurred if equipment is idle) so that he can go on to describe his theory in subsequent books. In other words, he found the economic descriptions of his own time too vague and imprecise to formulate his theory and so invented his own.

I am not one who is prone to hero-worship and I certainly do not worship Keynes nor accept his ideas without question. But it is clear Keynes was a very intelligent person and, more importantly, an independent-minded individual. Don't take my word for it - have read (or listen) to what philospher John Gray had to say about him.

Lucentis - a poke in the eye

2012-10-04T22:07:00+01:00

Yesterday I received my first eye injection to treat the condition that has developed with the retina in my left eye. I can't say I relished the prospect of someone sticking a needle in my eye and squirting liquid into it, but then again, I was nowhere near as aghast or horrified as the people I told about it - one person said "don't tell my husband, he'll throw up on you!".
No doubt, the ghoulish reader will want me to get straight to the bit where I describe the crazed doctor, laughing maniacally, stabbing me in the eye causing me to convulse and scream in agony. Well, there you go, I've just done it. Non-ghoulish readers, read on...

In the week since I'd been diagnosed, the distortion had spread across my retina so that it now affects the centre of my vision. This perturbed me enough so that I returned to the hospital and successfully (and politely) argued with the medics to get the treatment brought forward by a week. If the damage was likely to be permanent, it was paramount to get treatment as soon as I could.
I arrived at the eye clinic at 8.45am on a Tuesday morning for my injection, but was kept waiting for an hour. It's beyond me how you can be an hour behind at the start of the day, but the lumbering, administration-heavy machine that is the NHS is like that. That said, in my experiences, and as this story will demonstrate, the care that machine facilitates, to every citizen of the UK, regardless of income or means, is very definitely not broken.

My fellow eye-patients probably had an average age of well above seventy, and these veterans of the eye clinic, and life in general, sat there looking relaxed and possibly slightly bored. I have to marvel at the stoicism of elderly Glaswegians. If these old codgers and codgerinas could stick it, so could I.

The substance to be injected into my eye is called Ranibizumab, though it is more commonly known by its brand name Lucentis. It is derived from an antibody found in mice and inhibits the growth of blood vessels which may leak fluid into the eye, causing the edema (swelling) that is distorting my retina. Apparently Lucentis costs over $1000 per dose, whereas there exist alternatives priced at $40 per dose that are claimed to be as effective. I suspect massive profiteering is going on here and plan to investigate further.

Anyway, after 45 minutes in the large, main waiting room, and 15 minutes sat in a chair in a corridor, I was called into the small room. There was a reclining, padded chair, much like one you'd find in a dentist's surgery, a table and a cabinet stuffed with packets of medical equipment. It was soon apparent who the doctor was, as he swept around the room, talking in a clear, confident voice, not a hint of doubt coming into anything he said. There was also a specialist nurse and he too seemed sure of what he was doing, but spoke less and was more garbled. And there was another nurse who, after than showing me into the room, seemed to be waiting quietly in the corner.

I lay down on the chair and it whirred and reclined me. I asked the doctor a few questions about my condition and he gave me clear, matter-of-fact answers. Apparently, some people thought eating leafy greens helped this condition and, he said, it was always good to be trying things that made you think you were helping yourself. After I said I wasn't a smoker, he spoke a bit about how bad smoking was for macular edema and health in general.

Through all of this conversation, he and the specialist nurse were buzzing about me, swabbing my eye, preparing the eye and giving me a few anesthetic drops. I think this went on for about ten minutes and the chat not only informed me but helped keep me distracted. Since I wasn't wearing my glasses I couldn't really see much, but I was thinking "is the needle coming now?" They placed something like a large plaster over my eye and peeled a layer away to reveal a transparent layer. I think this was to hold my eye in place with the eyelid clamped back, but I was already doing my level best to keep my eyes still. I suspected the moment was coming when the nurse took my hand and began to squeeze and stroke it gently. Simultaneously, the doctor who was hovering around behind me and to my left began to move something towards my eye. Being so myopic I couldn't see it, which was probably a good thing, and then there was a strange sensation - not pain, more like a dull ache. A giant floater appeared in my eye and I mentioned it. The doctor apologised, saying that a wee bit of air had got in, but it should be harmless and would sink to the top of my vision when I stood up (the image on the retina is inverted, the brain reverts for you).

So that was it. I really couldn't fault the way it was done: friendly, professional and caring. If I was a macho man, I might have rejected the nurse's hand-holding, but I'm not and it helped. She was doing more than a job, she was caring about her job and the human before her. I could easily imagine a bean-counter, administrator saying "we don't need two nurses in the Lucentis clinic", but if they did, someone with more sense prevailed.

For the rest of the day, it felt like there was something in my eye and so I was blinking a lot and it was very teary. I suspect that the "something" in my eye was a little rough spot on my eyeball where the needle went in and that was irritating my eyelid. The eye was a little red, but other than that and a slightly dull ache around my eye, I experienced no other untoward symptoms.

The distortion in my left eye has got no better or worse in the last week or so, but it is quite possible that it stabilised by itself before the injection. My brain is doing a better job at compensating: unless I consciously look for it, the distortion isn't apparent to me most of the time. My binocular vision is still a little compromised as my brain struggles to reconcile the different images coming from my left and right eyes, but I think it's improving too. I try to give it a helping hand by increasing font sizes on the computer, for which the zoom function in my web browser (firefox) comes in very handy. So far, so good.

But, I'm remarkably lucky. Does every human on Earth get such treatment? Would I have got it 100 years ago in this country, without the NHS? No. I'm very, very grateful and I hope that one day everyone can expect at least this level of heath care.

Real retinal resolution

2012-09-08T18:43:00+01:00

Sometimes you notice a small thing that turns out to have much larger implications. So it happened last week. I noticed that the lines along the top of a spreadsheet seemed to bend, just slightly and only for a fraction of a second as I shifted my gaze - but I saw it. I thought at first it was some problem with my monitor, but when I saw it happen on another computer, I knew it was me and not the machine that was faulty.

I am very short sighted and this results in the retina being stretched and made very thin and prone to spontaneous damage. The retina is the surface at the back of your eye that has many receptors that detect light. Each receptor feeds pulses to your optic nerve that allow the brain to construct an image that you see, well, that you think you see (it's a long story). It didn't take me long to realise that the retina in my left eye was faulty and I wasted no time in making an appointment at the optician. This was Friday.

On Saturday morning I saw the optometrist and she was extremely diligent and tested my eyes and photographed my retina but could find nothing wrong. She then arranged to have a 3D scan performed to see if the damage was below the retinal surface, but that too was inconclusive. We both agreed that whatever had happened to my eye, it was not deteriorating, so she arranged an appointment at the hospital at 10am on Monday, but warned me to go to A&E if the condition changed over the weekend.

On Monday I spent the day having the same tests again and was given a thorough examination by a young doctor. She too was unable to identify the problem either by direct examination or from the results of any of the scans. She called in the consultant and he spotted a small hint on one of the scans and sent me off for a dye test. This test involves the injection of a yellow dye (fluoroscein) into the veins so that it will travel through your whole body; any disturbed or exposed blood vessels in the retina will give off a strong glow. I was warned that the dye would give me the appearance of a Homer Simpson, but it didn't give me more than a mild tan, though it did turn my urine a spectacular fluorescent yellow.
The dye-test showed the problem - I had a macular edema. The macula is the part of the eye that is most densely covered in receptors and it is what we rely on for detail in the centre of our vision. We rely on it for reading and perceiving fine detail in facial recognition. Edema means swelling - in my case a membrane at the back of my retina had ruptured causing liquid to enter an area just below the retinal surface. The consequence of this is that receptors in my eye have been re-arranged and possibly damaged.

The consultant explained all this to me and to be honest I was not at all alarmed. I already knew that my retina was damaged and, if anything, was reassured to have the details of it recognised and explained to me by an expert. I guessed that the damage was permanent but also guessed that, in time, my brain would be able to cope with this disruption, perhaps even learn the new locations of the remaining and undamaged receptors. The consultant confirmed this to me. He also assured me that there was a viable treatment that could prevent further deterioration, though this involved a series of injections into the eyeball and that the eye would need close attention for years to come.

An idea of the view through my left eye if I'm looking at the word "just".

For now, I need to live with a distortion in my left eye. In some circumstances, e.g. when outside or just looking around a room or driving, it is barely noticeable. At most I'll notice a brief kink in a window frame or a bend in a lamp post, but only fleetingly. If I use only my left (damaged) eye to read, then I can focus on a word with little distortion, but two or three words to the right do appear distorted, though legible. Using both eyes together is better, but I can feel the extra strain on my brain and eyes of having to reconcile two different images. But, in time, I'm sure my brain will rewire itself.

What I see when I blink, with added starship.

The distortion itself is always in the same place, a little to right and above my centre of vision. It is shaped like a flower and resembles five or so overlapping circles. Inside this shape my vision is distorted. I was pleased the other day that when I placed the Moon inside it (which was actually quite tricky) that I could still see it, although it was not round, nor could I see much in the way of features on it. The size of distortion is about half the area of my clenched fist held at arms length, so approximately 5 degrees across its longest part. If I close my eyes for a few seconds and then open them I can clearly see the outline of it, but just for a moment. If I blink rapidly I can see it as a blob, much like the blob left after you have viewed a bright object such as the Sun. Sometimes it reminds me of a poor Star Trek special effect, minus the Enterprise.

I can live with this level of distortion, but I do fear it deteriorating and spreading to cloud my entire vision in that eye. If, as is possible, a problem develops with my right eye then my visual acuity could deteriorate to the point where I can no longer read, but I won't go completely blind. Although the cause is different, the effects are not unlike the macular degeneration experienced in older people. I have to face up to the fact that significant loss of my eyesight is likely to occur as I age and that what some people face in their late seventies or eighties has started (just) before I reach 40.

This prospect is causing me some anxiety and I can feel a black cloud hovering behind me, dampening my ambitions and tempting me to retreat, dwell and introspect. To give into that leads to depression, so my aim is to reaffirm my ambitions and keep going regardless. I will take the treatment on offer, I will have faith that although my eyesight is failing, that it will do so in a stable way and that, at least for now, my brain can adapt. And, if things get so bad that brain cannot compensate then I will turn to computers and technology for help and rely more on my ears, for example by using audio books. But at the same time I must acknowledge this new limitation.

I feel fortunate that my interests are varied enough that the limitation will not dampen my enthusiasm for life. I am even more fortunate that I have a loving and supportive family around me and some very good friends. In fact, it has already occurred to me that limitation is not always a bad thing - I am prone to trying to do too much at once and so frustrate my ambitions in any one endeavour. Also, wherever I go, and wherever I look, I am now always accompanied by a colourful little amoeba-like flower when I blink, and, for reasons I don't quite understand myself, I actually feel some affection for the little thing.

Owncloud

2012-08-06T21:59:00+01:00

Go into the "cloud", use the "cloud", harness the power of the "cloud", "cloud" computing, "cloud" this, "cloud" that... blah blah blah. Of course, the word has some meaning, but it has been so abused, misused and used to confuse that I now find it irksome. But there are more tangible problems with the ideas behind it. (I'll stop using "scare" quotes now.)

The obvious problem with storing your data in the cloud is that you don't know where it is or who can see it. Another problem is that if you use software that is in the cloud, for example gmail, then you have zero control over changes to that software. If google decides to update gmail, you will be forced to switch to the new version at some point whether you want to or not.
I could live - and do live - with these and other drawbacks of the cloud, but there is one other problem that troubles me more. I rely on google for mail, contacts, calendar, documents, web sites, my phone, this blog... Although I have no particular problem with Google at the moment, it does not sit well with me to have so many eggs in one monopolist's basket.

So when I first came across ownCloud, I was immediately attracted to the idea. ownCloud is software you can install on your own webserver and you can control how your data is stored and accessed and since the software is Free and Open Source (under the AGPL license), you have as much control over it as you could wish for.

Of course, not everyone has the time, inclination, knowledge or money to run their own web server, but then perhaps there is a web-skilled friend you might trust, or you could choose to use a commercial provider who is offering a pre-installed set up for you. The project has a company (owncloud.com) that lives in symbiosis with the open source community (owncloud.org).

So far I've tried out the file sharing, music, calendar, contacts and gallery aspects of owncloud and experimented accessing it from my android phone. My server is in my house running on a low-power linutop 2. It's not fast, but fast enough for my purposes just now. I will write another post with more technical aspects.

Once I've finished a bit more fettling, perhaps I will invite some friends into my ownCloud, but will they trust me with their precious data?

Raspberry pi - beta lego

2012-07-01T22:28:00+01:00

My RPi now has a new home - a lego case. I tried to convince my 7 year old son to build it, but he felt playing minecraft and ace of spaces on the computer was more important, so I rummaged through his lego collection and constructed what you see below.

OK, so it's not going to win any prizes, but it does have windows. No, not microsoft windows, but windows through which you can see its LEDs. Just look at the cosy glow from its rear window. (I've decided that the ethernet cable gets plugged up its arse and its power goes in its mouth.)

I've also installed a new operating system, the beta of Debian wheezy for the RPi, and it seems to work just fine. I've fed back one minor problem I had (to do with he config tool) to the developer via github and I hope to contribute a bit more in the future.

My next task was to set up ssh access, which was easily done with this command:

sudo service ssh start and to make sure this happens after every boot, I issued this command:

sudo insserv ssh
Of course, this isn't much use if you don't know the IP number, so to set a static IP I just had to edit /etc/network.interfaces to read:

auto lo

iface lo inet loopback iface eth0 inet static address 192.168.1.23 netmask 255.255.255.0 gateway 192.168.1.1

When I'm ssh'ing into it, I don't type in 192.168.1.23 every time, instead I gave this IP the identity of "summerston" on my other computers (editing /etc/hosts on linux) in keeping with my policy of naming my computers after defunct Glasgow train stations (Summerston train station is actually open, but the original station from which it gets its name, but which was several miles away, has been closed for about forty years).

The last thing I set up was a VNC server so I could access the GUI desktop from my laptop, this was just two commands. The first installed the necessary software:

sudo apt-get install tightvncserver and the second started the server:
vncserver :1 I was then able to access the desktop from my laptop using KDE's VNC client called KRDC and also my phone using a VNC app. It looked just like this:

Raspberry pi - first steps

2012-06-27T22:06:00+01:00

After waiting a couple of months, my raspberry pi finally arrived the other day. For those of you that don't know, it is a ludicrously cheap ($35 or £25) computer that is sold as a bare board with a few connectors on it. In contrast to the out-of-the-box shininess that is now the raging fashion, this wee beastie is minimal and challenges you to learn something about it, and about computers in general. In fact, that's exactly why the raspberry pi came into being.

The tiny thing on top of the standard PC case in this photo is my raspberry pi. There are only two cables plugged in here: the ethernet cable at the back and the power cable (standard USB phone charging cable) at the front. You can also just make out the SD card which is slotted in the same side as the power cable. This serves as its disk drive. My first task after unboxing my new, little friend was to write the operating system to the SD card. This was simple enough, I downloaded the Debian linux image from the raspberry pi website, verified the file's integrity by its sha1 sum using this command

sha1sum debian6-19-04-2012.img

stuck the SD in my laptop (running slackware linux 13.37) and ran dd bs=1M if=debian6-19-04-2012.img of=/dev/sdb (first checking that the SD card was /dev/sdb - much badness can occur if you get that wrong!). I then put the card in raspberry pi, plugged it into my TV using the HDMI cable, plugged in a USB wireless keyboard and mouse and then... drum roll... plugged it into the power and... more drum roll... some LEDs flashed but nothing appeared on the TV. It turned out that the TV had to be on and set to HDMI before I turned on the pi and once I did that... yet another drum roll... it worked! Text scrolled by and the little raspberry pi logo sat at the top left of the screen, looking cute and somehow content.

I was then able to start X windows with the LXDE desktop by issuing the command 'startx' and soon I had a giant logo emblazoned on my 32" TV. After some fiddling around loading the sound module I got the sound working and was able to listen to the short demo piece of music that comes with the music player. The software to get the pi's hardware doing its stuff is still in development and the sound is apparently quite buggy at present. My next task will probably be to get beta testing the next version of the Debian OS for the pi.

So, what am I going to do with it? I don't yet know exactly what I'll do with it, except probably what any child would do with a new toy: play with it.

Science and family

2012-04-28T08:15:00+01:00

Last Saturday I helped out a science fair at my son's school called Science Rocks. It was a fantastic event and I thoroughly enjoyed it and it's clear that it entertained, educated and inspired all who attended. So much so, that I spent my Sunday afternoon searching a nearby park (Dawsholm Park) for insects with my son and friends who were all fired up as entomologists.

I had the pleasure of giving planetarium shows using Glasgow University's inflatable Starlab planetarium. I was ably assisted by son who operated its controls - no mean feat for a 7 year old surrounded by people, lots of noise and in the dark.

I'd like to share something with you which has been bouncing around in my mind this last week. Almost 50 years ago, a young woman came over from Iran. She spoke very little English but was determined to spend a few years in the UK to further her career in nuclear physics. She not only ended up with a PhD, but also found a husband, settled in Bearsden, Glasgow and secured a research job working on Glasgow University's linear accelerator. When she became pregnant she was forced to stop working around radiation and went on early maternity leave to have her baby. That baby was me. (Yes, the radiation exposure probably does answer a lot of your questions about me.)

Unfortunately, the concept of maternity leave was not well established back then, especially in the very male-dominated profession of academic science and so my mother was not able to resume her scientific career, but she did, after much perseverance, manage to return to work at Glasgow University in computing. Although disappointed in this, she was a fantastic and devoted mother and gave my sister and I the best start in life one could wish for. I know she was pleased that my sister and I went on to have careers in science and engineering and particularly that my sister was able to have two children without much detriment to her career.

My mother died a year and a day before the science fair, and that has been on my mind quite a bit this week. But I'm absolutely sure she would've loved it and been delighted to see such a great science fair being run by a group of mothers, with a little help from a few dads.

C++, SDL and oPong

2011-12-30T08:23:00+00:00

During the peaceful days between Christmas and New Year, I set myself the task of learning how to use the low-level graphics library SDL. My interest was piqued by playing a few games that have used SDL to good effect to cross platforms, especially to work smoothly on linux distributions.

SDL is written in C, which I know fairly well, but because most games and so online tutorials are now in C++ I decided to use that, even though I'm less familiar with it. Since I'm using a full install of Slackware Linux 13.1 (which is fairly old now, but works fine) on my fast desktop machine, I was pleased to find that GNU compiler g++ was already installed, along with SDL and all the other tools such as make and text and graphic editors that I needed to get started.

After a bit of surfing I found myself following this excellent set of tutorials from Lazy Foo - and did them dutifully - I read each one, downloaded the source code, compiled it and ran it and sometimes tweaked it to try out other ideas. I found them easy to read and the code compiled and worked flawlessly each time. Much kudos to Lazy Foo for the effort s/he put in there (despite claiming to be lazy). I also gained a good boost from this tutorial for C++ for C programmers.

So, what is oPong? Well, I decided that after reaching Lazy Foo's lesson 18 that I could try out creating an old game - pong - but with a new twist. In my version, there would be only one bat and it would move around a circle. Also, every time you hit the ball you get a point, but the bat also gets shorter, making the game increasingly more difficult as you progress. I spent almost zero time on how it looks, as you can see by this screenshot:

I found the process of writing it quite straightforward, with two stumbles along the way. I was surprised at just how low-level SDL was - it didn't have any functions for drawing circles or lines. All SDL can do is fill rectangles and draw sprites (i.e. graphics loaded from image files ). The upside is that SDL is fast and gives you fine control over the graphics.

The second problem I encountered was with handling the angles, which caused a hard to diagnose bug where the collision routine failed and allowed the ball to pass through the bat at its left most position. This was caused by the fact that the angles flipped from -PI to PI at this position. I remedied the problem with some extra code that carefully handled the difference in angles.
I'm not sure what, if anything, I may do next with SDL, but I enjoyed getting to grips with it and C++. Also, I am pleased that fairly low-level graphics access is still readily available on our increasingly complex computer systems. Clearly, I draw pleasure from some odd things.

You can download the source code for oPong from here.

Birth, death and ongoing life

2011-09-04T12:16:00+01:00

It's been a while since I blogged here and as the dramatic title may suggest, that's not because I've had a shortage of life events to write about; I was busy dealing with them all.

My mother died in April and although her health had been troubled for some time none of us expected it might be fatal. So it was fairly sudden which is shocking, but preferable to the slow, lingering and degenerative exit that my wife's grandmother is enduring along with close family. I miss my mother very much, but the memories are now starting to be fond and comforting instead of just painful.

We welcomed my sister's second child into the world in late January and my mother met him the week before she died. She also knew that in July my wife and I were expecting a daughter. The new life arrived, almost two weeks late at the end of July and, forgive me for gushing, she is just beautiful and a joy to have around. This is our second child, our son being well into his seventh year. This time we are much more relaxed as parents and perhaps that's why our daughter seems content. That said, our son was a fairly content baby too, so I suppose there is nature, nuture and luck involved.

Late spring and summer coincided with a particularly troubled time at work, not just for me, but for some friends too. Life sometimes seems to throw everything at you at once, but in a way that can be a good thing. For me it helped keep things in relative perspective. The important thing is to press on and, as a local, Kipling-esque expression goes, keep the head.

Old phone, new life - donut to froyo

2011-02-26T22:14:00+00:00

Desire Z and G1 - both on Android 2.2 "froyo"

I got my G1 phone (also known as an HTC Dream) in early 2009, a few months after it first came out. I recently replaced it with an HTC Desire Z (known as the G2 in the US) which I think is great. But, I really liked my old G1 and was a little sad to let it sit in the corner gathering dust. But, after 2 years, surely it was time for the G1 to retire, I mean, it can't keep up with Android Desires and iPhone 4s?

The truth is that HTC decided not to take the operating system on the G1 beyond Android 1.6 "donut". But could its admittedly meagre CPU clock rate or limited internal memory let it run Android 2.2 "froyo" in a way that in any way compared to my new phone?

To find out I decided to root the phone (gain complete and low level access to it) and install something called the Cyanogen mod. This is a build of the operating system made possible by the fact that Android is an open source project. I've heard people claim Android isn't properly open source, but I think that's just plain wrong. The google apps (gmail, calendar, maps etc) that usually come with Android are not open source, but they're not part of the operating system; they can easily be replaced.

The installation of the Cyanogen mod is not that straightforward, but it doesn't require anything truly hardcore, such as modifying the hardware or editing source code. Have a read of my detailed account if you'd like to know more. My experience was made more troublesome because my phone was locked to T-Mobile, which means that it will only accept a T-Mobile SIM card. This meant that the phone refused to do anything without a fair bit of hacking right at the start, which I found very frustrating.

Anyway, after a bit of a fight I got it installed and I was very impressed. My G1 is noticeably snappier running 2.2 than it was with 1.6 and, although not as fast and smooth as my Desire Z, it's not that far off. The browser is quite zippy, the pinch zoom works (perhaps with a slight stutter) and google maps work fine - it's all quite usable. More than that, it looks better too and even has window effects and animations (if you're into that kind of thing).

In fact, I'm left wondering why my Desire Z isn't much faster than it is. That could be because the Sense UI that HTC put on it is inefficient, but I suspect it's the other way around: the specs of our phones haven't increased as much as marketing and anecdotal "look how cool my shiny new phone is" comments have lead us to believe, instead any gains in hardware specs are swallowed up by new features to "sell" the product. Some features are useful, but others like pointless apps and stupidly high screen resolutions are not.

Well, my gratitude to Mr Cyanogen and his merry band and to the open source-ness of Android. I was amused to see that the little robot widget on my G1's new home screen sprouted a speech bubble that contained the few commands needed to extract the latest version of Android from a code repository and build it from source. One day I'll get around to building the Android source code myself.

On the sleeper

2011-02-22T18:50:00+00:00

I've found myself travelling around the UK quite a bit this last week, taking in three of my favourite haunts: Glasgow, London and Cambridge.

The majority of the travelling was on the sleeper train. The sleeper train departs Glasgow a little before midnight, when central station is huge, hollow, smooth and clean. You walk along platform 10 and are greeted by a conductor (I'm not sure if that's the right title, perhaps carriage concierge is more appropriate) who checks your ticket and who records your preference for tea or coffee and cooked or continental breakfast. I’m not sure why they ask you. I always get both tea and coffee, and even if I ask for continental, I get handed a hot packet with MEAT written on the label.

The carriages are definitely in the British Rail slam-door era, circa 1990. It's a bit like a working museum in some ways. Perhaps I give the impression that I don't like the sleeper. Nothing could be further from the truth, I love it. It’s clean and functional but is most definitely not “shiny”. In the toilets, I'm delighted to find that the foot button approach to flushing toilets and operating taps is still employed.

And, possibly my favourite bit: there's a restaurant car. Well, it's more like a dimly lit bar car. Again, it's not decked out in the latest fashions - it's all a bit early 1990s - but I like that. The stewards are endearingly grumpy. One tall, bald-headed Glaswegian steward entertained me upon one of my first trips on the sleeper by explaining how he was hit on the head with a brick upon entering a public house in Kilmarnock. "How delightful!" I chirped as I clasped my hands together excitedly, "and what of the other scars upon your face, what stories do they hold?" OK, I didn’t say that; he might have thrown me through the window.
On my most recent trip I was genuinely pleased to spend a good half an hour talking to a lovely old lady whom I knew from my student days at Glasgow University Observatory. She's 80 this year, but full of energy and was charging down to London to exhibit a collection of original letters and photographs sent between various famous astronomers of yester-century. I handled each carefully mounted and protected A4 wallet with great care as she handed them to me over our table of wine, sandwiches and Laphroaig. Marvellous.

She told me her habit was to finish her refreshments and retire when the train reached Carstairs, where (I think) the train gets shunted together with the Edinburgh sleeper, rather clusmily. I didn’t have the staying power of this near-octogenarian, so bade her goodnight before Carstairs.

I made my way down the narrow corridors past the row of small cabins. It’s so narrow that you have to angle your body and walk slightly sideways. But, as I was walking down the corridor a hand reached out from a cabin doorway and grabbed me, pulled me in and threw me to the floor. It was the henchmen of my evil nemesis, Professor Moriarty. We struggled for a while until I bested one by trapping him inside a folding bunk. The other I managed to eject through the window after electrocuting his metal teeth with a broken table lamp bulb. I then climbed into bed with Eva Marie Saint. I may be getting this bit confused with a few films. Hmmm, hard to be sure.

If you get fed up faffing around at security at airports or are bothered by short haul air travel's pollution and wastefulness, or just fancy something different, give the sleeper a go. And do so soon before they make it all intolerably shiny like everything else these days.

Magnatune.com

2011-01-01T18:26:00+00:00

I was very pleased to get a subscription to magnatune.com as a Christmas from my wife. Magnatune is a bit like fair trade for musicians in that 50% of its revenue goes to its musicians and their terms and conditions are equitable and open. For example, magnatunes does not demand an exclusive deal from its musicians.

While you read this blog post, why not hit play on the little embedded player below and have a listen to an album I recently enjoyed from Magnatune?
Between Worlds by LYNX and Janover

It's not a competitor for iTunes though (but that didn't stop Apple rejecting the Magnatune app for a while) and as such don't expect to find any Lady Gaga amongst its 10,000 tracks. It does have a very wide variety of music that is of high quality in terms of recording and musicianship; this is to the credit of the site's founder John Buckman. Apparently he sifts through a large number of submissions and rejects more than 95% of them. At first this sounds like a weakness, but after listening to a dozen or so albums, I'm none-the-wiser as to what John Buckman's taste in music must be, except possibly that it must be very broad.

You don't need to subscribe to listen to the music, you can access it all for free via the website or via a variety of offline players including apps for both Android phones and iPhones. I mostly listen via an application called Amarok which is part of the KDE desktop environment. Magnatune donates some of its 50% to help with the development of such open source or free (FOSS) applications.
Before getting my subscription I enjoyed the messages that played between the songs. One of them went somthing like "Hi, I'm John Buckman's 76 year old neighbour. I don't much like most of the music on this site but you probably do because you're listening to it. Why not consider paying $15 for a subscription...". In addition to freeing you from such announcements, the subscription also permits you to download as many albums as you wish with zero DRM and even to share each album with three of your friends.

I applaud the good (i.e. not evil) folk behind Magnatune.com and wish them every success.

Cause vs correlation

2010-12-18T11:10:00+00:00

This morning I was delighted to come across this story about a startling correlation between the number of mobile phone masts in an area and the birth rates. Given the talk about mobile phone signals and health (for which there is no good evidence), even I was tempted into thinking "oooh, mobile phone masts get people horny or more fertile".

But of course, it's actually quite obvious when you consider what data is being analysed. Lots of mobile phone masts in an area just means that lots of people live in that area and lots of people means lots of births. Hence there's a correlation between the number of masts in an area and the number of births.
Looking at the data, a more interesting analysis would be to take the births per thousand people and correlate that with the number of masts per thousand people in each area. My guess is that there might be a much weaker correlation, but it may still show some interesting effects. For example, is the birth rate higher in areas with greater population density in which more phone masts are erected? Is there more NIMBYism against phone masts in prosperous areas with lower housing density?

This housing phenomenon must be an age-old one and mobile phone masts are just a new proxy for it. I recently researched the history of the defunct mining village of Mavis Valley and in the 1901 census data (which, alas, isn't freely available on the web), I found that 49% of the village's population was under 15. At first I thought "wow, compare that with our aging population these days", but then I realised that the percentages for my own street - about 20 three and four bedroom houses built in 1995 - had a very similar percentage of under 15s. Sure enough, when I checked, I found that Mavis Valley had undergone a major expansion in the decade before the census and so young couples had moved into the new houses and started breeding.

A magnet for numerology and pseudo-science (i.e. nonsense) is the sunspot number time series, which I spent three years of my life studying. Your eye and brain might be attracted by the graph in this example: But look more closely and critically and ask yourself these three questions:

How many graphs did the author compare with sunspot number before choosing to blog about this one?
A time shift of 3 years was needed to expose the correlation and no data before 1947 is included - is there a reason for these choices other than to improve the correlation?
There are 10 distinct unemployment peaks but only 4 of them line up with peaks in sunspot number, the other 6 don't. Are you convinced by a sample of 10 data points of which 6 do not agree with hypothesis proposed?

Knowing how to ask those questions, and to think critically and as objectively as possible, and at the same time be aware of unavoidable subjective choices and influences, are skills that a good scientific training brings. This is the aspect of science that is almost always lost in the media and this is exactly why the article in today's Guardian delighted me.

It's been a bit cold

2010-12-12T16:43:00+00:00

My garden in the winter of 2009/2010

Last winter was unusually cold and snowy in Glasgow, UK. Already this winter seems to be even colder.

In the last two weeks we had two significant snow falls and temperatures plummeted to -15oC, staying below -10oC for almost two days.

Many countries are used to this kind of weather, but this is quite exceptional in Glasgow and so no-one - authorities nor residents - knows quite how to handle it.

Have a look at this graph of temperatures from Glasgow airport:

(View the spreadsheet of data originally from wunderground.com.)

To see just how exceptional these temperatures are, consider the mean min and max temperatures for Glasgow in December: 2.2oC and 7.0oC respectively. Compare these with Oxford's min and max December temperatures of 2.1oC and 7.4oC.

Glasgow is about 4 degrees north of Oxford in latitude and so it should show lower winter temperatures. In reality it doesn't because Glasgow benefits more from being on the receiving end of heat brought across the Atlantic by the Gulf Stream.

Apart from the obvious problems of clearing snow and ice, such low temperatures present completely new issues. At about -15oC diesel significantly thickens as the paraffin and other constituents of the fuel solidify. This blocks the fuel filter and causes the engine to lose power or fail to start, as happened in some parts of the country.
Salt also becomes less effective at preventing ice formation at lower temperatures and I saw that several major roads within the city that were certainly treated with grit (rock salt) were not free from ice. In fact, at about -21oC, no amount of salt can prevent water ice from forming.

The other issue that was seen during last year's winter and that is still evident a year on is the damage caused when water repeatedly freezes and thaws in cracks and cavities in the road. This happens because water expands when it freezes, which breaks concrete, rock, cement, asphalt and tarmac. The results are huge potholes and sunken manholes on almost every road.

I do tire of hearing people complain that our political leaders should have been better prepared under such extreme conditions. The resignation of Scotland's transport minister seemed to be more about poor media handling on his part, skillful (but ultimately pointless) attacks from the opposition and some bad luck in exactly when the snow fell (peak of rush hour on a Monday rush hour). Perhaps he did make some mistakes, but I can't tell what they were after reviewing the media coverage. On the upside, I'm no fan of this SNP government so if this speeds their exit, I will not complain.

But, all this raises the question: after two winters of unusually severe weather, should we (that is the government and individuals) now prepare for such eventiualities every winter? I'm starting to think so and I think I'll see if I can find some explanation for why our mild Scottish winter climate might be changing.

Welcome to mcnalu's musings!

2010-12-12T08:41:00+00:00

(This post was meant to the be the first on this blog, but I accidentally published the "wondering" one first.)
I'm not entirely sure what will appear in this blog, but it will probably be a variety of posts on subjects that have amused or interested me to the extent that I feel the need to express myself. Hmmm, that's just a definition of blog I suppose. Rest assured that future posts will be better than this one. My previous attempts at blogging, along with other stuff I've done, can be found at mcnalu.net.

Wondering about wonders of the night sky

2010-11-27T11:46:00+00:00

I said to my 5 year old son a few clear nights ago, "Get your coat and shoes on and come and see Jupiter through the telescope." This was met with some enthusiasm.

It was a cold, clear night and Jupiter and the Moon were to the south and I set the small scope up on my driveway with my son prancing around me happily. I found the Moon first so I could align the finderscope and then after a little fiddling with screws, I called him to have a look. He continued prancing for another minute or so and then came over.
"Oooh! That's very bright" he said. The moon is very bright through even a small telescope.

Then I swiveled it onto Jupiter. I could see the disk, faint banding and three of the four Galilean moons. I then called my son over again and he looked through the eyepiece.

"But that's rubbish!" he said.

Clearly he expected this but got this He listened quietly to my attempt at explaining what he was seeing, but seemed unconvinced, but I hope my few words lodged somewhere in his mind.

The next night I was doing something similar with a telescope with an evening class at Glasgow University observatory. There was much less prancing around. They were very impressed by the Moon, pleased and interested at Jupiter but disappointed by the smudge that was the Andromeda galaxy. Of course, when you explain that the smudge is a galaxy of several hundred billion stars and that the light they just saw left those stars over 2 million years ago, then that smudge takes on a new significance.

We were also treated to a meteor, a couple of satellites and for a moment we were all perplexed by a strange, flickering orange "star" drifting across the sky to the north. After a bit of theorising, we concluded it was a chinese lantern, which we then confirmed with binoculars and a fleeting glimpse as it zipped across the telescope's field of view.

The night sky is beautiful to the naked eye, but it has an even deeper beauty that can only be appreciated with your mind.