Half SQL: semi-opaque tables reduce developer effort

Your relational data design could steal a trick from NoSQL: database representation of your application objects doesn’t need to be broken down to primitive values. Some opacity of relational data is beneficial.

Relational databases impose rigid structure on the data: data is organized in tables, which are made up of columns. Traditionally the columns could only be of primitive types (numeric or alpha), although opaque types (BLOB, CLOB) and XML are now universally available. The rigid structure makes it easy to reason about data, to guarantee data quality in the database layer and to build tooling on top of the database. Querying based on values of individual columns is well defined and highly efficient. Downstream usage (reporting, analysis) becomes easier. Relational structure works well when you need referential integrity across entities that are not updated together.

But relational databases have downsides. One of the downsides is the notorious difficulty of changing the database structure. Martin Fowler wrote: “A lot of effort in application development is tied up in working with relational databases. Although Object/ Relational Mapping frameworks have eased the load, the database is still a significant source of developer hours”. Guy Harrison blogging for Tech Republic: “Change management is a big headache for large production RDBMS. Even minor changes to the data model of an RDBMS have to be carefully managed and may necessitate downtime or reduced service levels“. There is “impedance mismatch”  between RDBMS and application layer: “Application developers have been frustrated with the impedance mismatch between the relational data structures and the in-memory data structures of the application”. Even a change confined to a single table (e.g. adding a column) requires significant effort and synchronizing rollout of database and application layers.

The frustration with the amount of developers’ effort that the relational databases required was one of the drivers behind the rise of NoSQL starting about a decade ago. NoSQL databases differ from the relational in many other ways and switching to NoSQL is not an easy step. Fortunately, you can solve some of your problems by using just one item from the NoSQL bag of tricks. You can greatly reduce the impact of single-table changes (such as adding a new column, the most frequent type of change) by making your table definition semi-opaque. Don’t throw away your old and trusty RDBMS. Turn it into a Half SQL data store: in each table, select a small number of key columns that may be used in indexes and keep them. Hide all other fields from RDBMS by placing them into an opaque container field of a BLOB type. As a simplified example, Orders table may look like this:


Your application will be responsible for serializing and deserializing those blobs. Adding a new object field will be invisible to the RDBMS. When you need to add a new field, you will only need to change the code in the serializer/deserializer. And if you use a good serialization library (if your application is written in Java, please, don’t use built in serialization; there are many libraries that are faster and more flexible), even those changes in most cases will be NOOP because your library will take care of those automatically. No data migration will be needed. You will be able to write test to verify that your logic works before and after the change. And you retain the RDBMS goodness of referential integrity and super-fast queries over the indexed columns.

Stashing all “other” object fields into a BLOB column could save you quite a bit of effort.

The Mandibles – a story of a human impact of government finance

This review was posted at Amazon.

Lionel Shriver’s The Mandibles is a dystopia focused on financial issues, but you won’t find Wall Street or government characters in it. It shows how overly simple – and popular! – government decisions ruin the lives of the regular people. It makes an argument that federal government debt is a serious threat to our society and an attempt to get out of debt easily by renouncing it would be a disaster, collapsing the economy and plunging the whole country into dire poverty. In the ensuing economic collapse, shortages of all goods rival what is reported out of modern-day Venezuela. Homelessness is massive. The chilling perspective surely provides some food for thought. The book is well written, with likable characters and vivid scenes.

There are some decidedly Randian notes in the book: the spirited defense of free markets, extended dialogue on economics, cartoonishly negative and arguably mean-spirited depiction of socialists. There is even something like a Galt’s Gulch that I liked better than the original. Rand’s Galt’s Gulch was a utopia and the weakest part of Atlas Shrugged, in part because utopias are generally boring. Shriver’s version is decidedly bleak and thus more realistic. And the whole book is written mostly from a “little man” perspective, which was severely lacking from Rand’s writings.

The book covers 2 distinct time periods: first two thirds of the book are set in the late 2020s – early 2030s, when US is spiraling into collapse. The events of the last third of the book occur in 2047, when the situation is not quite as dire on the level of basic necessities, but federal government reasserted itself and became an oppressive monster. Punitive taxation inhibits economic activity. Government controls every little transaction, down to a parent giving her child a small allowance. Life becomes even bleaker than in the first part because there seems to be no future to hope for. Yet both parts of the book are united by an uplifting thread: spirit and resourcefulness of two persons, a young and an old, finding answers to the toughest challenges.

I found the first part of the book plausible. The events in the book are set off by a decision of a large group of countries to introduce a common currency backed by hard assets. This particular event may be unrealistic, but it doesn’t really matter. Anything could serve as black swan and the book would work with only minor changes if the triggering event was different.

The second part of the book appeared to me to be more schematic and less persuasive. The 2047 period echoes the key element of Vernor Vinge’s Rainbows End: government insisting on inserting itself into everything people do through electronic chips they own and control. While Vinge focuses on technological aspects of this arrangement, Shriver’s book is mostly about routine daily transactions. Interestingly, Vinge expected strict government chipping rules to be flouted and chips hacked, and his society, while not clearly described, was broadly functional. In the Shriver’s book, government control is inescapable and the resulting society hopeless. On this, it’s a bit hard for me to believe Shriver. There is another example of the similar absolutism in the book that sounds dubious. US state of Nevada breaks away from the totalitarian USA. The landlocked territory named USN badly need trade because it is too small to manufacture all the different goods available in the modern economy. US government prohibits any relations with the enemy territory and advertises dire consequences for seditious noncompliance, but doesn’t physically enforce the border. And in the book a spirited smuggling economy does NOT emerge! This makes no sense: while US people are under the spell of government propaganda, the Nevadans know that the border is non-existent and have all the incentives to make trade work. The lack of smuggling is decidedly strange for a book that extolls free markets. Black market is relatively free because it is remarkably difficult to regulate: government doesn’t do that and various non-government strongmen can only do so much. Smuggling and black markets have always existed where governments attempted to restrict trade – just look to Prohibition and the War on Drugs as examples.

The author also makes an astute observation that in a severe depression our entitlements system will lead to a bizarre phenomena where elderly would be the only people with meaningful income. They would become the anchors of their families. Woe to a family in which no one draws a Social Security for they will not know where their next meal will come from. This picture, while dystopian, is not a figment of author’s imagination: it would be familiar to people from former Soviet bloc who lived through internal conflicts and USSR breakup wars. Those government retirement checks were meager indeed, but amid civil wars and economic collapse there were often no jobs to be had.

What’s wrong with The Giver: Sameness and communism don’t work

I’ve recently had a conversation with a couple of high schoolers about The Giver (a popular middle school reading assignment and a 2014 movie). The story is a dystopia set in an apparently post-apocalyptic world. The key feature of this world is Sameness: there are no seasons and no weather, all people live in identical housing and so on. Population of this world doesn’t have memories of the past, and their present is free of any strong emotions. They lack color perception and don’t know love, but there is no warfare, strife or hunger.

The world is supposed to come off unsympathetically dystopian, but it’s not all that cut and dry. Upon reading the book and seeing the movie I wondered how many people would be willing to take such a deal. And lo, the high schoolers I talked to expressed an opinion that the world of The Giver may be a nice place, especially for the older people who no longer care for romance and just want to live out the rest of their lives in peace and comfort.

Of course, it wouldn’t work this way. Seniors would be the worst off, actually, because in The Giver’s world, the authorities simply kill off (“release”) seniors.

For the others, the deal in The Giver is also not as good as it seems. We learn about the world through the eyes of Jonas, a teenager (12 years old in the book, but 16 years old in the movie). He is young and naive and doesn’t yet know much about his own world. He is shocked to learn that a routine act of “release” from community applied to elders, unexpected babies and sometimes adults is not a benign exile, but an execution. So it’s not as nice as it seemed at first: powers that be won’t send you to a war, but they will still kill you. If you’re lucky, you will live to advanced age, but what happens if you become unproductive due to an accident? This is hardly a utopia.

In addition, we learn from Jonas’ mother, who is a judge, that their community applies a whole specter of punishments, “release” naturally being the ultimate penalty. She relates that she sentenced someone to release. This fact alone breaks the beautiful picture of a harmonious society. The way their small community (a hamlet, really), “releases” people left and right, it might as well be called a death camp. And the world doesn’t look as perfect once you realize that people still routinely break rules and are punished for that. Is there really no strife in that world or any miscreants are swiftly executed before situation gets out of hand?

OK, so the society was totalitarian, but did they at least live comfortable lives with all their basic material needs taken care of? No way! History tells us that humans need incentives. Imposing Sameness removes material rewards as incentives. With punishment being the only incentive, a society can’t survive. The largest scale experiment of this kind, the USSR, collapsed much sooner than it reached anywhere near the level of control and uniformity depicted in the book. It was beset by deficits throughout entirety of its existence.

People tried to create utopia many times. Not a single one succeeded. Ask Governor Bradford of Plymouth Plantation how well communism worked even with seemingly highly motivated group of people who had no one to rely one rather than themselves. Answer: not well at all. Once individual incentives are removed and the fruit of labor is split between all members of the community, people start finding excuses to avoid work. In his writing, Governor Bradford relates how pilgrims, who arrived to settle a colony in the New World, attempted to do all agricultural work together and share the crops fairly and evenly. The result was a disaster that nearly wiped out the colony: people were unwilling to put much effort into tough farming work, even though their collective survival depended on it. Seeing this, next year they decided to parcel out land and let each family work its own lot. The quick end to the communist experiment saved the colony. Governor Bradford describes the improvement in these words:

The women now went willingly into the field, and took their little ones with them to set corn; which before would allege weakness and inability; whom to have compelled would have been thought great tyranny and oppression.

Sameness may be superficially attractive and communist ideal will probably always resonate, but it is a false calling, doomed to leave an impressive body count and ultimately fail every time it is tried. This is one of the most important lessons we could learn from all of human history.



Bradford, William. Of Plymouth Plantation, 1620–1647. Edited by Samuel Eliot Morison. The University of Chicago Press. http://press-pubs.uchicago.edu/founders/documents/v1ch16s1.html

Machine Learning and Artificial Intelligence

There is a popular line of thinking about Artificial Intelligence (AI) that says that Machine Learning is the pathway to AI. This idea gained momentum partially because Machine Learning is the hot topic (and buzzword of the decade), but also because we understand ML fairly well. With ML in hand, it is easier to make machines that can learn and become smart, while making machine smart out of the box is a much more difficult task.

This approach makes sense. After all, the only intelligence that we know – human – is learned, not built. In other words, even if we were to invent Asimov-style positronic brains, robots will still need a period of learning after being fully assembled. Of course robots will need to learn a whole lot faster than humans – nobody wants a robot that takes 18 years to become functional. Fortunately, robots can learn much faster than humans. But to learn fast, robots need a lot of input to process. It means that they will have to learn from shared information (as opposed to the personal experiences alone). To give a hypothetical example, a robot would become a good driver much faster if it can learn not just from its own driving history, like all humans do, but from the experience of all the other robot drivers. Robots will need to remain in constant communication with each other (whether centralized or decentralized) in order to keep up with all the changes in the world around them and get better at whatever they’re doing.

Now, how will ML help us achieve AI? By itself, ML is just machine code capable of syntactic manipulation based on certain statistical rules, which is not intelligence of any kind, not even a weak AI. Will it lead a computer system to developing intelligence? The Chinese Room argument says it’s impossible to jump from syntactic rules to semantic understanding, so there is still a big phase transition to be made before AI becomes a reality. Patience, please.

Wer wartet mit Besonnenheit

Der wird belohnt zur rechten Zeit

(Roughly: Whoever waits patiently, will be rewarded at the right time. © Rammstein).

Disclaimer: opinions expressed in this post are strictly my own – not of my employer.

Please NO: Apple patents camera-blocking technology.

Apple patented technology that would allow blocking the use of cameras in iPhones wherever it is deployed. Who is it made for? Certainly not consumers! How many iPhone buyers ever said: “I wish they disabled my camera here”?

In a disingenuous propaganda move, Apple described  this technology as being able to “…stop smartphone cameras being used at concerts”. Right. It’s about concerts. It can’t possibly be meant to be used by corporate or government security. And I’m sure no police department would think of wiring this device to a squad car flashers, to prevent bystanders from filming police work. It’s strictly about concerts. Nothing to see and record here, move along.

Humans in advanced technological societies will be increasingly religious

Religiosity is heritable and thus must be underpinned by genes. What is religiosity and why would it be a valuable trait that is passed from parents to their children? Religiosity is, to a large degree, a wiliness to accept things on faith and to obey rules and commands. This trait becomes valuable in a complex society that surrounds people with incomprehensible dangers. A toddler can’t comprehend electricity, but accepting and obeying the rule of not messing with electric outlet improves her life expectancy. A little later, learning not to run into traffic becomes valuable. There are numerous dangers that humans must learn to recognize and avoid, acting purely on faith, without personally examining them. Arguably, the more complex a society’s technology, the more dangers are lurking everywhere – dangers that need to be avoided simply on someone’s say-so. Those who do that will have better chances of survival. Call them religious people – and we will see how a highly technological society could also be highly religious.

United’s poor “multi-factor authentication”

United Airlines (united.com) recently “upgraded” their Web site security. They sensibly discontinued 4-digit PIN logins and require a password of at least 8 characters – standard practice these days. It would’ve been a reasonable change, if they didn’t leave a loophole one can fly an airliner through.

As a compliment to stronger passwords, united.com also required account holders to set up “secret questions”. Leaving aside the question whether this is a good security measure in general, United’s implementation is recklessly poor. A user can’t enter their own answer – one must select from a small list of curated items. For the question “What color was the home you grew up in?”, there are 12 choices available. “What is your favorite cold weather activity?” gives you 23 options. Those are low numbers – but it gets worse! When trying to reset a password, a user will be presented with 2 questions – and only 10 choices to select from for each question!

United reset password fruit 1-10

So you only need to guess 1 out of 10 twice – and you are in.

This is not extra security, this is security theater. But of course in air travel, security theater is the norm (great job, TSA!)