Confessions Of A Former Data Integrity Addict
Even though I’ve started my career doing scientific research, I couldn’t help getting drawn into the world of software development. Twenty years ago, while I was still studying at the University, LISP was all the rage. However, as soon as I’ve entered the world of business software, relational databases became the focal point.
Needless to say, I instantly fell in love with the relational theory. So simple, so elegant, with strong mathematical underpinning. To this day, I am convinced that, using relational technology, it should be possible to solve pretty much any regular business problem out there.
But the confusion crept in regarding the boundaries of the applicability of this theory. I wrote about these boundaries in several previous articles (Should Database Manage The Meaning?, Shared Data and Mobile Data, and so on). Needless to say, such soul-searching writings provoked an avalanche of emotional outburst from the database crowd.
Mission Critical Software
Fifteen years ago (which is when I’ve started developing business applications), all the apps we were building were mission critical. This means that there was zero tolerance for any, even the slightest discrepancy in the validity of the data. The grave reality of developing a banking system, for example, is that such a system has zero tolerance for any inconsistency.
The reality of mission critical systems goes something like this: there is a formal bureaucratic system put in place in order to provide a consistent service to its customers. Such system is characterized by a bunch of rules describing all the possible valid states that the system can be in. The system is therefore labelled as a finite state machine, which means that it is possible to enumerate all the acceptable states that the system can possibly assume.
As a corollary to this arrangement, it tacitly follows that any other state that may creep into the system is absolutely unacceptable.
Relational databases (or, RDBMSs) have been identified as the ideal technological candidates for enforcing this mentality. These products are capable of declaring all the valid states of the system, and are also capable of enforcing the rules that will guarantee that only such acceptable states ever get assumed by the system.
In that respect, RDBMSs resemble reputable law firms. Similar to how high-octane law firms are geared toward enforcing the latest regulations on the business proceedings, RDBMSs are there to ensure that the mission critical validity of the system state is a given fact.
Non-bureaucratic Software
After I’ve been drilled with various experiences of developing and implementing mission critical software, I’ve stumbled upon areas where such air-tight bureaucratic requirements were not the norm. This is similar to those real life situations where you don’t necessarily have to go to a high priced lawyer in order to conduct a transaction. Sometimes, it’s good enough to go to the Notary Public office, which is considerably cheaper.
Or, sometimes we don’t need to be that formal at all, and a quick informal notice will do. Even in case of a dispute, such quick informal notice can provide a lot of ammunition for us if we (god forbid) get dragged to the courtroom.
But you see, at first I was incapable of realizing that. Under the weight of my heavy-duty formal training in building mission critical systems, I have developed a knee-jerk reaction to being a hard-ass guardian of any, even imaginary validity of the system’s internal state.
Without realizing it, I have become a data integrity addict.
That addiction helped me develop various fascist traits, such as torturing the end users of the systems I would build to obey even the tiniest frivolous rule, just so that the perfect integrity of the internal state of my system gets protected. I’m embarrassed, when looking back, at what a nuisance I was at that time.
What I’ve failed to understand at that point is that not every transaction that transpires in the course of conducting business is mission critical. Sure, there are always those highly visible, bet-your-business transactions and documents that you’d never treat casually. These are the things you’d first take to your lawyer, and go over them with a fine toothed comb before you commit to anything.
But life would be utterly impractical if we were to treat each and every transaction in the same fashion. Nothing would ever get done.
As an example, take one of the most typical transactions that pretty much every person goes through on a daily basis: emailing. Granted, there are emails that shouldn’t be sent the moment they have been formulated. Some of these emails must first go through a thorough scrutiny, possible hitting several control instances, including expert lawyers. Only if the lawyers give their seal of approval should such emails be sent to their intended audience.
But realistically, how many emails qualify for such intense scrutiny? In all likelihood, we could generalize and say that only a handful, and certainly less than 1% of all the emails that are sent, do indeed qualify to be scrutinized with a fine toothed comb.
What would happen if all the emails were to become, by default, candidates for such scrutiny? The regular business would crawl to a grinding halt.
The point is that there are numerous transactions collating throughout the business world that are not so bureaucratic by nature. It would be ludicrous to harness such transactions with a straight jacket of data integrity and in the process kill all proceedings.
Technologies such as Ruby on Rails are a perfect fit for such not so overtly bureaucratic systems. When building solutions for non-mission critical systems, we can relax the database constraints, we can allow for a little bit of a temporary inconsistency in the internal state of the system. We can offer more trust to the human experts using such systems. It’s a win-win proposition.
Leave a Reply
You must be logged in to post a comment.