Chris Reed’s Blog

I’m back from a month of meetings in Hyderabad and New York, to some interesting ideas about lowering the barrier to engagement with engineering on the web. Jon Udell describes a technique he calls “wiring the web“. Yahoo has shown us how we might attempt what might be called “plumbing the web“. One might be forgiven for thinking that we’ve got a big renovation job on our hands. What next: replastering the web? Wallpapering the web and picking matching curtains? Well, in a sense, yes.

The impression given by phrases like “wiring” and “plumbing” is that there’s a right way of doing it - even, perhaps, a certified standard way of doing it. Of course there can be any number of ways of laying out the wiring plan, or designing the plumbing for a house (the fact that “plumbing” and “design” rarely go hand in hand in the UK I’ll leave aside). The point to the rather laboured analogy is that you don’t go around connecting live to ground, or mains pressure to the radiators. But on the other hand, if you visit your neighbours’ house it will be very definitely their own, with their own wallpaper and their own curtains. And (despite the fact that we may not be able to fathom it) they like it that way.

This diversity in styles and preferences is of course what has made the online world such a fun place to be. Tagging, though, is increasingly about consensus building - trying to encourage folks, once and for all, that it should be “New Hampshire” rather than “NewHampshire” (albeit that the techniques by which we might get there are gentle, flexible and intuitive).

At CJ, we have started to explore some ideas for supporting and encouraging diversity in tagging by focusing on tag-rules as the unit of currency to be published and shared across social networks. Let’s say you tag with “New Hampshire”. Whereas I prefer “NH”. So long as I have a rule that captures the synonymy, that says my “NH” refers to the same thing as your “New Hampshire”, then I can benefit from all of your “New Hampshire” tagging. It might be the case that this synoynmy rule is a useful one that we agree on - you might adopt it and benefit from all my tagging too. But then again your gamer pal who’s more into NetHack than politics uses NH a lot, and you wouldn’t want the same rule importing all his NetHack resources. Or at least if you did, you’d want them mapped to your choice of tag for Nethack.

Using tag-rules, the idea is that your own view of the uber-tag-cloud is contextualised and tailored by an interplay between your own tagging idiolect, and the way in which you (and your tagging) interact with others.

Of course, not all users — indeed not even very many users — are likely to want to play with tag-rules. But you only need a few structurers who start introducing tag-rules into communities for the impacts to be rapid and huge for the entire community.

There are more questions here than answers - and building concrete tools around these ideas is still months away. But the possibility of harnessing the anarchy of idiosyncratic tagging opens up some very interesting, flexible and light-touch mechanisms for structuring tag communities.

We’ve been going on and on over the last few weeks about the Long Tail, and how the need for mass customisation that it demands might be met by multi-agent system thinking - or what you might call MAS customisation.

Multi-agent systems have been causing a storm in academic computer science over the past decade or so. Gartner developed a hype-cycle analysis. Platforms and tools abound. The IEEE standardizes. But the technology has so far failed to make it in to the mainstream, due in large part to its lack-luster appeal as far as the alpha-geek community is concerned. Things, however, are starting to change. Russian oil tankers, European organ transplants, US parcels, Swedish electricity and Scottish patients are all making day-to-day use of robust agent software. As industrial strength solutions appear, and agents start to contribute to social networking apps, mechanical Turk orchestration, ubiquitous comms and the rest of the emerging post-Web 2.0 world, real interest is starting to grow. CRM seen from a multi-agent thinking point of view becomes customer-centric with an agent trying to figure out what is best for the customer it represents. Health-care data systems become patient-centric, with an agent offering a single point of contact to a web of care. Logistics becomes a parcel-centric series of localized negotiations between the agent representing the goods and those representing storage and transport facilities. Multi-agent thinking supports “centric”-centric design, engineering, deployment and maintenance of complex software that encourages rich ecosystems for development and execution. The trick is going to be tying centric-centric design into the big SOAs which are going to continue to dominate in the medium-term (even though they’re starting to lose their sheen).

The Long Tail is the marketing buzz of 2006. The book at the centre of that buzz has a title that says it all: The Long Tail: Why the Future of Business is Selling Less of More (Anderson, Random House, 2006). Inverse power laws are hardly new - but interesting every time you find them in a new place: both design and art show some nice examples; a colleague once pointed out to me that brush stroke direction in painters like Van Gogh follow the pattern, and from there I noticed that the same is true in colour distribution in many carefully designed objects. Pick up a crisp packet, or a web home page for a big corporate, for example, and notice the percentage of each colour - plot a rank abundance and you’ll often notice an inverse power distribution. The graph’s not new. What’s new is the idea that the tail is where the action’s at.

In any domain, the big players that have customer-facing activities struggle with this idea. IVRs, call-centre scripts, and high-street chain retail stock policies all point towards a focus on the big hump at the expense of the long tail. This is not, as some old style marketeers would have you believe, democratization of the customer experience, but rather, homogenization: how often does the IVR menu from your telecommunications provider or bank not have the option that describes your situation? How often does the retailer stock only shirts that are too short or too broad? The result is that the seasoned customer - like the seasoned form-filler in any bureaucracy - learns how to transform their needs into a pattern that is expected by the provider (you select the menu option that is most likely to get you connected to a human regardless; you settle for the baggy look in shirts).

What marketing has recently woken up to is that providing service that really matches what people want is likely to win custom big time. And there’s money in them thar hills. The problem is that business processes aren’t designed to handle the enormous variation that the approach necessitates, and most software isn’t designed to handle it either. This is the killer problem of mass customisation: taking what you do and tailoring it to each individual customer. Dealing with a mass market of millions is no longer a case of saying “Aha, you’re a size 16 - Have one of our millions of size 16 shirts” but rather a case of scaling up bespoke tailoring so that it can support millions of custom sizes.

Slivercasting is an interesting example of how the Long Tail is being appealed to, but there the selection is all at the publisher’s end. At a push, even podcasting can be seen as Long Tailish - but the customisation is, for each feed, an all-or-nothing affair. For true mass customisation the number of possibilities is infinite (measured on the reals, for example) or else at many orders of magnitude greater than the number of customers, and the customisation is a process of negotiation between customer and supplier. And the costs are comparable with other mass market offerings. As far as I know, there are no live Long Tail infrastructures that offer true mass customisation in this sense. Yet.

It seems to me that this is a long sought after killer app for multi-agent systems, MAS (that long search is interesting in itself - but a topic for another day). With individual agents looking after individual customers, what becomes hard is the population of those agents’ heads with appropriate beliefs, rules, intentions, and whatever else. With the process of acquiring those data under the control of the agent (according to the MAS credo), the goal becomes much closer. An agent gathers data from the customer, selects business rules from the supplier, infers social structures through interaction patterns, and aims to build a rounded picture of the customer (with respect to the service). The application of rules, selection criteria, decisions and so on is localised at the agent, where all and only the relevant data and algorithms are on hand. If you’re in the shirt-making business, the agent might not be able to cut the cloth, but it takes on responsibility for the customer’s interests (as far as they impinge on your shirt-making), in, for example, negotiating the logistics chain from source to destination. More straightforwardly, in a customer relationship management (CRM) system for a telecoms provider, the agent applies business rules to the customer they represent, tracking the customer as a whole (as it impinges on the telecoms service), tailoring, prioritising, extending the rules for the case at hand (imagine no longer getting junk mail advertising services you already have!).

But more than the examples that leap to mind, what really excites me about the marriage of MAS and mass customisation (what we might call MAS customisation) is that the design philosophy and goals of the two are in perfect harmony: every individual in the system is unique; every individual is valuable (not euphemistically, but literally in a monetizable way). It is this match in the approach that will really yield benefits as new, unanticipated problems in implementing MAS customisation crop up in the first few domains where it supports an obvious business plan. With the right design philosophy in place, the engineering problems can be tackled as individual, isolatable engineering problems - not soul-searching architecture revisions.

Dealing with mass customisation is going to be one of the key challenges in service delivery over the coming few years, and there are going to be many problems. But MAS customisation is a strong contender for building at least some of the pieces of the solution.

Business rules are on the up and up - IBM planted a flag early on of course, and last month, Oracle made a catch-up play. An increase in tools, deployments and algorithms is providing the technological supply, and the intuitive appeal (to senior management) and increasing articulation of business logic inside large corporations is driving demand.

The JBoss / Drools manual has a very nice description of when to use rules - basically, it boils down to having complex logic (you know that queasy feeling you get when you’re not sure whether you’ve just added a close brace to the fourth nested else block or the fifth nested then block? - that’s complex logic) and frequent modifications to that logic. But then the bit that comes next struck me:

To quote a Drools mailing list regular (Dave Hamu): “It seems to me that in the excitement of working with rules engines, that people forget that a rules engine is only one piece of a complex application or solution. Rules engines are not really intended to handle workflow or process executions nor are workflow engines or process management tools designed to do rules. Use the right tool for the job. Sure, a pair of pliers can be used as a hammering tool in a pinch, but that’s not what it’s designed for.”

I certainly agree that it’s all too easy to see some new technonugget as a panacea (programming in XML, anyone?), but I believe workflow is exactly the sort of thing that can be modelled well with rules. Of course, you need both backchaining and forwardchaining, and you need good ways of tying rule firing with state changes in the world, but there is a strong correspondence, particularly if you use some of the neat BPM tools - like JBoss’ own jBPM. Just because the Web Services and Ontology folks foundered trying to build WS-Events doesn’t mean to say that approaches that are unencumbered by the Semantic Web baggage can’t make good and rapid progress.

There is a problem, however, and it is a problem that hits every rule engine of any weight (i.e. that is seriously designed for deployment). The problem is conflict. Let’s say you’ve got one rule that says a customer is to receive preferential treatment if they have an account. Then, there’s another rule that says a customer should be dispreferred if they’re order value is below some threshold. So what do you do if you have a customer who has an account and a low order value? The standard BRE algorithms solve conflicts by attaching an integer value to rules; if two rules conflict, you pick the higher ranked rule. Acount-holders are to be preferred (rank 5/5); low-order values to be dispreferred (rank 3/5). This is a bit like listening to Mozart on your laptop’s built in speaker - you can make out the melody but you lose the subtlety. And more importantly, it fails to have the intended effect.

Perhaps we shouldn’t be surprised: the RETE algorithm that lies at the core of most efficient rule engines is, after all, 30 years old (to put it in context, that makes it a contemporary of the Appled ][, the CN Tower and Bod). LEAPS, the cutting edge heart of the new version of Drools, is only a couple of years more recent. Both techniques are rooted in GOFAI - good old fashioned AI - with all the brittleness and limitations associated with it.

The problem has long been recognised by the builders of business rule sets because they have forced BRE vendors to provide ways of allowing priorities to be changed dynamically. But the current solutions are the equivalents of turning the volume up on the laptop speaker. What is required is a way to avoid the twin gotchas of logical spaghetti and caveats ad nauseam. Both of these bite when a rule base is extended and modified (one of the motivations forthe BR approach in the first place, remember). Anyone who has a passing acquaintance with statistics will have an intuition for the idea of independence - two things are independent iff the probability of both is the product of the probabilities of each. There is an analog in rule systems: two rules are independent if the action of one firing does not affect (either positively or negatively) the firing of another. If rules are not independent, you can get conflict that needs to be managed explicitly (e.g. by introducing a new rule that combines the two dependent rules). The problem of logical spaghetti is keeping track of the ever greater links of dependence between different rules, and dealing with the instabilities they introduce. A particularly common form of interaction between rules is specialisation, and specifically, the capturing of exceptions. You might have a rule that says that parcels should normally be shipped overland. Then you want to capture the special case that urgent parcels should be shipped air freight. Except for those that are over a certain weight. Unless they’ve got special authorisation. And so on. Every time you add a new caveat you need to duplicate the old rule and develop two new versions, one with the exception and one without. So there is a risk that your ruleset grows exponentially with ever more caveats. The two gotchas together pose serious maintenance problems that restrict the application of rule based systems.

Tackling the problem of conflicts head-on is deeply hard. The solution can be captured in a single word: “usually”. If only our rules could be built around the fact that conclusions usually follow (rather than always), and if only our rule engine would process those usuallys making sure that exceptions don’t hold, we’d be saved from both gotchas. There are ways it can be done; IBM have made a start and so have we, and in part II of this post, I’ll explore some of the technical details and practical ramifications.

Because promised second parts get posted within a week or so. Usually.

Just what the world needs: Yet Another Blog. And a CTO blog at that; surely there are enough CTO blogs from real CTOs in huge conglomerates like Intel, Microsoft , CapGem and Sun? (There’s even a generic CTO blog for generic CTOs). Getting a more-or-less company-sanitized view from the Big Names can be interesting in a vicarious way. They’re so far up the tree they must be able to see a long way. But I sometimes get the feeling that they’re a long way from the action; that layers and layers of middle management cloud, polarise and distort the picture. After reading one too many pieces where the goal seemed to be to name-drop as many times as possible, we decided to start something new rather than just moan about the shortcomings elsewhere.

So this is Calico Jack’s CTO blog. We work in an exciting area, blending three areas of tech: multi-agent systems (a newish way of doing massively parallel distributed computing that is just starting to make it out of its academic kennel), nonclassical logic (think rule engines that don’t break), and convergent comms (traditional telephony plus every type of messaging with a focus on the mobile). It’s a unique mix that gives a small company like Calico Jack a very specific take on emerging trends, and offers me an opportunity to get a picture that is informed by the landscape of the market (that Paul focuses on) but has the resolution that you can only get by having access to the detail of the technology. It is this picture that I hope to share with you through this blog.