Glacial Erratics

The following is part of a document I wrote to evangelize the value of layered idiomatic software interfaces to people at work who might be able to encourage this sort of behavior in the future. This versions leaves out some of the specifics of the project (creating a lightweight interface to the Knowledge Base). Much of it is speculative extrapolation and interpolation that needs a tune up. I welcome your tuning comments,    (2UN)

Introduction    (2UO)

Complex systems development is a synthesis of technical understanding, tool and language use, creativity and serendipity. Improved development processes can be achieved by enhancing and amplifying these areas.    (2UP)

Little can be done, by definition, about serendipity. We can make room for it by not overcrowding development schedules, but beyond that luck is luck.    (2UQ)

Creativity results from a synergistic pairing between an individual or team and their environment. People know when they are in the groove: they understand the tasks at hand, they have good tools and they have the skills to do what can be done. Creativity is enhanced by the presence of usable tools which make themselves present-to-hand on initial inspection but quickly become ready-to-hand with use, turning the user into a fluent creator.    (2UR)

This document, in its subtext, describes how tools, when combined, are language; language that can be made more expressive through attention to grammar, idiom and reusability.    (2US)

The SOAP interface to the KnowledgeBase (KB) developed for the Sakai Educational Partners Program demonstrates a style of tool use that may be beneficial to future KB development. The interface:    (2UT)

• Attempts to be decoupled from the rest of the system.    (2UU)
• Layers code to expose only that which is necessary, granting an idiomatic interface to behavior.    (2UV)
• Places control into the hands of the developer and the client system (whether it be local tool or remote client).    (2UW)
• Provides data in a flexible, but straightforward, standards compliant fashion.    (2UX)

Coupling    (2UY)

When object oriented programming reached the general public it was touted as the solution to many programming ills. At the forefront was the belief that objects, especially their ability to be inheritied, would allow and encourage reuse.    (2UZ)

To a limited degree, especially in closed systems, this has proven to be quite true. However, over large systems, or throughout or between organizations, code reuse has proven a challenge because inheritance is a quite static and all or nothing function.    (2V0)

Inheritance and its partner, complex object hierarchies, sets up a system of tight couplings between objects in a system. Small changes in one area can often imply large changes throughout the system, either to method signatures or data structures.    (2V1)

Loose coupling is spoken of in two fashions: One is the management of strong contracts in code through the use the Interfaces. The other is through the use of separate services, especially web services, to provide an application. While the former is worthy of attention in the KB environment, understanding of such is left as exercise for the reader.    (2V2)

A service based architecture for application development modularizes the various functions of an application into discreet chunks. Each chunk has a set of responsibilities and a well defined public interface that it presents to the rest of the world. Information sent in response to method requests should be as complete as possible and require little to no return to the service for continued operation. This results in an encapsulated environment (much like a robust object model, but on an action rather than object scale) wherein the details of the operation of the service can change without impacting the interface. The boundary between the service and the client is solid, permeable only in controlled and known ways.    (2V3)

Traditional monolithic applications pass information within themselves, with little regard to boundaries between subsystems. If more information is required, a simple dive into the system is all that is required. While this is initially convenient, changes in the details of operation can have cascading effects throughout the system.    (2V4)

Layering    (2V5)

Walking hand in hand with coupling (coupled with coupling) is layering. Layering wraps an existing monolithic code base with interfaces to features that act as services. Code external to the wrapping layers can rely on the stability of the wrapper that hides complexity underneath. The layer itself deals with complexity, providing a simple facade.    (2V6)

Layers play a primary role in the creation of idiom (we use idiom here to mean short turns of phrase in a language which capture a large body of meaning: a shortcut of large expressiveness) because they can be used to create a unified method that aggregates what traditionally have been multiple methods. For example analysis may reveal that the following set of methods is a common thing to do in the KB (for the KB programmers in the audience this is not meant to be accurate):    (2V7)

   my $search = new Search();
   $search->doSearch($query);
   my $docids = getDocuments();
   $docids = cullDomains($docids);
   $docids = cullArchived($docids);
   $docids = cullVisibility($docids, $visibility);
   my @titles;
   foreach my $id (@$docids) {
        push(@titles, Document->new($id)->getTitle()->getTitle());
   }    (2V8)

An idiomatic method of doing the same could be to provide a layer containing a doSearch() method that works as follows:    (2V9)

   %docInfo  = KbIdiom?->doSearch($query);    (2VA)

Idioms in software code are valuable because they perform two, somewhat opposite functions:    (2VB)

1. They allow one to express a great deal in a small amount of space by hiding complexity in a safe fashion. Safe because the inteface they use is well established. A developer using KbIdiom? can rely on the interface it presents regardless of changes elsewhere in the codebase because 1) KbIdiom? is a contract and 2) doSearch has a grammar which is simple and does not rely on the rest of codebase for its data typings: primitive or relatively primitive types are used. (Consider for example the issues that might occur if $query were a QueryObject? and the signature for constructing a QueryObject? changed.)    (2VC)
2. Idioms in code also do not prevent access to complexity. A truly flexible language must allow the speaker to get down to the fundamental core of the language to be able to express all that they need. Idioms do not simply detour complexity: they contain it, which means that complexity can be studied, understood, and used, if (and only if) necessary. It is important to emphasize this container notion: Idioms do not replace complexity and the power and flexibility that complexity implies. Complex systems are required for solutions to complex problems. Idioms in code hide, and provide a marker for, complexity, where necessary or valuable.    (2VD)

The rest of the document goes into specifics on how the created tools help to achieve the desired decoupling and layering and the implications this has for implementing future services. I can provide some details on these things, upon requent.    (2VE)