Software Architecture "methodology"

As a software architect, buzzwords and the buzz itself comes at you in various guises. Sometimes it's the business itself that heard about company X employing methodology Y and they want to do the same. Most of the times it's a hotshot from company Z that wants to change the world and do things radically different, promising totally different flexibility and capabilities.

I'm a bit tired of all these words. The only thing that counts are activities, as they produce the output that can really be used, and revolving around that output are only a number of factors. No single methodology will ever succeed in setting up a dynamic company. It's the people themselves, how they interact and communicate, the level of politics, the culture and how quickly the development team picks up new technologies that are really worthwhile.

Architecture methodologies are basically all about the same thing... guidelines about how to execute the development activities and a registration of the foreseen limitations and recommendations for that team.

The latest thing is of course SOA. It's often thought of as a new methodology, but it's actually just a different perspective on your organization. It cannot be called an architecture though. It does force systems and data to be organized a certain way, but it does not dictate how. So, SOA is a direction that a company decides to take, which might make sense, but it isn't anything else than that.

I see EAI architectures at the core of architecture, influenced by SOA, whereas SOA is the organization of the infrastructure into a set of services. It's as if SOA is basically a aggregation of resources to make it an artificial and digital interface that other organizations can use. So, SOA is more like a vertical whereas EAI definitions are mostly describing the trees of the "what-otherwise-would-be" forest of resources. EAI defines the guidelines in getting the resources set up, SOA defines what to expose. EAI is driven by the output of SOA and SOA is limited by the level of organization in EAI.

Once you think about it, the only thing that changes are the levels of connectivity and the level of digitalization. Organizations in the 60's were already capable of exchanging information, it just mostly traveled on paper. Currently, the velocity and size of information has grown so much that paper processing is just not an option any more. That is why the complexity of security and interpretation is now being born by digital systems, whereas this previously was interpreted and verified by human beings.

So, although SOA makes it really interesting for companies, it's far from a silver bullet, although pulling off a successful infrastructure of that kind will very likely make you more flexible. The problem is however that in every architecture, a number of assumptions are always baked in. Sometimes, when businesses significantly change, the architecture can't change along with it, thereby requiring large investments to bring about that change. Every business and organization has a "heart" to it where data is kept or basic business processing is done. If the business changes direction, then that's where the most changes need to take effect, impacting all other services that depend on it.

As a methodology for architecture I quite like TOGAF. If you've read the book, you'll understand the role of an architect better and the place such a person takes in the organization. Within each organization, the politics are different though and thereby the expectations. It's always a challenge to find your right place and produce those results that make people happy. Sometimes you need to make some noise in order to wake up the masses.

As discussed in earlier posts, I strongly believe that taking more time for a design prevents a lot of troubles at a later stage. Troubles at a later stage are times more expensive than resolving that in the correct stage. Remember that if you're a project/program manager or director. You can't take shortcuts in IT. The taxman always rings the doorbell later on and will ring it more than once along different projects.

The latter statement basically means that a single shortcut taken now to get things done will eventually require interest payments later down the line that far exceed the initial amount and from my experience, this can be multiples of that. I've worked in a project that should have taken about 250,000 EUR or so to develop. It crossed the million by a good couple of thousands.

Further reading:

http://www.cis.gsu.edu/~mmoore/CIS3300/handouts/SciAmSept1994.html

Project Dune was released yesterday and I intend to re-evaluate software engineering from the ground up, first as a set of activities and practices rather than thinking of it as a process only. In another sense, it's as if the texts assume that people basically deliver good work, but as if each activity isn't always followed and each activity isn't bound to the next phase. I don't really believe that. I think that too many engineers are insufficiently aware of the impact of not executing certain activities or not following a certain best practice. The problem is mostly to do with pride, wanting to be creative and differentiating, or over-estimating their own set of skills (especially against fellow engineers in the same team). Better yet, it's difficult to come up with an experienced team of engineers that beyond knowing their profession, also know how to work together and get things done without introducing new unknown technologies (for the sake of curiosity and learning something new).

Maven

I've never had a good chance to look at maven, but just today I've for the first time generated the "project(code) management" layout through maven. I'm very impressed at its capabilities and dependency management and intend to use it for any project following up. It's just too good really and there's not much sense to keep hacking ant files in order to manage the project.

As an intro to maven... It's a great tool to start out a new project and manage the project configuration, build, dependencies, tests, releases and sites with. It does a lot of stuff out of the box in a standard way and you could more or less say that it's a 3rd-4th generation "make" tool. Where with any you still had to "program" tasks, with ant you can just configure pre-defined tasks that generally run in a consistent way (since every project needs it) and be done with it. Having a consistent layout also means that you don't need those ${src} and ${build} variables any longer.

It has the ability to produce manifests for jars, version/build numbers and can automatically deploy to locations you register. There is inheritance between configuration files, so that you can set up a chain of configuration files for your development "street" as you call it. And due to the registration and management of library repositories, your dependencies are automatically tracked and updated whenever the project thinks it needs them. No more looking for lib files and downloading them, it does it for you.

For Project Dune, there's an effort starting where we'll document software engineering in a very straightforward way. Rather than calling it a "methodology" like Scrum, XP, RUP and whatever, the idea is that we just focus on the activities that must happen anyway regardless of the approaches used to fulfill those activities.

Then the idea is that besides the activities, you might need the output of one activity as input to another or trace back information back to the source. You could imagine a certain "thread of information" running through a development life-cycle.

Every sw project has such a cycle, but not everybody commits to executing all the required activities to guarantee quality. For example, I imagine that RUP adopters execute a lot of activities, but not every activity necessarily adds value. And people working solo on a project may never do active reviews.

The idea is that the process doesn't become the holy grail of an organization (the administrative efforts), but the activities that are part of engineering. I don't have a full view yet how this is going to work out, but the site on project dune will be modeled after that vision. It's not very likely to get the same size as a process-driven description, but a minimal site where activities are clear and what they contribute to a project is already a very good start.

Work on roadmaps, sw engineering, technical debt

Project Dune is developing a new roadmap for the last half of 2008 and to kick off 2009. Being open-source, you can always read the latest developments on the project wiki.

Part of the roadmap now became strategy, which will probably be moved out to the start page close to the mission, thereby making the roadmap a practical guide of where something is going and why, without necessarily focusing too much on the actual activity.

Another part of the roadmap of Dune is the initiation of a set of processes, practices and tools / checklists / material to support the process of software engineering.

A particularly interesting term that was coined by Ward Cunningham is the term technical debt. As a quote from the wikipedia site:

The analogy to financial debt is that the poorly-written code requires "interest payments" of maintenance effort, which would be smaller or non-existent had the code been developed more carefully and with better long-term planning. Rearchitecting and rewriting the code to be more maintainable is analogous to paying off the debt principal.

Other sites:

http://www.agileadvice.com/archives/2006/12/technical_debt.html

http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx

The graphs here are very good examples of the consequences:
http://kanemar.com/2006/07/23/technical-debt-and-the-death-of-design-part-1/

As part of Project Dune, I'll need to think about how to come up with practical guidelines for software quality and explain these terms in more detail, probably with extensive use of links. The idea is to become the main site where people look for quality information. That should include links and understanding about software engineering, development, language choices, etc... Ideally documented in such a way that it goes from a high-level overview to a detailed overview.

The first objectives are to come up with a generic process / vision of software engineering itself. This is no easy feat! It's difficult enough to align people on the same vision within a single team as there are always people that have different opinions or people that have the thorough conviction that things are done better in a different method.

Thus, the idea is not to suggest "Agile Development" or any other more specific method for software engineering. It should go one level above all this and just state the following:

• Objectives of the activity
• The activity's function
  

These two sound exactly the same, but are somewhat different. The objective is basically what you're trying to achieve. The function is what you're trying to prevent/discard during that process or what you should pay specific attention to.

I'm wondering whether project managers in SW projects nowadays have sufficient knowledge of project analysis and the terms used for sw engineering to be able to steer back to a winning situation. If you look at the term "technical debt" for example, its actual scope is quite large. Besides technical debt though, there are other reasons why projects fail, which are more on the area of communication and social interaction (the requirements need to be correct).

A good initiative for the Dune project would be to try to come up with 2 or 3 main areas and coined terms that contribute to project failure. Then subdivide the areas and identify the causes.

The idea being that the identification of root and main causes for project failure (and supporting cases?) would clarify the need and use of certain activities in a project. I'm not using the term process, as I'm not too fond of that term how it's interpreted in general. Activity means something that a tester/developer carries out as part of his job or task and which is easily part of a process. The process is thus basically a sequence of activities done with a number of available tools and materials, nothing more. Understanding the process thus doesn't mean how to comply with some corporate or industrial standard. It means that you understand what you're doing and how that fits in the big picture.