Identity, culture, who makes these decisions?

Think about this:

"What and who determines company culture and who makes decisions in a larger company?"

I find myself sometimes confused about impossibilities and assumed ways that a company "works", or its culture and why it exists that way. The reason is that for many things inside that culture, there is not often a conscious decision by anyone backing it up, but rather some kind of "common assumption" that it simply is the way it is. This is detrimental for innovation and creates a certain culture of non-repudiation.

Could you say that the decision is made by anyone in a reasonably high position, which is then communicated and assumed or communicated as truth and reality by others?

To whom belongs (the culture of) the company in the end? Is it truly (fully) hierarchical? Are decisions consciously made?

Who do you talk to when you really feel like doing something, but do not carry out because you feel uncertain? Why do you feel uncertain? Who reflects actual decision authority? What would happen if this feeling is ignored and the idea is persisted anyway?

Film: The Corporation

I was watching the film "The Corporation" two days ago and can seriously recommend it. The film was produced by a number of top-thinkers in the field, all sort of left-ish.

It's a film about environmental impact, horrible worker's conditions and the responsibilities that the corporation should assume in order to minimize those. The film shows that the only responsibility sought is that "responsibility towards the shareholders", not all the stakeholders that have to do with the corporation, like its employees or customers.

The film opens the whole discussion on ownership of human DNA. While a human by itself cannot be patented ( and therefore 'owned' for an amount of time ), the very DNA sequences that make up your body *can* be patented, at least in the U.S., on discovery. There are certain rules along the lines of this, but clearly this is a worrying issue.

Already in software, the patents create an incredibly difficult legal mine-field to navigate. With the speed that these sequences are patented, if the same thing happens on this field of research, there would not be an end to law-suits in any direction, with only the common society suffering from the flow of money being used in the fights between the companies, not for actual research.

Specifically the company Monsanto was mentioned in the film with respect to their product they are developing to increase cow milk production. The product has not been tested thoroughly, is causing birth defects for cows or other diseases and has not been properly tested if it has adverse consequences for people in consumption.

Monsanto is also involved in the famous Terminator seeds. Another call for common sense here.
In one way, society is going forward. In another, very quickly backwards.

I think the whole issue of the corporation is totally misunderstood and the score-card that the corporation is presented is fatally wrong. At the moment, the only scorecard that the corporation will be evaluated against is making money in absurd amounts. There are now laws governing its waste and production and emissions, but it's not enough, because it doesn't require corporations to innovate on what they are doing wrong. Only if the CEO is really responsible is this ever going to happen.

I also believe there is a lot of ignorance around this issue and too much media-control of the corporation. The media, being privatized, will be likely to gain lots of money of corporations for *not* broadcasting news that may damage their reputation. This happened with Fox News as can be seen in the film. Some former employees of Fox retaliated, won a court case. But then the court case was overturned on the basis that it is not illegal for a news company to lie.

Human Development

As with the last few posts, I'm recently thinking about human development a lot and how technology could apply better to help it. One of the first things that comes to my mind is better dissemination and use of "poverty information".

Google, luckily, is also hosting sessions on Human Development, which is good, because only very little 'information' about poverty and human development is spread around.

A swedish organization, GapMinder, is spreading the statistics of Human Development in a really cool, interactive and visual way. No more boring statistics and numbers, it actually starts to feel like something. Of course, these statistics don't do anything, but I'm personally convinced that the lack of media attention to human development (since who wants to know about depressing facts?) contributes to how the world looks today.

I think and hope that a couple of projects will come up that will take this further by a couple of levels. I've personally thought at times to create a Google Maps app that shows the Human Development statistics, colored per country. An added benefit of GMaps is that it allows detailed information to pop up, where a server application can generate graphs and comparisons with other countries.

Maybe joining this information with the CIA World factbook provides some further new insights. (Note that this book is considered public domain, as it was created for the government).

So.... let's see what new things come up. A couple of NGO's and organizations are finally starting to recognize and understand the Internet better (they're not techies like everyone else, mostly care about social interaction). I'm thinking of going to the World Social Forum next year. Have been to Venezuela. It takes a while to let things sink down. Then you realize how little you know about social situations around the world.... And then what? (e daí?)

How about a "coin of social exchange"?

I'm thinking recently about the philosophy behind 'running a business'. This is a very new area for me to explore, but this makes it more interesting to the exterior because I do not depend on historical views and teachings. :)

What I think is underlit is the social value that businesses could provide, but unfortunately cannot be expressed in any 'coin' or as a measure of value. How many businesses really provide social value nowadays, only for the purpose of providing that value, not just for money?

The customers of today are not the customers of 100 years ago. In that time, people were shopping for primary needs. Shovels, food, clothes to wear, etc... Money was created as a means to make social exchange easier. It has grown into having a life on its own almost.

Nowadays a large number of products are based on secondary needs and you could say 'tertiary' needs in some cases. Since society has evolved significantly in many countries. The intelligence of these customer has risen dramatically and also with this the desire for novelty. 10 years ago, browsing the Internet was a truly amazing experience for many, nowadays we shrug our shoulders.

If a company sets only monetary goals, but no social goals, it misses out on many things that a company could do. For example being a leader in corporate social responsibility and not just a follower of the law. Shift your company strategy and set social goals as your objective, all within the constraints of your financial capabilities. This stimulates innovation.

"Running a business" successfully does not necessarily and solely mean "making scandalous profits". That's however what American economics and views teach us. It can equally mean providing a sustainable point of interchange between a company and a consumer and expanding this business to new areas, seeing economy as something wider than utility and money, including social benefit as a type of currency in the equation.

You may become stinking rich if you play 'unfair', but at what cost?

If you do focus like no one else on your customers and consider your financial revenue as a secondary consequence, maybe then your company brand might actually mean something to customers! So, a successful formula for this view on business, in the order below:

1. Consider profit secondary to customer experience and feeling of fairness.
2. Maintain financial capabilities.
3. Be innovative with new products, because that will elevate the user experience. New things provide good novel experiences. Old things do not, unless offered in new ways.
4. Develop a brand.
5. Make sure you have one or two things that are slightly "evil" or "evil-obvious" in your policies, but not so evil that it outbalances the customer experience.
6. Back to 1.

The above points give an example of what I would consider "running a business". It doesn't focus on money-making, although it is an important aspect, the primary focus is its reputation and place in society. How do customers perceive you and what can be done about any negative views?

Example of considered-evil companies are software companies. Car companies are not seen as evil, even if they make good profits, because the company makes an effort each and every time when they produce the car. So it is reasonable and fair that the consumer pays for its acquisition. Software companies have made a moderate investment only in the beginning. But afterwards, the same charge is applied for the same software at zero cost of reproduction. This feels unfair, since no effort is spent for each and every company. That feels wrong (if it really is, is left to discussion).

Poverty and the loss of dignity

I live in Brazil and am frequently confronted with poor people. As many people, I initially understood the essence of the word "poor" to mean "lack of income" or something similarly economic.

The real sense of the word "poverty" however, I think is more associated with a lack of choices and capabilities in the social sense. I watched the Holland game on the Recife beach last week and talked to another Dutch person about the issue of poverty in Brazil. This person was specifically upset about how homeless people have no privacy or intimacy, since they live in the street and there's nothing that is private to them any longer. A specific example was a couple in the street, living under a piece of cardboard with a grudgy blanket, lying close together, with buses and cars driving by.

Many traffic lights in Brazil are used by poor people as a commercial station, since it's the only place where people are required to stop. So, people with cars have at least some money, the poor people depend on that middle-class to sustain themselves, so all traffic lights are full of merchants and window-cleaners.

With poverty thus, I see there are even define-able levels. But the levels are not so much economic, more social.

I found the article on Wikipedia shows some good links and explanations of what poverty really is, rather than just assuming it as the lack of the flow of money:

Poverty

There are some documentaries running on the issues of poverty that show how poor people also have a need to be recognized, not so much as being a poor person, but simply as a person. This fight for recognition sometimes causes them to do severe things. One guy in Brazil in Rio de Janeiro at one point kept a whole bus hostage for a number of hours, screaming out of the windows and threatening the people inside. He fired some shots, but never at people. He maintained the hostage situation only because cameras were aimed at him. It was clear he was under the influence of drugs. Later he stepped out of the bus with a single hostage and the police promised not to hurt him. A police officer approached from the sides and took a shot, two shots fell and the pregnant hostage was hit, dying on the way to hospital. The suspect himself was apprehended by a number of policeman, protected from an angry mob. The guy was suffocated by police on the way to jail due to the number and manner of police officers piled on top of him.

No mercy?

When you ask people how poverty should be resolved, the one most single answer of them all is: "Education." But this only goes so far. It implies that these people will be able to resolve their problems themselves as far as you teach them the basics of living in society and teach them skills and if everybody has access to education, the problem of poverty is resolved.

After the war in Holland, the politicians organized themselves to construct what is now called the "social welfare" system. 50-60 years later, this system is now under attack from diverse groups that object to paying for family situations that will not (likely) occur to them. For example, some single persons object to paying for the health costs associated with having children (costs like nurses, hospital checkups on babies, etc.). Why should they be accountable for those costs?

This is a view that is individualistic and capitalist. Only pay for what you use in a society and no dime more.

Poverty itself, I believe, also has the properties of a vicious circle. You cannot attack just a single point (education?), or you will fail. If you attack many, are there sufficient resources available to make the change worthwhile?

One of the latest news articles showed an organized crime group that over the past 5 years has bribed government officials in the health department to sluce all money for ambulances and health spending to that particular company. They would charge twice for every ambulance sold basically and divide the profits with the officials and themselves. The total "profit" of this group amounted to something like R$60 million. A good amount of money that would have been happy to enrich the overloaded and under-equipped hospitals in Rio de Janeiro for example with waiting queues of 7-8 hours. Of course, these are "public" hospitals, not "private".

This is the cost of an individualist society which feels no responsibility for others outside the family. A society where politicians do not make choices in favour of society itself but choose to use their appointed post as a source of corrupt income. Only a relatively small layer of society can still thrive in this situation. But the people that need protection the most, the poor, suffer harder and remain deprived of their dignity. It is possible to ignore the poor for some time, until at some point the pressure becomes so high some action needs to be taken.

Final numbers. 10% of the richest people in Brazil, 18.5 million people, were earning 68 times more than 10% of the poorest people in Brazil, another 18.5 million. The same factor in Norway and Holland was 9-12.

Your Media Rights Under Fire

I was looking at the link to the BBC, where the BBC claims that if you watch the World Cup at work on broadband, the business is liable for TV license fees. Legally, they're right. But is it just?

I lived in the UK for 2.5 years. It was an important and interesting time for me. Also, I was under fire from the guys of the TV license because I did not own a television set or any other device set up to receive a TV signal. I got letters almost threatening that non-payment for the license would subject me to 2,000 pounds worth of damages. As if everybody in the UK owns a television and nobody would be excluded. It's true that they find a lot of people that actually do have televisions though.

http://news.bbc.co.uk/2/hi/entertainment/5074406.stm

I also know that the BBC is producing useful and informative programs. Since I live in Brazil now, I see the very difference between 100% commercial television and public television. It would be a bad move to dispose of public television altogether. The information that is available from a 100% commercially run media system is appalling. It is very important to have an objective media system, or at least acceptably objective to inform the public about politics and what goes on in a country. The commercial TV stations do not know or care how to do this properly.

As you may already know, I am reading Lawrence Lessig's "Free Culture", a very interesting and important book on the subject of our culture, our media, our entertainment. Culture is the expression of who we are as human beings, which can extend to social groups like countries, religions, types of music, etc.

More and more, the very essence of ourselves, our culture, is reduced to content, a word I have started to loathe. The copyright laws in the US have been exemplary of the kind of industry control that media companies have over culture and its distribution. Most of those companies have soared on the success of reuse of other works, especially Disney (reusing the Grimms stories in the beginning and even Mickey Mouse is not "original"). And now it is becoming worse:

http://news.ft.com/cms/s/fa07af4a-fadc-11da-b4d0-0000779e2340.html

The RIAA has been very active in campaigns on file-sharing and claims success in this area:

http://www.usatoday.com/tech/products/services/2006-06-12-riaa_x.htm?csp=34

All that society wants to do is share, it's quite natural for it to do so. But the original author deserves compensation for his work, the sense of property needs to exist and I think also needs to continue to exist, up to a certain point. If there were infinite control of an authored work, no derivative works could or would exist, threatening the continuity of our very own culture or its richness.

I think what is important to remember is the idea of "having right to media and information", the right to be part of the consuming part of media or culture. At the moment, the industry is changing this "right" to a "permission", consciously or not, because they feel threatened by the advances of technology that makes the act of copying of authored works cheaper and cheaper. Rather than embracing this technology itself, they choose to hamper it in order to persist old and outdated business models.

What society really wants to do with culture is different than how publishers want to sell. Which side does the law favour? In the case of society, it is probably government to defend the cause of its citizens. This means sensible copyright laws that offer a balance between the rights of the authors and the rights of consumers. But the industry is continuously sponsoring the rights of the author. Who sponsors the rights of the consumers?

http://news.ft.com/cms/s/fa07af4a-fadc-11da-b4d0-0000779e2340.html

Because the publishing companies already have enormous control over what gets published, they also have enormous control over our culture and how our youth experiences and views our society. Basically, the industry has a very high control over how our society shapes itself through the publishing of media. One of the biggest fears of people is the reduction of media and culture to a uniformly shaped mass. No focus on diversity, but focus on mass-dispersal, because that means profits.

http://www.drmwatch.com/

http://en.wikipedia.org/wiki/Digital_rights_management

I think from a social perspective that the efforts on this control of authored works is wrong:

http://defectivebydesign.org/

"I think that people need to be better informed about DRM, its implications and be informed on every disc so they have the ability to choose! Then we will come to some kind of democratic and public decision on DRM and together, we'll form the true opinion what's right and fair."

Holland wins against Serbia Montenegro

I've been watching the game yesterday with other Dutch people in Recife. We joined together in a local pub/restaurant with a couple of tv's, somebody brought in a cake and there were orange shirts and flags everywhere. The public was mixed Brazilian and Dutch. This event was mainly organized by the local NBSO, which is a commercial support organization (check here).

I liked the way the Dutch were playing and the whole of the World Cup so far. Things have been pretty decent up to now with seemingly no violence at all. The biggest fear for me of course, is that Holland will have to play against Brazil :). Everybody in Brazil loves football. It's very big as a sport and there's hardly anything else in the country that generates so much patriotism and emotion. Work officially lets people go home after 1500 when a Brazil match is due to start since it's hopeless anyway to get them to work seriously during a Brazil match.

In the meantime, I'm for Holland and when NL is not playing, for Brazil. When the two meet, I suppose I'll be secretly favouring Holland in their efforts. :)

The need for continuous thought...

This blog entry is mostly focusing on the topic of innovation. The words "process" and "policy" get involved as well.

When a company starts up, it does not have a documented process to follow, even less does it have a certain "policy". The employee needs to use, as his only tools, "common sense" and "domain knowledge" in order to get from A to B. If common sense and domain knowledge are correctly applied, the company makes profit. If the employee makes the wrong decisions all the time, the company tanks and goes bankrupt.

Enter the 5-year old company. Employees have routinely been taking decisions and have established a so-called "process", which is the way how things "are/estar" done. If circumstances do not change, the application and utility of that process can be very successful.
( "estar" comes from Portuguese, meaning a temporary "to be" )

Enter the 30-year old company, 20.000 employees later. Now things are established through "policy". This means, ensuring that "processes" and decisions conform to a certain "grand strategy". This is when things "are/ser" ( meaning the definitive "to be", the Portuguese "ser" ).

Policies can be hyper-destructive, because they are very difficult to change. It is as if they live their own lives, have no owner and no person within the organization generally feels powerful enough to change it. Changes in policy generates enormous quantities of confusion in an organization. Why?

It is funny and sour here that the word "policy" actually means: "A plan or course of action, as of a government, political party, or business, intended to influence and determine decisions, actions, and other matters" (Source: The American Heritage® Dictionary of the English Language, Fourth Edition).

So, the description of policy does not carry the heavy sense of "mandatory", but the interpretation of employees is exactly that. Making a better decision for the sake of your project success may result in assuming the risk of being put in the spotlight for non-compliance. Policies are suitable and useful as a guideline, when assessing which option to favour if the alternatives are about equal.

The companies function differently, because there is a difference how (much) people use their mind. The startup company requires people to use their mind continuously, but the large corporation can live with a very large number of monkeys that apply a certain process, what they would call "best practice".

The world changes very quickly, so what was established as a process some time ago needs to be re-evaluated every so often. Communication about (requests for) changes in processes generates confusion and a barrage of "re-alignment and re-organization emails".

A process should be a helpful tool in order to define the responsibilities for the parties involved. In that sense, the "process" serves a very useful function. But it is also a potential hiding cover. It gives people the potential to stop thinking and use the process as a cover excuse for lazy-ness (or worse, blame the process as the point of failure, ignoring for now whether the process is actually wrong, incomplete or correct!).

Here we enter an interesting arena. The question is not whether we should dump processes altogether. The question is what we need to understand about each process in order to make it a useful tool in our collaboration. Understanding the process in this way goes much further than knowing how it actually works. People need to understand why it is there in the first place, what purpose the process serves outside the context of the routine or practice it is applied in. This is exactly the definition of responsibilities. Without the understanding of those definitions, nobody can use the process efficiently and elaborate on it (become contributing employees).

If employees do not have experience with alternatives, they will not understand additional reasons why the process is the way it is, simply because they cannot imagine how in practice things can go wrong. This "wisdom" only comes from exploration, which is forbidden in the context of the process because you are required to follow it. Thus, processes have the tendency to train monkeys that never think outside the box.

With this problem of never thinking outside-the-box, other people that do understand are inclined to try to prevent the failure by mounting another process, in an effort to help lesser experienced people. Such processes often formalize the responsibility that the employee should have assumed in the first place. Instead the process provides yet another cover for the employee to hide behind and it can make them even lazier than before, countering productivity.

When the environment or market the company operates in does not change, then the mounting of processes is not necessarily a problem. The process is basically the description of a routine that you must follow like a machine in order to be productive. Factories could do this for example and then you might get good results, everybody happy, no problem.

As in IT especially, markets, environments, tools and "insights!" rotate every three years or so, the process building blocks age very rapidly, which in turn very rapidly ages the organization. If you do not have the infrastructure available to create 'process change or adaptation', you will eventually age until you reach mortification.

People that do think in the organization would already have left by that time. So, losing your key employees is a good indication that your organization is aging too rapidly and not adjusting quick enough.

Project Dune restarted

I've decided to re-kickstart Project Dune using the Google Web Toolkit. This is both an opportunity to learn more about the GWT in a practical environment, possibly creating re-usable components along the way, and also finally gives me the ability to really focus on the behaviour and functionality of the QA app, like I always wanted. Coding Struts & Forms is basically so incredibly boring (copy data from here to there, configure in .xml, it's just numbing).

The project can be found here:

http://pdune.sourceforge.net

http://www.sourceforge.net/projects/pdune

I'm looking for designers and developers to help out in the effort.

Google Web Toolkit

It's all over the Internet. Google published the toolkit that is useful for making AJAX enabled web applications. It generates HTML and JavaScript on the fly and you can test the web application inside a JVM during development (relying on Tomcat).

I'm convinced that this is the pre-cursor to a radical re-thought on MVC frameworks and architectures. I don't think it has ever been this simple. HTML and JS personally for me have been the most difficult things to get right. MyFaces and Struts look ancient technologies compared to what Google has done. Amazing job!

http://code.google.com/webtoolkit/overview.html

Worldbank Initiative

http://youthink.worldbank.org/

I just viewed the Webby awards and discovered this great initiative by the world bank. Remarkable to see an institution for that purpose being so responsible and showing such great involvement into the society (after all, they deal with money & economics, like IMF). If only the IMF were more like the World Bank and had the same insights... *sigh*

http://www.amazon.com/gp/product/0393324397/104-4868404-3764761?v=glance&n=283155

The "YouThink!" is interesting by itself. It treats a whole lot of issues that activist groups are fighting for. This is not done from the activation perspective, but more factual, scientific and "current, common knowledge".

A lot of different tests out there to get people interested. Great website.

Streets of Casablanca & HDI

I've just finished up on the film "Streets of Casablanca". It is very gripping. It highlights three children in Casablanca, Marocco, living in the street. One of their buddies gets killed by a gang of other street children and they are trying to arrange for his burial. The IMDB probably has a better description of the film, but I would definitely recommend to have a look at the "Extras" section of the DVD. It shows that the film crew also spent a good amount of time working with these children and supported several projects that helped these children to get off the street and build a better life.

There is not a good way to measure "human development" or other figures that properly reflect standards of living. I went to the United Nations Human Development report website here to have a look:

http://hdr.undp.org/

An interesting project could be to start mapping this data in color maps using Google's map API. At the moment it's just a collection of numbers and it does not really say much yet.

The majority of reports I've browsed through are "comparative" reports. I've read a paper here that puts Brazil in quite a good spotlight, but when you look at the majority of the population, poverty is all around. We're seemingly doing not too bad, but from the streets here I know better. An amazing number is that 10% of the richest people in Brazil hold 46% of all wealth. 30% of the poorest have only 3.1% of all wealth. These do not state absolute numbers unfortunately.

XSL:FO processing

I'm looking into using Formatting Objects lately for the MoneyJar project. Nothing is useful until it is possible to show the monetary amounts on an invoice for customers. :)

The problem here is again that there are many standards, each with pro's and cons, some of them much work to implement, others very easy but difficult to use for others... and the list goes on...

So I am doing some research in the area of XSL:FO and maybe write or extend another little editor around that. There is an editor on SourceForge called FAO, but I don't think it is very intuitive for people to use. I had lots of problems using it because I don't understand XSL:FO in the first place. Also, I don't think many other developers are willing to trawl through 400 pages of XSL:FO documentation in order to write a quick template for something.

XSL:FO is specifically created for the "printing" industry. I'm basically not doing anything else than rendering PDF at the moment based on a full "xsl:fo" document, but this final "xsl:fo" document is generated through an XSL-XML pipeline as well. So, the invoice generator I have set up prints itself in an XML format. This XML is forwarded towards the XSLT-engine, where it generates "xsl:fo". The final "xsl:fo" is then converted into PDF.

Yes, this costs processing power, but so does C + files + ASCII and the latter solution is much more error-prone. The best layout & printing capabilities of the latter is nothing compared to the powerful layout capabilities of XSL:FO + XSLT + XML, not to mention its portability to other document formats. Plus that if you have a printing press set up that accepts PDF to be printed and mailed into envelopes, there should be a solution out there that can automate this without the need to purchase an expensive package. So there you go, I'm sold!

So, if you know of a 'complete' editor for XSL:FO that does it all and is WYSIWYG and can be used to 'drag' element data or attributes into a view of some kind, that would be totally awesome :).

Money Jar

I've just released a new project on SourceForge. It is a financial library for Java with high precision. The target is to streamline the use of financial numbers in Java applications. BigDecimal supports the precise numbers. Surely, many people already use BigDecimal for finance, but issues like standard rounding and the currency the money is in are not resolved.

This library wraps BigDecimal into a Money class and has some features like automatic rounding based on JVM-wide configuration. It also does not allow calculations to take place between two incompatible currencies.

Since this project is more incremental, I intend to add more and more features to this library. The objective is to add "invoicing" support with some factories and helper classes, so that people can prepare invoices much easier. Another is to add a "tax engine", which is configured according to rounding rules, summation rules, tax application order, etc. to the nation/state where you live, so that taxes is much easier.

Researching about the subject of billing and taxation is a real head-ache. I do not intend to make this a library that you can use straight away in any part of the world. The configuration requires you to do the legal research, this library supports you in setting up the environment so that it is all resolved by itself.

I have some future plans a bigger project that uses this library that also involves some kind of distributed computing.

Open Source Progress

Together with some colleauges, I've written an article that was submitted to the Open Source Software Forum in Porto Alegre in April. We'll hear more if it was accepted 17/03.

The ideas are generated after thinking about corporate software use and licensing. The license costs in that area are huge.

The difference between corporate open-source vs. desktop open-source is that there is more "utility" for desktop opensource apps for any particular user and the user base is much larger.

Probably this is why there are so little corporate opensource projects available. I'm not talking about Apache webserver, Struts, JBoss or other projects. Those are platform projects. I'm more talking about specific solutions for specific companies, like CRM / HR / ERP / workflow etc. Yes, projects exist here and there, but the momentum behind them is very low.

So the idea was to generate an additional driver (money) behind opensource development, whilst keeping it open. We do this by proposing cooperatives that drive a system's development in cooperation with many other cooperatives across the globe. Ideologically speaking approaching an idea like syndication of some sorts and a main controlling body when the project gets out of hand.

We've analyzed the difference mostly from an economic perspective (the reason why anybody should even care or look into it, since opensource adoption by itself is hardly a goal). Next article we hope to present is more targeted at the specific composition and organization behind the cooperatives.

Back from WSF 2006

I'm just back from WSF 2006. There was some very little coverage on CNN about it, the majority of CNN was about the World Economic Forum in Davos.

As said, there was a lot of Bush-bashing. However, this was not a gathering of just 'leftists' etc., there were many different people there. It is a bit sad to see that HC gets all coverage on CNN with some radical statements, but the actual topics being discussed are not really being published. Also, although there are some sessions highly politically motivated, the actual essence of the WSF is social action. This does not have anything to do with politics, just changing governmental action whatever the government is composed of.

I went to a couple of interesting meetings:

* A workshop on the legislation of labour in developing countries

* A presentation about the work of a foundation that protects low-skilled labours in the Amazon, so that these people do not turn into slave labourers. The definition of slavery here is not specifically "being traded", but it means co-ercing people into labour with very bad conditions by making it appear as if they owe the owner of a farm. This is done by not declaring the bar bill, charging for the trip from their home-city to the Amazon (the 'gato', which is the guy who lures these guys, is posing as a travel agent and states you have to pay to get there) and more of these practices. For these reasons, one month later they get one bag of rice, maybe some trinkets, and that's it. Some drink water from a hole in the ground. Housing is basically a rooflet of wood with some poles underneath. Beds are wooden without mattress. When working with raw coal, no protection is given against pore-infiltration of coaldust... etc... etc... The federal government is on it now, clearing out the farm owners that violate this solely for the purpose of profit.

* A presentation about incubating "cooperatives" of low-skilled labourers in Brasil. This is where recycling-goods collectors can collaborate together, get better rights together and learn about administration and do things together basically for the good of them all. New hints, new ways of work, etc...

* A presentation from the "Mitterand Foundation" about the new society, which is suffering from "predatory capitalism" and especially why this is the case. They explained issues like water management, the right of people to have at least 4 liters of water a day, the privatisation of drinking water! , terminator seeds discussion (seeds that can grow only once and require farmers to continuously buy the seeds from the same supplier).

All in all, a variety of topics. It's definitely worth a visit and next year in Kenya, I am already planning to go again.

Keep It Simple, Stupid!

I've worked all weekend on the Patty analyzer and things are getting quite stable. It has been ported to Linux and Solaris in the meantime. It should now be in a state where people can actually start using it more thoroughly and depend on it.

Download: Patty

Other news is that things at work are also going smoothly. We've recently had a long port of entity beans to Hibernate and also an effort to move from an old JDK1.3.1 to JDK1.5.0. One of the most ugly hacks I've ever seen in that project were that Java code compiled for JDK1.3.1 was actually stored in the database and loaded through a custom classloader. *ugh*.

The title of this post refers to that... Just because you can store blobs and code in the database does not mean that you should. An "innovative" idea may be really cool to code in, but it certainly does not mean that it is a good solution. Far from it. And having 4 different executables with different names that all do something similar is really annoying too. Like the Oracle client installation or ClearCase.

Or projects with dependencies on large frameworks of 80MB in download size. Some of them having licenses or evaluation codes that you need to register for.

My objective with Patty is to make it a very simple thing to install and run. Download Tomcat 5.5 (around 7 MB in size), unzip and run it. Deploy 'drill.war' in there. Set up the agent library (42 - 160kB) in 5 minutes in a Java runscript and people should have a profiler ready for the project.

Progress on Patty

I've been working hard to push out an alpha 0.1 release. This is a sneak peek into what this is evolving into.

The actual GUI doesn't look particularly pretty, but the functionality is all there.

At the moment, I'm finishing up on heap analysis. The first thing I did was the ability to iterate the whole heap to analyze heap usage. Immediately thereafter I worked on the ability to trace objects that you can reach from all instances of another type of object. For example, if you are analyzing a web server application, you really want to find out what kind of objects are referenced in your HttpSession and the total size that is referenced in the instances overall. My idea is to average this out later, so you get an understanding of the average session size.

Garbage collection needs some more work for visualization.

To be able to get to alpha-1.0, I'm still verifying with runs in JBoss now. That seems to start up quite nicely at the moment, so I'll probably look at another J2EE application for download somewhere, deploy it and actually start using the thing for real now, instead of the meager test application.

Patty went Alpha

Patty, the project for Java Performance Analysis, has gone to version Alpha today.

I've included a selection mechanism for code coverage and method execution analysis and thread contention analysis is working too. It can be downloaded from sourceforge.

I'm currently working on the Beta version, which needs to include dynamic class instrumentation or de-instrumentation. This has just been tested and is working fine now. There is just a little bit of work on the command sockets that I implemented, basically a TCP stream that communicates between the web-application and the agent library.

Another feature planned for Beta is garbage collection analysis and 'which-object-references-what'. The idea is to show a couple of 'root' classes (objects in the VM, but will be shown as classes in the analyzer), so that you can easily track where memory is being held or where it increases.

Maybe a couple of those analysis dumps can be used in the future to perform some heuristics checking on memory leaks or whatever.

After Patty goes Beta, I want to write up some stuff on performance analysis using Patty, so that the tool can be used much more effectively. Some charting is going to be used as well and some better heuristics on the averaging and trouble points ( potential 'trouble' page for instance, where the problem points are listed).

Garbage collection in Java

I'm going through a lot of information lately about garbage collection, starting from the 1.3.1 JVM. Interesting process overall. The current project I am working in uses the 1.3.1 and is going to upgrade soon to a newer JVM.

The collection itself does not truly impact any server application until the loads get higher and the space settings suddenly start becoming very inefficient. Especially in the 1.3.1 VM and with a relatively very large memory heap maximum, the full gc can take up to 25 seconds to complete.

This is mostly due to inefficient use of Java objects and holding on to references. This happens when server applications decide to "cache" a lot of information to prevent database queries. If the sizing at the design stage is not properly assessed, then the application will perform fine for some 200 records, but if this increases to 20,000 then certainly someone needs to take another look.

My advice is to keep caching to a minimum or use a standard caching framework like 'ehcache' to do this for you instead. It is certainly better than making your own implementations in your application, plus that the caches there work together with Hibernate and exist closer to the persistence layer.

Java 1.4 is already improving this by parallelizing garbage collection. This has enormous positive impact on the server process, but only when the server really gets busy on CPU of course. I'm going to read up soon on 1.5 collection and hope to post some useful documents in the next 6 months to be posted on the Patty site, along with the performance tool that I am working on.

Regarding the performance analysis tool, it is coming along nicely. Already there is a framework for web access now that is going to be expanded and the agent library seems stable. I'll need to expand the testing application to also have some thread contention and to load more classes, libraries, maybe even include Derby + Hibernate and CGLib so that I can see how it performs with proxies and dynamic bytecode instrumentation.

Until then!