Wednesday, July 04, 2007

Development with GWT, one year down the road

With GWT being relatively new, it is now approaching a final build and may very soon be upgraded to production status and taken out of Beta. This is a good time to reflect on one year of GWT usage and my experiences so far.

When GWT came out, I was on an overnight standby and had all the time to look into the technology. I looked at the samples and very quickly got very enthusiastic. Now after one year, I still see the immense value that the toolkit provides and since the first release, many very important and cool changes and additions were made.

When Java came out, there was a lot of hype around the technology. Nowadays the Java language is accompanied by good practices and some people even have written rather large books about Java Design and coding patterns.

There are things one should know for GWT development as well. Many of the regular J2EE patterns are not exactly applicable to GWT overall, so many of the historic J2EE patterns are simply not useful.

A couple of things that I think every project should think out before starting on a GWT project:
  • What is the final size of the project? Will it be possible to fit this in a single module (compiler memory usage) or is it necessary to modularize from the start? If so, how will this be modularized?
  • How to structure components, modules and libraries to facilitate re-use of development in other projects as well? Use of imports!
  • It is highly recommended to think about a strategy for history browsing and perma-links. See History class of GWT and "onHistoryChanged".
  • Focus on the re-use of widgets and make developers aware of the importance of abstraction and reuse. I have found it is much more important to develop components that can be reused in different contexts than it is to solve a particular problem at hand in a particular way. Make sure to review that code.
  • Test the application on different platforms and on different browsers as you go along.
  • Develop the application on different platforms too. It makes sense to use Linux with FireFox and Mozilla by 2-3 developers of the project, where the other half uses Windows with maybe different versions of IE.
  • If the project will get very large, consider running in "noserver" mode from the start of the project. You will have to model your development environment slightly differently to be able to achieve this.
  • Develop proto-types of screens from the start. Do not develop screens before the proto-types are ready and you have an idea of the final Look and Feel of the application overall.
  • Hire a CSS expert. Make sure that your CSS tag names and approaches are consistent, make sense with your developers and its development is aligned with the HTML approach that GWT embeds. Code the general look and feel into the "gwt-" tags. For cases that require different approaches, derive from other tags and apply the differences there.
  • Give back to the community those cool things you are developing or document your innovative approaches. Post on the GWT newsgroups or help out with the development of one of those UI libraries out there. Some companies require specific legal sign-offs for contributing to open source projects.
  • Program against interfaces where applicable, not against specific classes.
  • Usability becomes much more important on the Internet. It's not just putting together some HTML pages anymore. Make sure you have someone on the team that understands usability issues and can design useful, easy screens for people to use.
Hope that helps. Good luck in your GWT projects!

Wednesday, June 27, 2007

Global Warming (An Inconvenient Truth)

I watched the (now) famous film of Al Gore yesterday. It's about global warming and the lack of human response against CO2 emissions and our pollution. I recommend watching the film and establish your own opinion. When assessing the topic with other sources, please use credible sources and not believe every article that run-of-the-mill media throws at you. There is a lot of propaganda out there that is not based on scientific results, or attempt to derail scientific results so far that are clear indications. Watch out for that!

http://www.climatecrisis.net/

Some companies believe that complying with gas emissions and other more rigorous standards will put them out of business due to unfair competition. Well, maybe. But that would point out that our social standards, those standards that we are looking for as consumers, need a bit of a push, plus significant consensus-building at a UN-level to establish better environmental standards worldwide. Why bicker about trade equality and so on when in a couple of years time there won't be a planet to bicker on? It's quite gullible!

What if all of this is simply untrue as some magazines try to make us believe? Bruce Schneier wrote a very interesting article on rare risk and overreactions. I'm making this point, because the investment on fighting terrorism has been very high over the past few years due to immediate emotional involvement, while global warming is a much more serious problem, but not felt as immediately needing a resolution. So, in a way it applies to this article as well:

http://www.schneier.com/blog/archives/2007/05/rare_risk_and_o_1.html


So I highly recommend seeing Al Gore's film. I will change my ways and opinions because of it. While you're at it, watch this video too:

http://www.youtube.com/watch?v=5g8cmWZOX8Q

1992, Rio Earth Summit.... so.... what has actually happened since? Back 15 years ago, we had people that were already seriously concerned... Where are the significant changes and modifications that have taken place since then?

Even though we're adults and we think of ourselves as pretty smart in general, we ourselves still seem to act like children when it concerns the custody of this planet. This role of custodian seems to be necessarily replaced by the role of a full-time janitor. So, when do we start to clean up this mess and make sure it doesn't repeat itself?

Thursday, June 21, 2007

GWT 1.4, Tomcat 6 and Comet tutorial

I've been experimenting with Comet applications a little bit and integrated this with GWT 1.4. The tutorial is sort of working, but still needs a bit of work in order to become more stable.

The tutorial is here:

http://gtoonstra.googlepages.com/cometwithgwtandtomcat

Please send me any comments, bug fixes, etc..

Tuesday, June 19, 2007

Contextual search...

This blog is called "radialmind" for a reason. It is based on my perception that the mind is radial and not linear. The whole concept is rather easy to explain. It is easier to consume a book by going through the hierarchy of it, that is, the TOC, the individual chapters, the paragraphs and the lines than it is to read the book from start to finish. This is the same concept that people from "mindmapper" use for example to document your ideas. It's not a linear documentation, it is radial.

You'll notice that when you start to read the book linearly from the start, each time you hit a header or paragraph header, you need to tell your mind to switch context. That is, put the particular following content into a particular context. If the hierarchy of the book is very poor, it will be very difficult to follow and read. This is because, I believe, you need to start back at the core of your context and take a different path to another part of the context where you will fill in the information that you are going to read.

Searching the web has some similar problems. All major search engines produce linear search results. There have been some search engines that do this differently in a sort of "related-words" kind of way, but these have been very poor because it takes a long time to get to your actual context from the point where you are (or where the web page is, rather).

I think it makes sense for words to provide an initial search context and then connect to other contexts of through verbs.

This post ties back to my post about "semantic web search". Rather than focusing on "nouns", entities, we should focus on contextualizing information through verbs and interaction.

So... maybe... as a thought and discussion to develop... nouns provide an initial context to the search that may be dead wrong. But the verbs further contextualize your thoughts into the specific items that you are looking for?

Room for further thought in this particular domain...

Comet and Server push

There is something I'm checking out in my spare time. A technology that exists for a while called "comet". There are other alternatives in server-push, but I prefer first to look into the one that seems a bit more standardized and thought out. I've read other threads in newsgroups that state comet is overkill in many situations.

Server-push isn't actually "push" in the sense that the server initiates the connection. It's more like a delayed client-pull with features on the server that prevent excessive resource drainage.

In this model, the client connects to the server and waits for information. Browsers have timeouts, sometimes servers do, so it will reconnect every x seconds if there is no data sent over the link. So, on error it reconnects. If the connection errors x times, it will stop connecting and display an error about unavailable services.

Comet has in the meantime be implemented in Tomcat as well as Jetty. Jetty has documented in more details how they implement server-side processing and has some statistics about expected resource usage and server loads.

A problem with a server could be the number of connections, but the first thing that runs out are processing threads.

The comet model basically uses asynchronous I/O processing with thread pools. So it's a way to multiplex your client connections to get time allocation by one of the threads in the thread pool. This of course requires the same session and client information to be available. A thread gets assigned to a connection when there is data to be read or when an internal server event (through the application layer actually) writes data to a client channel.

Using this model, the browser will also have to implement some event switch. It receives the data and does something with it. This could be a chat window or popup and so on. In general, a browser can only have two connections to the server. This may be a little bit limiting for data transfers, since the comet connection continuously consumes one connection. This makes image loads and asynchronous data loads take longer and become serialized. One way around this is to use virtual hosting techniques to assign the comet connection to some kind of HTTP event server and the other connections to other web servers that process data requests. A potential problem here is data security with javascript applications that are not always allowed when connecting to a different server than where the script came from.

Well, the usual applications arise from comet technology. chat and so on. But there should be other possibilities as well if we get the network security right (and issues like NAT and so on!). Consider connections from browser to browser without an intervening server. The immediate services are basically event-driven client applications that react on server events. You might connect to a certain site and whatever is going on in the system will notify you of that occurrence. This is a very interesting feature for many sites, since the user will no longer be 100% responsible to pull all that information to him.

Moreover, from a database access point of view, perhaps persistence frameworks can finally become more consistent. If the server knows that you are watching some kind of information and possibly editing it, any other user that edits the information before you should cause an event to be sent to your browser to notify you there is new unseen data to consider. The browser might even retrieve the new data and show it alongside. It should not be too difficult to get this done. There are caching mechanisms for example that can help in detecting which objects are being viewed by whom and when objects get refreshed in the cache through an edit.

Saturday, June 16, 2007

High Precision Event Timer

I've been suffering a bit from a somewhat sluggish machine at times. Sometimes I run VMWare and that is especially annoyingly slow. I already posted the max_cstate thing (powersaving functionality).

Here is another thing I tried. "hpet=disable" on the kernel start line in /boot/grub/menu.lst.

What is HPET?

I am still experimenting with hpet disabled, but so far the Gnome desktop seems more responsive and things slightly faster. I used to get some 'stutters' on the mouse cursor sometimes and more processing times, but things seem to run better actually without the precision timer.

At the same time I noticed that some applications crashed suddenly. Not very frequent, but when load was high.

I'll keep this value for now and see what happens. I can always revert.

FaceBook. The new web?

Web 2.0 and YouTube gave us "user-generated content". It is where we post our videos, audio, photos, text, blogs etc. online for everyone to see. 90% of everything is junk (maybe like this blog :).

The other 10% is funny, interesting, insightful, challenging, or whatever. Some later developments are new ways to play around with that content or host even new things that people didn't think of before. There are a million ways for example that we can interact with one another. Yahoo Pipes is all about processing news and information and delivering it to you through a kind of processing pipe.

FaceBook
is slightly different. You can inject content and pictures on a simple level, but you can also host embedded applications integrated with FaceBook. FaceBook is a bit like an existing portal on the web somewhere and then you can request your services to be integrated through this portal and use their API to interact with other services of FaceBook. If you consider "infra-structure", this is what FaceBook provides. You provide immediate business logic that is hopefully new to everyone.

Here are examples of this new kind of thing. The previos link shows the reasoning behind FaceBook, which sounds very interesting.

One of the last lines reads:
the Facebook Platform is primarily for use by either big companies, or venture-backed startups with the funding and capability to handle the slightly insane scale requirements.
Yes. If something is really successful and with the current efficiency of our social networking capabilities, "novelties" travel through our network at an insane speed. Not necessarily faster than general broadcasting, but there's also no filtering by a third party in the case of broadcasters. It could be that a 3rd party through other interests decides to downplay or diminish a certain event, which, when taken as "raw information" might be very important for everyone to know.

These snowball effects can increase load on any server farm in an instant. If you manage to get your company's link on CNN, BBC or Slashdot or any other large site, you'll certainly be sure of a lot of traffic instantly that may last for a day or two. If you consider social networking sites where people might actually return daily, if the services provided there are really good there is an exponential growth pattern and insane growth requirements. Just ordered that big iron? The next day you'll order 10 more. Whoops, your bandwidth is running out. Whoops, the firewall got attacked. One angry user just launched a bot-net attack on your servers.

Infrastructure, infrastructure, infrastructure and lots of investment, instantly. And on the business side you need to keep things interesting, or the network will quickly drain out. What happens when another site comes up that offers similar services and something new that you didn't think off? Is there any sense of "loyalty"? You're not talking to individuals necessarily. It would be interesting to see how individuals behave as part of a social networking site. Do they exhibit more a kind of "flock" behaviour (they go where "the rest" goes?) or are their actions still based on individual decisions?

If we can recognize "flocking behaviour", this may be good when the business grows... but wow, it can be very bad for business if the flock heads the other way.. there is no stopping it!

Here is another interesting post on one of the facebook blogs:
There is a valuable lesson in all of this. There is a ton of money in developing platforms that make it easier for people to express themselves quickly and easily. Following this thread I can imagine the future value of virtual worlds such as second life where users can pick and choose everything down to their clothing, height, etc with the click of a button. Life is a story. Those applications (software as well as physical devices) that make it easier for people to share their story for others to watch unfold will be the ultimate winners when all is said and done.

Thursday, June 14, 2007

Semantic Search

I'm not an expert at websearch, but here goes... Some rants and ramblings on semantic web search.

I watched a program this week with a well-known philosopher. The program was about technology and media mostly, as well as social networking sites and so on.

One question asked during this program was whether semantic websearch would soon be a possibility and when exactly this is likely to be happening. The response was, from the philosopher, that he didn't think semantic search would ever take off and is basically dead in the water. The argument was that the context and meaning of certain words differs from one person to the next.

Although this is true, then maybe semantic search does not really mean searching for things in a general context that is known to be true, but search in specific contexts that the search engine understands belongs to that person, his perceptions and beliefs (formed by life experiences, human contact, environment, country culture, tradition and so on).

One thing that I suspect is not mostly used in web search is the verb. Most searches strictly use nouns, but the context of that noun can differ enormously if it is not accompanied with a verb. The verb would put things into a more specific context to a great amount, but it is not yet in a personalized context.

Steve Yegge blogs about the differences in "verb" and "noun" thinking from the perspective of a programming language. You could say that programming languages are in a way means of communication with a machine, to express ideas and so on.

Anyway, as I said, I have no idea to what amount search engines currently use verbs or contextualize searches to be more specific. It might consider search history as one way of improving hits, but this is not very reliable as our priorities and contexts can change very rapidly.

Regarding implementations of such a search engine... It would be a search engine that exists today with the added difference that user interaction (with user profiling) would add a context indication to particular pages. I don't think it is necessary to actually define all contexts prior to classifications. If you work with neural networks for example, the computer has no idea what it is doing, but the end result of each calculation comes close to what is expected.

It would be a great idea for research. To tie a neural network at both ends for a search engine and see what comes out. The difficulty with this neural network is of course how to heuristically define numbers based on the page... Or rather, how to encode the content of the page in such a way that together with the input of words and the user profile, the end result will be a particular score.

Another approach is to focus more on the verbs and start counting occurrences and take that as a contextual factor.

Perhaps the most limiting thing in search is that the search itself is badly expressed with words? I have a certain contextual idea of things that I am looking for... What is the best way to tell a machine to go looking for that particular context? We could store user's profiles, focus on verbs and all of that, but what about location or approximate location?

Some search engines provide advanced searches and this may be very helpful in this regard. In order to get anywhere, I guess it makes sense to include psychologists and anthropologists in the discussion to understand thought, expression and context better. There may be ways to convert these things in different ways to gain a more meaningful communication dialogue with a machine.

People mostly consider semantic search to be : "teaching the machine". Punishing it when the results are not what you are looking for, rewarding it when it is exactly on the mark. But if the context differs from one person to the next, there is a never-ending cycle of punishment and the machine just gets confused. Some things that are in the same context for everybody will get very high search ranks. But searching should be more effective than that. It should also aim to expose the niches.

Monday, June 04, 2007

Python

Tonight decided to take a look at Python. I know a bit of Perl and did create a couple of scripts in this "language". More formal and stricter languages like Java, C# and Visual Basic seem easier to learn, but compared to Python generate a whole lot more code.

I was pleasantly surprised by the offered modules from Python and how little code it takes to accomplish something. The documentation is quite up to speed and it offers some quite ingenious unit testing capabilities. You just wrap it into the docs. It then becomes a test case plus an example for somebody else to use.

Now, I find Python slightly easier to use in comparison to Perl. It's slightly less cryptic and uses more the concept of "function" than "operator". I was very pleasantly surprised to see it has support for SMTP, WWW, XML, UNIX, threads, concurrency, data types, (easy) iterators, functions, modularization, serialization, profiling, C module extensions, classes, embedding and so on. It's in a way sort of comparable with certain features of Java.

My advice: Give the tutorial a spin once... You'll get to know the capabilities and the rest from there is just reference work!

http://www.python.org/doc/

Low-level == innovation, high-level == entrepreneurship

If you work in a place where innovation is stimulated (like C.E.S.A.R), there are certain ideas that you can truly identify as innovation and others that are new or providing services that do not yet exist, but are not necessarily "innovation".

I'm looking at a couple of ideas all around for new technological "break-throughs" and a good number of these ideas, described as innovation, actually are "new systems" that just do not yet exist in the market. There are big problems here for the execution of these ideas:
  • Large systems are very risky to build and the effort required to actually complete them is always at least two times higher than the estimated effort. (see Vista, see internal "large" projects).
  • To make money of these "larger" systems is difficult. You start from a couple of clients maybe (if you succeed in selling it), but this is not the beginning when you start to make money. That is only after x years.
  • Support, after-care etc. are difficult to arrange. Your "development" doesn't just stop right after development is terminated. You may actually need more people to be able to seel than you did during development. Do you really want to start a company?
  • Maybe the system does not exist in the market, but is the problem actually more or less resolved in other ways? (is there truly market need?)
  • By the time the system is finished, the market may be gone.
Thus, you should consider:
  • If you want to start a business, aim for some niche market, develop a large system and focus on customers and selling product/services. This is not innovation in its entirety. Maybe only one line of code will be.
  • If you want to innovate, but not start a business as usual, focus on smaller parts at a lower level in the system.
Lower level innovations are for example search algorithms, distribution libraries, audio/video codecs, operating systems, embedded software, graph algorithms or the combination of the above to solve a *very* specific problem in technology. The focus should be on developing *technology* not directly applicable to any market problem. You'll have to focus on better efficiency in most of the cases of innovation.

Monday, May 28, 2007

Congress in Porto Alegre

I've come back from Porto Alegre, where I visited a congress on business administration (export, import and business between France and Brazil). There were some good presentations and the overall event was interesting.

We actually know a couple of people from this region, my brother in law and another couple that spent some time in Cambridge, UK to work with specific professors on their topic of interest. We met with them also. It was a lot colder than Recife.

Just on the outskirts of the city is the lake and a region they call "Ipanema". I had a look at the larger park close-by, which was used to host the "International Social Forum" some years ago.

Well, there was a bit of time left in the evenings luckily, which we spent with my brother-in-law to visit some good restaurants in the area. This particular area had a good lot of serious italian restaurants that I can recommend. Another one was the "Bistro da Rua", which is also a good place to just drink a bottle of wine and chat with some friends.

Tuesday, May 22, 2007

Linux Ubuntu on Dell

Dell is going to offer some hardware that comes with Ubuntu Feisty (7.04) pre-installed. These are exciting prospects for the propagation of Linux into mainstream markets that are non-geek. Does this mean that through the evaluation of Dell, Linux is finally considered mainstream-compatible?

Well, I know for one that Dell runs a lot of Windows software, not just offering it with their hardware, but they also run this internally. So they are not particularly interested in software as a technology (or as religion). We need to consider this move from Dell also from the perspective of support.

Windows, as the OS only and a couple of "productive" applications is a basic platform. Let's see the following formula:

known hardware + known software == x probable support calls

However, when we start installing "3rd party" software into this mix, suddenly the support calls can theoretically grow significantly larger. Not linear, but supposedly also exponential, especially when a commonly used piece of software happens to be incompatible with their hardware.

Ubuntu, as we know, has a repository with a very large amount of software on it that should satisfy most people's needs. I cannot truly remember the last time I downloaded an RPM or DEB from the Internet and installed it outside any repository.

This is a different kind of support that they have probably investigated, the support of testing the known software in the repository on their known hardware. If this provides all software tools that people need and it all works, then this is a fantastic service and may bring costs down significantly on the customer support front. Plus... since more than Dell customers use the software as well, for Dell support to resolve issues, they are no longer alone. They can count on the large number of forums, blogs and the community to help people out.

The question is thus... will Dell treat the community well? If they manage to become a responsible member of the community and contribute their productivty back, they gain much more than what they originally invested, as the momentum of the Ubuntu community should definitely increase. Openness in communication, testing of software on many different machines with many different kinds of users, hardware that works, software that works... Will this generate happy customers? Is this the future of computing? If mainstream picks up, does this mean that the market will *demand* Linux on their computers for community support (no fee!) and pre-tested distributions and hardware/software packages?

Some people are already discussing that Dell might start their own Ubuntu repositories and become full mirrors. If the cost of running those mirrors and hardware/software compatibility testing is lower than running a whole array of customer support reps, then this would actually lower the opex for Dell, which means more profit! exciting indeed...

Friday, May 18, 2007

Swamped...

Bit swamped at the moment, so little time to blog about anything. Will deploy a new release of Project Dune over the course of the weekend, which should include the new timesheet module. That will only be sort of Alpha-ish, since the locking functionalities won't work yet. Also, it needs a breakout class for the timesheet processor, so you can export your times reported to another system.

Well, great. We've been playing with GWT 1.4 for now. The hosted browser fixes and memory leak fixes really do help development, so you could say it really is maturing.

I'm looking forward to the suggestbox, number and dateformat classes and the Rich text editor for the document management part. Haven't had any time whatsoever to deal with HTML<->DocBook conversions though, let alone putting in place some simple templates for PDF conversions using XSL:FO.

Wednesday, May 09, 2007

Performance of VMWare on Linux disappointing?

Probably only for laptop with power-save functionalities... but you never know:

cat /sys/module/processor/parameters/max_cstate

echo 3 > /sys/module/processor/parameters/max_cstate

This value is related to the number of level that the processor takes to save energy (reducing its performance and power consumption when idle).

Friday, May 04, 2007

Computing challenges of the future

Since the first computer we've come quite a long way in integrating the computer into our daily lives. This has made significant changes to the formation of our society, in the context of expectations, beliefs and ethics.

There will be a time though that all processes (activities) that we perform are already automated and then we'll have a number of companies competing on the same level. Consider an application. It performs a certain kind of activity for the user. We have applications that work on private data at home (photo editing, accounting, office work like writing a letter, and so forth). At the moment we see these activities becoming more "online", so that we can do this from any place and not just from home.

The majority of applications has "options" and functionalities available that the user can choose from. By using these options and functionalities, we come close to executing that particular activity that we really wanted to do. There are new activities that can be invented that were otherwise not possible, but in general it is a replication of another activity in a different context that is transferred to another context for another need. (like sending a letter not just by writing a card at home and then posting it, but sending it from your PC and now sending it from your phone, also called email). Different contexts, but essentially the same thing. Our society however changes its perceptions just by using that technology and more importantly, changes its expectations and constant needs. (people become dependent on the technology).

There will be a time though that many of these services or functionalities are already automated and that "aggregated value" by itself can not be provided by just automating such activity. It needs something else.

I guess that one of the challenges of the future is that the computers need to be more helpful in their assistance to making decisions or assistance in detection. This probably means a lot of data-warehousing and processing and analysis and then use the outcomes of this whole process. It goes way beyond any kind of simple algorithm in this case. We need to mathematically understand better how the world fits together.

So the future of IT seems more geared towards "optimization" and "efficiency" than just "automatization" the way it's still enjoying things. After we have automated the majority of tasks and all these companies offer their products, the next step is optimization of these tasks. To be able to better understand activities, you must understand more about that activity domain. Or rather, linking back to my previous post... a computer engineer will not be able to compete on higher levels any longer just by having knowledge about computing or engineering. You'll have to invest in gaining knowledge of other problem domains and then apply the knowledge of one domain to the other, cross-over.

Saturday, April 28, 2007

The IT worker of the future

We all know that the US has a large number of IT companies or departments in mega-corporations and we also know that over the past couple of years, a large number of people were deemed "redundant" in their own country. Their jobs were off-shored to other countries like India, China or Brazil, mostly because the actual work they were performing is similarly available as a service elsewhere at a much better cost. With the advent of the Internet, getting the results of that work and the project team interaction is very easy.

What we all probably need to do is re-think the objectives of companies, as I wrote about in earlier posts. Whilst I am not going to even start to highlight particular companies in this posts, (it is irrelevant), the concept and context of a corporation and company responsibility should be evaluated. Is it meaningful that a company is only aiming to make money in this world? What if we, as a democracy, change the legislation to make them also aim for other objectives? Should we introduce protective laws against job off-shoring?

If the trend continues, then I am afraid all IT workers will need to start learning more than just IT. The technology is getting easier all the time or at least more accessible and documented. Just look on any search engine for some particular problem and it is likely that you get many possible solutions. People with lower wages in other countries have access to that very same information. How can you make a difference?

I believe that much of actual software that is written today will become more commoditized. Not to the level of steel, just more. If this happens, then it is no longer enough to just know about technology. In the future, you will very likely need complementary knowledge in order to keep your job in your own country. You must bridge the gap between technology and some other activity that is really valuable to the business. This kind of activity is much more difficult to execute than following a requirements specification. It is also an activity that can possibly produce a lot more value for your employer than if it would just acquire an existing solution.

The first thing to consider is that IT doesn't really matter. It is there to be used, but it's a bit like a truck. The truck itself provides no value just by its existence, but the way how it is used is meant to bring the cargo from A to B. So, the shipment of cargo from A to B is the actual thing that provides value, not the truck moving from A to B. The actual activity of building systems is meaningless, because you do it for a certain goal. That particular goal has meaning, not the system itself. The more you understand this distinction, the better and more efficient the systems you will build are.

I think that the IT people of the future can no longer sustain themselves in some richer countries unless they understand how they should improve themselves to better contribute to actual goals. This is possible by (better?) applying IT knowledge onto a better in-depth knowledge about the problem domain. Just building systems isn't going to cut it anymore. Focus on the problem, find out everything about the problem domain, find out how it really works, then shape your technical solution around that.

So, knowing about IT is still important in order to apply that knowledge to the problem. But it becomes much more important from a value perspective to understand the actual problem domains in which we are working. You should aim for being that person that writes the specification for producing something basically.

Thursday, April 19, 2007

Workflow systems

I'm looking at workflow systems for work and I'm very much enjoying it so far. The main package I'm looking at is called "jBPM". It's quite a large package to understand.

Workflow programming, to begin with, is different from actual programming in that it focuses on the process and on the execution of things, not on appearance or transformation of data. What I quite enjoy is that this statement means that you focus on the middle part of the construction of a system, rather than focus on the beginning (screens, user input) or the very end (database) or how some requirement fits in with your current system in an architectural way.

So, in order to successfully construct workflow-based systems, you first need to change your thinking about IT and how systems get constructed. It's not necessarily a very (pedantically) 'clean' way of programming (all data is stored in proper elements and there is compile-time dependency between all objects and all that), but it provides much more decoupling between functions, which on thus on the other hand improves reusability and componentization.

You should think of workflow as a series of automatic or manual actions that need to take place, easy to re-arrange and reusable. These actions are arranged in a diagram, which specifies which actor or pool of actors can execute those actions. The final part of the diagram also states whether these actions are executed in parallel or in series. To top things off... the whole process "instance", that particular order or that particular holiday request will get persisted into a database when it has no way to proceed immediately because it is waiting for external actions. You do not get these features "automatically" from any software language.

The latter word "language" is specifically mentioned, because you will start to mix your domain language (say Java or C#) with another "graph-based" language, in this case XML. So, the XML specifies authentication, actors, pools, the "flow" of the process, whereas the domain-based language helps to express componentized functionality (such as performing a credit-check, updating databases, gathering user input and so forth).

If you re-read the last paragraph, you notice how different workflow-based programming essentially is and how it could provide enormous value to IT systems if it is done right.

As an example, if you know Mantis or BugZilla, you know from the developer's or tester's point of view what the screen looks like for bug manipulation. Well, this screen is a single screen with all components in editable form, but from a workflow point of view, it should be constructed differently every time with only those components required for that particular step... For example, if you start to update or categorize the bug, then you do not need all the fields available in the screen to do that. When you have a process that dictates the need for a CCB comment, then you also have far too many elements on your screen to do specifically just that.

The point is that in general, many applications show everything on screen that you may not necessarily care about at that time and the user needs to compensate for the lack of guidance by understanding the process documented elsewhere in a very large document... Wouldn't it be great if we could just look at a screen and get our business process right and know what to do at the same time?

The other thing I notice is that many SRS documents are not necessarily focused on the process, but are focused on data. This shows me that there must be a huge opportunity in this area to provide consultancy on core business processes and merge this with IT specific knowledge.

Software is something that should be changeable in an easy form. Some people interpret this as "scriptable", but scripts are software as well and you wouldn't want to have production-editable fields that allow business people for example to modify those scripts at runtime. So there are only specific scenarios in which scripts actually add value, because if only your developers modify those scripts, why not write it in a strongly typed language instead?

Workflow based systems, due to their high decoupling factor and focus on process, might be slightly easier to change than J2EE, .NET, C++ or ASP systems. It matters from the perspective of flexibility and how you allow systems to grow with changing needs, because the needs *will* change.

Lastly, someone at JBoss mentioned how workflow systems are very much in their initial phase, comparing it to RDBMS systems. It's not very clear how they can be used effectively or in particular environments... what else we need for workflow systems to become very successful. The key part in improving these opportunities is to take the best of both worlds and merge this into something better. We may have to take a radical step into something else and see where it goes. I am just considering... wf based systems may be slightly slower due to use of XML and process persistence... but with a couple of machines and putting all processes on similar systems, what do we need for widespread deployment of this technology?

There must be things missing here, because not everyone is using it yet :).

Sunday, April 01, 2007

Yet more

Here are yet some more design works:

Some cartoon art


galaxy explosion kind of thing


gel/glass/liquid button graphics. Can contain any text or icon really.



1024x768 wallpaper for my computer.

all of the above taken from tutorials to learn more about designs. The last took most time, but also was more actual drawing to do. The last has a very interesting, unintentional detail, which is the base of the tree. I sculpted the tree out of an actual picture and then noticed the trunk looks like a foot.

So, probably I'm going to read up a bit more on design theory from here on. Things like logo construction, colors and those kind of theories.

I actually tried some cartoon characters for drawing, but that didn't end up in much yet ;).

More design work

These are two more designs I created with Wacom and some standard functionalities.
This is a modification of a fractal. I basically heavily embossed it and layered the result on top of the original. Then grayscaled part of the picture for a concrete effect and layered other images over it for a sort of glass like effect. As you can see, I don't master glass yet, which is why I started with the following image, one of an orb over a newspaper. That one looks much better.

This orb is produced solely under photoshop. It started with a simple sphere in a flat color, than I added an inner shadow effect to create some white at the bottom right (it should have more in the final image). Then another gradient in the middle, which was masked by a radial gradient. A hotspot in white at the top and then a selection of the initial orb that had a fill of black and transparency, which was scaled down to about 40% in height, slightly blurred and slightly moved to the right. The glasses and newspaper are from a photo. And the problem I run into obviously is not paying attention to the existing shadows in the picture :).

G>

Saturday, March 31, 2007

Wacom tablet

I just bought a Wacom tablet to also pay attention to a different kind of creativity. I never really drew anything in my life basically, but the computer helps a lot with imagery in many cases. This one is called "beauty", but then in Japanese :).

Well, as with all things, the first try isn't much of anything, but here you go. I have an incredible lot to learn about the art, graphics, how to use applications and so on.

One of the things I did start out with is a way to create fonts. I managed to create 5 characters and got them working, but then I dropped it because I didn't see much need further.

Other things in design that sound very interesting and that I will try out, probably in the order given:
  • Logo's
  • Image manipulation
  • Create paintings based on photo's
  • Actual drawing from scratch
So far, I have used "inkscape", "gimp" and "fontforge" as the tools of choice. InkScape is really cool and quite different from Gimp. It's dealing a lot with shapes where the Gimp deals more with pixels and photo manipulation.

I can recommend getting one of those drawing tablets. It's a lot of fun and the navigation on the computer feels totally different.

G>

Monday, March 26, 2007

Beryl. The Windows Aero?

Beryl is a windows manager for Ubuntu Linux that puts some eye-candy on your desktop. For those people not aligned with Linux and subject to marketing efforts of Windows and having been woo-ed by Aero only, this is Beryl:

http://www.youtube.com/results?search_query=beryl&search=Search

Beryl uses OpenGL to animate all those pretty effects. The number of options you have for customization is absolutely astounding. You can configure all effects for window events and then also configure individual timings and what not. Beryl also shows the X Cube, which is an invention of X/Linux. This cube helps in choosing your workspace.

I'm running Beryl on both NVIDIA 64MB card at home and my 128MB laptop computer. The 128M laptop is quite a bit slower and to remove very large windows with effects you notice the extensive cpu load, but still if you believe the video's, the performance should still be better than Aero.

Some people have loaded videos on YouTube that show how Beryl continues to run properly on standard current computers and do not need these "special" Aero graphic hardware that now seems to be demanded by Windows Vista to run with Aero. Have a look for yourself:

http://www.youtube.com/watch?v=xC5uEe5OzNQ

So, I guess we're finally here then. Windows Vista sucks and is already getting discounted:

http://arstechnica.com/news.ars/post/20070323-microsoft-announces-more-discounted-vista-licensing.html
My advice is... if you are looking for something new... get Ubuntu! I am using it on two computers right now and there is no reason for any of my productivity apps to move to Windows. The only reason anybody should ever have to move to Windows is for games or multi-media editing applications (Yes, there are plenty available like Kino, Cinelerra etc..., but their usability and stability is not very good at all).

Actually, you can move *now* to Ubuntu, but go straight to the distribution named "Edgy", which is the latest release. As soon as Feisty Fawn comes out, you'll be able to upgrade.

If you are that person that will have to reinstall anyway:

http://www.extremetech.com/article2/0,1697,2082982,00.asp

you might as well consider it in your "upgrade". I haven't even told anyone about the features of "apt". Imagine a very, very, very, very large repository of software on a server. And this server is connected to the Internet. This server contains all definitions of package dependencies, compatibilities, kernel versions, distributions etc. available. And then from your PC, by typing in:

apt-get update
apt-get upgrade

*ALL* (yes, *ALL*, not just kernel, not just the "main" Linux software or packages, but *ALL*) the software on your computer is automatically upgraded to the latest version possible, taking into account dependencies on other packages, taking into account if you have removed certain end-softwares that no longer use a certain other piece of software.

That is APT.

So, get Edgy from here:

http://cdimage.ubuntu.com/releases/6.10/release/

burn your CD and go for it. When Feisty comes out officially, you will be able to upgrade over the Internet (yep, free and gratis!). Then start reading from info here:

http://ubuntuguide.org/wiki/Ubuntu_Edgy

http://www.ubuntuforums.org/

Also use Google. Type "+Ubuntu" at the beginning and anything else that is of your concern. The reason is that this normally shows the ubuntuforums and other information. Notice that the Ubuntu people generally provide easier and more generic methods to resolve your problems. If you go with other distributions, the resolutions are generally more technical in nature.

http://www.google.com/search?hl=en&q=%2BUbuntu+%2Bberyl&btnG=Search

Have fun!

G>

Saturday, March 24, 2007

Custom loader in Ubuntu Linux

I got a bit bored by the static-ness of the whole thing and decided to create a custom loader screen for use at the company I work for.

Here are details on how to get started:

http://www.ubuntuforums.org/showthread.php?t=323520


Well, I'll also post a couple of details on commands to run:

make (to make the .so)
cp your-custom.so /usr/lib/usplash
ln -sf /usr/lib/usplash/your-custom.so /usr/lib/usplash/usplash-artwork.so
update-alternatives --config usplash-artwork.so
( select yours )
dpkg-reconfigure linux-image-$(uname -r)
( recreates initramfs )

That should do it.

Very important thing. There is no need to restart inbetween your tests. It is not very easy to find, but usplash has a testing mode. (Alt-Ctrl-F7 gets you back to X)

usplash -c

Here is some source code that shows some simple animation to get started with:

http://www.ubuntuforums.org/showthread.php?t=278301

G>

Friday, March 23, 2007

Guice in GWT

I am separating some functionalities in Project Dune. Transaction control was initially handcoded in each service request, but there were two problems here. The service request started the transaction in an HTTP specific stub and the business logic was mixed with this protocol-specific code. So the separation puts the business code into a separate object and transaction control is managed on any method that this business object does.

The first attempt actually was a transaction filter that always creates a transaction and then closes it, but this is expensive and one problem with GWT is that the SerializableException gets consumed and serialized as a response stream. So the filter will never see this exception being raised.

I have thus used Guice (pronounce this as "juice") to deal with these problems. The way it works is like this:
  1. A servlet context listener is used (see web.xml configuration) to initialize a Guice Injector and bind this to an attribute into the servlet context.
  2. A servlet filter always opens a Hibernate session. (not a transaction!). It will also (try-finally) close this session.
  3. When a GWT service needs to service a request, Tomcat will create the specified servlet.
  4. Each servlet derives from a BaseServiceImpl abstract servlet.
  5. The BaseServiceImpl overrides "init( ServletConfig config )". Through the "config.getServletContext().getAttribute()"
  6. I am retrieving the injector object, created in the listener.
  7. The injector object calls "injector.injectMembers( this )", which will "instrument" any annotated members in the servlet instance.
  8. When the injector sees a request to inject a field or method parameter, it will look this up in the registry and also attempt to inject any annotations that may exist in the to-be-injected instance.
  9. And so through very simple annotations, it may result in a couple of cascaded injection requests when a servlet gets instantiated.
The very nice thing about Guice is that you no longer have to deal with XML files. It is all programmatic. As soon as you have your "injector" instance, this instance will have been configured with a couple of bindings. Those bindings have matchers on classes and methods and if it finds anything that is annotated, it will perform that instrumentation.

Notice however that Guice is *not* installed on the classloader. This means that just setting "@Inject" on a field for example will not do anything *unless* you retrieve the instance through the injector instance. This latter part is not very easy for
everybody to understand right away, but is the most important aspect (no pun intended) about Guice programming as I have found so far.

Code example? You will need aopalliance.jar and guice-1.0.jar for this to run, downloadable from the Guice website:

ServletContextListener:
================
public class GuiceServletContextListener implements
    ServletContextListener
{
    public GuiceServletContextListener() {
        super();
    }

    public void contextInitialized(ServletContextEvent servletContextEvent)
    {
        ServletContext servletContext =
            servletContextEvent.getServletContext();

        // Create our injector for our application use
        // store it in servlet context.
        Injector injector = Guice.createInjector( new TransactionModel() );
        servletContext.setAttribute( Injector.class.getName(), injector );
    }

    public void contextDestroyed(
        ServletContextEvent servletContextEvent)
    {
    }
}

TransactionModel:
=============
public class TransactionModel implements Module
{
    public void configure(Binder binder)
    {
        binder.bindInterceptor(
            any(), // Match classes.
            annotatedWith(Transactional.class), // Match methods.
            new TransactionInterceptor() // The interceptor.
        );
    }
}

Transactional:
==========
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface Transactional {
}

BaseServiceImpl (a base class for any servlet in the application):
==========================================

public abstract class BaseServiceImpl extends RemoteServiceServlet {
    .....
    @Override
    public void init(ServletConfig config) throws ServletException {
        super.init(config);
        Injector injector = (Injector)config.getServletContext().
            getAttribute( Injector.class.getName() );
        injector.injectMembers( this );
    }
    .....
}

CustomerServiceImpl (implementation of GWT service):
====================================
public class CustomerServiceImpl extends BaseServiceImpl implements CustomerService {
    ......
    @Inject
    private CustomerBO customerBO;
    ......
    public CustomerDTO saveCustomer( CustomerDTO dto, boolean isNew )
        throws UIException
    {
        try {
            Customer customer = customerBO.getCustomer( dto.getCustomerId() );
            if ( customer == null ) {
                checkAccess( WebConstants.CUSTOMER_ADD );
                // not found, so create it.
                customer = new Customer();
            } else {
                checkAccess( WebConstants.CUSTOMER_EDIT );
            }

            MapperIF mapper = DozerBeanMapperSingletonWrapper.getInstance();
            mapper.map( custDTO, customer );

            customer = customerBO.saveCustomer( customer, isNew );
            CustomerDTO dto = customerBO.getCustomer( customer.getCustomerId() );

            return dto;
        } catch (ApplicationException ae ) {
            log.error( "Could not save customer", ae );
            throw new UIException( ae.getMessage() );
        }
    }

    ......
}

CustomerBO:
=========

@Singleton
public class CustomerBO extends BaseBO {
    ......
    @Transactional
    public Customer saveCustomer( Customer customer, boolean isNew )
        throws ApplicationException
    {
        Session session = getSession();

        try {
            if ( isNew && customer != null ) {
                throw new ApplicationException(
                    custI18n.getMessage( "error.cust.already.exists" ) );
            }

            // validate will raise an ApplicationException if the customer data is invalid.
            validateCustomer( customer );

            if ( isNew ) {
                session.save( customer );
            } else {
                session.update( customer );
            }

            customer = getCustomer( customer.getCustomerId() );

            return customer;
        } catch (HibernateException he ) {
            log.error( "Could not save customer", he );
            throw new ApplicationException(
                custI18n.getMessage( "error.save.customer" ));
        }
    }
}

==========================

Obviously this code can/should be extended with a variety of things. It should probably check if there is already a transaction ongoing. It should probably add parameters to the transaction interface to find out how the method supports transactions (required, supports, requiresNew, etc) and so on. But for simplicity's
sake, this is the bare minimum.

Notice how, once you have started the injector in the ServiceImpl, the CustomerBO does not need to be declared specifically in the injector. This is some sort of automatic cascading injection effect which happens because the injector is already processing dependent classes. So, luckily, you only need to use the Injector once, which will inject all your other classes where you want them.

Also have a look how to do this for interfaces. What is lacking in the CustomerBO is a separation of persistence with business logic. If you separate this further, you have a start to be able to switch the implementation of your persistence framework.

I am personally contemplating to bring the transaction boundary more forward and wrap this around the methods (where required) of ServiceImpl instead. But I am not sure whether this will work.

Good luck on your own incantations!

Thursday, March 15, 2007

Gluon

One of my colleagues has just released a version of "Gluon", which is a plugin for Eclipse that allows you to develop in BREW.

" GLUON is a fully open source IDE for BREWTM, based on Eclipse and CDT plugin. It increases your software quality and productivity by providing high-value features that are not available on the current development environment. It leverages BREWTM development on companies and individuals, making development process easy, enabling developer to focus on application."

Check it out. If you are into any kind of BREW development, you'd certainly love to see the features and ability to debug.

(btw, the design for the page was made by another colleague in only a very short time before launch).

The plugin will be discussed at the EclipseCon. Follow the link on the page.

Monday, March 05, 2007

Conversation with the machine

The way how machine communication is evolving is starting to change, more into a conversation with hints provided by the system to improve efficiency then a list of operations to execute one after the other. The machine initially started as a processor to receive a list of holes on paper, which would process inside the machine, with the output of another paper element that had values on them.

With the introduction of the desktop and the monitor, the way to use a program would be typically to prepare a file or a command with options on the command line and then hit Enter. The program would execute and do something, either print on screen or print on the printer.

The GUI in Windows and Apple in the 80's and 90's made this slightly more interactive, with generally the limitation that it did not have access to worldwide information immediately. The GUI could however look into databases or execute other processing whilst the user was active on the screen. This is a very important point.

The web applications in the 90's started to come up as well, whether intranet or internet. These applications would typically be a mixture of serverside script and client-side script for some simple validations. Now, the user would typically not have to install and maintain a binary on the client side, which facilitates a lot for the maintainer on the server side. Everybody could just log on and use the services without many problems.

In Web 1.0, HTML however, we would generally prepare a form or a screen that we only submit at the very end of our operations. In a way, you could say that we would have to know how the server would process the information, more or less, before we could efficiently use that screen. Will this ever change? I don't know yet, but it's possible.

With Web 2.0, the way how we interact with servers is slightly changing. Maybe it's HTTP technology or we haven't reached the immense bandwidth yet that is necessary, but I see slight trends in which our conversation with the server is getting more continuous (or approaching a continuous state).

For example the suggest box. The system in its entirety already goes off to the server after a couple of keystrokes and from there, it might all start to change the conversation. Maybe it will hide a couple of fields thereafter or it will allow new functionality to appear from there on.

The changes in this field are that we are becoming more integrated with information overall, but let the machine help us better to use the system (and ourselves as society) better and more efficient. This is the power of IT. It is quite interesting therefore to know how scalable continuous conversation systems are. The messages are typically quite small, but involve XML translations, bandwidth and a good deal of common sense architecture and timeout decisions.

So, the point is to show that initially, human computer interaction was quite basic and the human operating the machine had to know intricate details about the functioning of the program. Over the years this is changing, with MS-Word for example we could already more or less just use it as a rich typewriter. It probably can be made more efficient for example by letting the server find similar phrases in other books to avoid claims of plagiarism, to give a weird example.

So, a probably useful addition to how we interact with computers is that we allow the server to give us hints. Make it as if the computer is looking over our shoulder what we are doing and analyze our intent whilst we are interacting with it. Suggest improvements or more efficient work will let the machine teach us. A more powerful example is where the machine learns from the wisest users automatically and then makes the efficient suggestions to less efficient users of the system.

How can/will/should user-machine conversations evolve from this point onwards?

Thursday, March 01, 2007

Scrum project management

I haven't written a lot here, nor have released any new version of Project Dune because we are writing a lot of new functionalities. This image shows a crude task burndown graphic for SCRUM project management. The module that we created basically is aimed to allow users to very quickly manage the states of their tasks for projects they are assigned.

It is time for a new release though, so very soon you should expect some new movements.

The project is also expanding in active members. We have three active developers at the moment, working on a timesheet module, scrum project management and evaluating quality, unit testing etc.

Tuesday, February 13, 2007

Carnaval in Recife

It's getting carnaval again. Starts this week. On Friday, everyone will power down the machines and they start a party in front of the building.

The company has a "bloco de carnaval" called "Dá O Loud". It's a joke between making a lot of noise and "download".

Saturday, Recife organises "Galo da Madrugada" (see picture). Imagine 1 - 1.5 million people on a small island... it's quite insane really. Anyway, I'm going to a different kind of carnaval more in the country-side (on sunday). It's in Bezerros and should be quite good.

The other days? I probably would work a bit on personal projects, take it easy, go to the beach, play some video games? etc...

Tuesday, January 30, 2007

Project Dune now stable...

Project Dune went stable yesterday. It still needs a detailed customization guide and a user's guide to be really complete, but the main beef is there for people to use and to provide feedback on.

The next steps for this project are aimed at separate modules for SCRUM project management and document management. Naturally, those modules will have certain breakout points as well to allow integration with Project Dune.

The project itself will slow down in new features and aims to resolve issues that the user community will (or should) raise. Then with some more stability under it, we can take it to other levels later on.

Friday, January 19, 2007

It's not all the same kind of development

I have had the opportunity to work on many different kinds of projects, and you cannot treat software development for embedded hardware the same as web server application development, not by any long shot.

Embedded software, operating systems, low level applications are 70-80% preparation in design, memory layout, memory management, documentation and planning. In contrast to high-level languages used on webservers, etc. The more low-level and smaller you go, the better you need to plan the software, since the tools available (or rework) to modify changes is going to be more difficult. And the lower you go in the system, the more other systems built on top are likely to be severely affected.

So, 70-80% planning in embedded systems with not a lot of coding (just make sure you *do* it right). This is against 20-40% of planning for application software, but requiring a lot of coding effort. The high-level languages have plenty of libraries available and the development tools help significantly with refactoring nowadays. Planning beforehand for the software is actually a risk factor, due to changing requirements from the clients and changing insights along the development process, or a change in the way how things could/should be done.

Therefore, one could "perceive" embedded software development as a "slow" process, but this is not truly the case. A design is in itself a produce of intellect, but it does not immediately convert itself to a materialized ability to the end user. Nobody however can deny that a good design produces immense benefits in the long run for a certain operating system or function.

The final argument has to do with update cycles. Embedded software *has* to be correct. Webserver or PC software can be updated (should not, unless for new features), but we have sort of generally started to accept that for PC software you produce a version that you think works and then continuously update the software after it is released. Everybody does it, so why not you? With embedded, this is simply impossible, since the user does not generally have the ability or skills to do this. And in general we tend to think of a product as "bad" if they have to return it to the shop for 'an upgrade'...

Thursday, January 18, 2007

MS Vista vs. Linux source code control...

MS Vista has been delayed a couple of times and this is often attributed to different factors. This article shows how Windows did the version management. Check it out:

You have a very large number of branches that sometimes need to be integrated. The whole point of these branches is to establish "quality control exit gates", where the code will not pass upstream unless it passes the quality control check. This is a nice thought and that part might work, but there is another consideration for quality to be made here... immediate synchronization with other code. Now, the person writing this article is convinced that without this level of "management", nothing will ever ship.

Consider the Linux kernel. There is basically one mailing list and one mailing list only. There exists an idea of "separation of expertise", but not really "separation of concern or by branch". This does not mean that branches are not used, but they are temporary. The only longer-lived branches are kernel version branches (2.2, 2.4, 2.6), so you have different streams there.

The mailing list is considered total chaos (if you live up to MS terms). Everybody has access to every comment, even if they don't work on it and comments on everything. Filtering on your area of interest may be quite difficult, but in the end you can gather insight into everything that is going on. Messages in the middle establish "code freezes" (sent by Linux) when another version of Linux is to be closed off.

What I am ignoring so far is that Windows is much more than just a kernel, so the comparison is not truly fair. The problems that Windows may have faced in Vista is not necessarily "quality control", but immediate access to new features and managing the streams of code in real-time. Now, I am not saying there is a solution to this dilemma, but there is a problem to be solved.

If you rely on the quality gates, then a new feature that you depend on may only arrive after two weeks. Forget about putting pressure. Then you need to start a political fight with another department with a different budget and different bosses to get this feature in or *your* project will delay. Truth is that it's not truly the concern of that other department.

Open Source does not have a deadline and no requirement to make everything backwards compatible. Windows this is highly desired. So Windows needs to maintain backwards-compatible API's using a new structure of the kernel, but "looking" the same. We all know this is an impossible task...

So... we can argue... is the shipping of Windows Vista truly a miracle and good management, or could it have shipped way earlier if they would have managed things differently? Their personal experience shows that the way how it initially worked wasn't actually working very well...

SCRUM

There are different methods for project management besides the "Project Management Body Of Knowledge". The ancient "accredited" methods put a single guy as the responsible person for finishing and executing a project. This is just an observation at this point.

"Scrum" is another way of "managing". Actually, it is not really management, but mostly delegation. The scrum method is derived from agile project management and the team becomes responsible for its planning, estimation and execution. Well, the team led by the scrum master, sort of like a quarterback.

What I like about the method is that it allows you to extract information about the daily progress. For example, each task that is defined in this method will be estimated for a single day and one day only. There are no tasks that take two days. If there is one, it will be broken down into two parts. This facilitates management and tracking.

In the regular project management, a task may sometimes take 3, 5, 7 or 20 days. In the minds of some developers, this gives them "slack" and sometimes too much slack that the time is not properly controlled. At the end of the task, it may either be finished or backlogged. Imagine the surprise... And in steps the project manager to negotiate with the client.

Having it broken down in single tasks really facilitates tracking. The cool thing here is that the "whip" is no longer in the hands of the project manager. There is a public board on the wall that shows the tasks to be done, who is working on what and there is therefore transparency on other people's performance. This sort of means that everybody in the team has an invisible "imaginary" whip. Even though you wouldn't publically make anyone accountable, the people involved perceive a certain team pressure for complying with their agreed tasks and feel thus pressure to finish. This makes the integration within the team a lot stronger if the tasks *are* completed and opens up an interesting discussion if tasks are *not*. (this is where people management comes into play).

In the context of "Project Dune", I am planning to build a separated module for this type of management, so that electronically it is possible to track projects this way. There are quite a number of manual tasks involved still that should be automated.
(feature point management, project burndown, product burndown and impediment burndown).

The whole essence of scrum is making a sprint of 4 weeks and putting your foot down to complete it and manage your difficulties (impediments) plus tasks on a daily basis. This really helps to integrate your team better and establishes a much nicer team atmosphere than other management methods can do. However, this does depend on the commitment and attitude of the team.

Wednesday, January 17, 2007

More on OS development...

I'm just playing around with the toy OS and learning quite a lot about how a computer works and what the kernel for a computer really has to do. I am not even working on device I/O or advanced features at all yet. Just memory management.

Memory management is subject to the following considerations:
  • Where to put the kernel code and data (you do not want to overwrite this)
  • Where to put the GDT and IDT
  • Where to put the memory "page directory" (full 4GB directory == 8 MB)
  • The ratio between kernel space vs. user space (linux = 1:3)
  • How to implement an optimal "malloc" routine
  • Page swapping (but not required, a kernel may also fault)
  • Optimization of the above requirements, like is done in Linux > 2.4
  • Memory page sharing through copy-on-write for example (by default the process shares the same code + data, but on any attempt to write to the page, the kernel makes a copy of that page)
So... great... lots of considerations. I am just at the part of where to put everything. My decisions haven't been ideal yet, because of boot-up limitations mostly. But slowly I should be able to work towards page allocation for a process, assigning it to a process and then subsequently the ability to run a certain process (albeit hardcoded in the kernel itself for the time being).

As said before, this is not going to be anything big, but will greatly assist in understanding other OS's a lot better. Maybe even I could use this knowledge to jump-start another project for a limited device and make the machine do something really specific in an easy way.

G>

Monday, January 15, 2007

How to start writing your own OS

Writing a fully fledged Operating System is not a task that can be finished by a small group of people, but it is still possible to write one that gets near to "DOS" kind of functionality, all by yourself. What you put in... you decide! It probably only runs on your hardware, or it doesn't make use of specialized features of hardwares, but you should ask yourself what kind of things you want to have for that particular OS and code away!

The tutorials here show you the start of writing one. If you get into the downloads area and look at one of those tutorials, you can even get a template OS to start from. Notice that Linux 0.01 is also part of the list and is a good read (and a chuckle!) to work from.

The first thing to do when writing an OS, I think, is "choose your tools". If you don't get them right from the start, you'll spend significant amounts of time rewriting it later. I am using gcc + nasm, since these are widely available and free. Then, start by building a hard disk image using the info from the site here.

You probably do not want to get started with a boot loader (it's really complicated!) and with the boot loaders available now, why should you? I did the following on Ubuntu Linux to create a hard drive image to start from. You go up to the point where they start to copy the guest OS, because that guest OS... is basically your real OS that will run inside qemu (more about qemu later).

http://wiki.xensource.com/xenwiki/InstallGuestImage

In the steps where "grub" gets installed... I had to manually copy a menu.lst into the "/boot/grub" folder. It didn't get created automatically. Maybe I should have "chroot-ed" instead into the "/mnt" and then try it again. If you take care of this image, you only will need to do it once.

Anyway, I copied in a menu.lst that looks like:

default 0
timeout 3
hiddenmenu

groot=(hd0,0)

title
root (hd0,0)
kernel /kernel.bin
boot

Of course you will need to replace the "kernel.bin" every time you recompile. I have this in my Makefile as such:

losetup -o 32256 /dev/loop1 ./test/hd.img
mount /dev/loop1 /mnt/temp
sudo cp kernel.bin /mnt/temp/kernel.bin
sudo umount /mnt/temp
losetup -d /dev/loop1
losetup -d /dev/loop0


And then you can test the OS through Qemu (better than VMWare, since it has a shorter time for startup and it is free):

qemu -hda ./test/hd.img -boot c -std-vga -localtime

Qemu is installable through apt, but the kernel module needs to be downloaded from the site. No worries... plenty of documentation there. On my Ubuntu Edgy I downloaded, untarred, ./configure-d and used make; make install and it all worked.

The cool thing nowadays is that you can downloaded various OS's from here and there and merge the concepts and see what happens. Also, an OS is not necessarily multi-user or multi-tasking, nor does it necessarily need to page out memory to disk and back again (you can simply page-fault and panic).

Unless you have a specific need or idea to follow, I wouldn't even start to believe it would be the start of the next "better" OS... But it's good to write one "OS" in your life to better understand how they all work.