I have been on a Data Management team for almost 9 months now. I am part of the technical team that complete many of the requests initiated by the business. When I first started, the choice over tools to produce reports are fairly standard, upstream teams populate a number of staging tables in an Oracle database in a schema belonging to our team.

We, the technical data team use either Toad or sqlplus to extract and data from tables and produce the necessary reports based on requirements. Sounds pretty straight forward right? Well sorta.

Now some other context for the team. The infrastructure we use is shared amongst many other teams whose activities often take priorities. Often we notice instances of instability with the Oracle database which in turn cause frustration and delays with our work. Unfortunately this is something we had to live with for the time being, and I believe this is often the case for many projects out there.

Around early December last year, it was decided to provide DataStage access to my team. This was intended to help alleviate some of the strain put on the Database, and hopefully migrate some of the intensive processing to a Sun Grid environment where DataStage jobs are executed.

This brings me to the Agile part of this post. As a technical team we are faced with a number of tasks to do while at the same time maintaining standard operational duties (i.e. keep those reports coming). DataStage as a tool and an environment was newish to the team overall, but not completely unfamiliar of.

So the team is under some pressure if you could understand. We are given a new tool to learn and use, we need to convert some of the existing reports in SQL to DataStage, and at the same time re-structure (at least the Data part), so that we can better utilise the infrastructure as to put less load on it.

At the time, I thought some of the Agile methods I have learnt may be relevant in this instance.
So this is what we did do:
1. All reports are analyzed, with an aim to streamline the overall process. Due to the time constraints, we thought to ‘pilot’ some of the most urgent and pressing reports. This is similar to identify the ‘must-haves’ features for a software.

2. Reports are assigned to individual developers with the aim to convert them to DataStage jobs. Every day at the start, we did a quick stand-up (~15-20 minutes), to track progress. I had all the development tasks on a whiteboard, and I encouraged developers to update them instead of emails. This actually helped the conversion process a lot, as we start to discover synergies amongst the reports which helped with 1. Whole iterations are about 2-week long, with concrete deliverables.

3. We failed early, a couple of things that we did in SQL was deemed technical challenging to do in DataStage, so we decided to have hybrid jobs where much of the heavy lifting of data processing is done in DataStage, and the remaining is done in SQL.

4. We adopted pair-programming. Initially when we first started on developing DataStage jobs, we generally have 2 people work on the job at the start. And when it has progressed to a degree, the team members may be inter-changed as to foster knowledge sharing. This seems to happen naturally for us, we already have a peer-review process in place.

Now the outcome, over the past 3 months I find things go a lot smoother for us. We still had the occasional database downtimes, but we were still able to mitigate much of the risks around that. There have been times also when the DataStage grid environment was heavily used, so we had delays there too. But the redundancy of data being available in both the database and the Grid environment, meant that we have more flexibility. But this is more a by-product of what has changed in the team not a direct result of being Agile.

Some of the real benefits of Agile for our team are:
1. We seem to have a far better understanding and visibility of what is available to us and what we could provide. Silos of knowledge in individual members are now shared much more readily and willingly.

2. We take far greater ownership of the work we have, comparing to the ’spoke-model’ where work is directed from the team lead. We seem to have more parallel streams approach to work.

3. Mistakes and errors are detected early and resolved early, this combined with 2. meant we achieved higher quality in the reports that we deliver and in shorter timeframe.

Development of DataStage jobs follow a ‘promotion’ model. Where individual developer’s sandbox is used for much of the development activities, and when the job is peer-reviewed and deemed good enough for prime time, it is then ‘promoted’ to the released folder. We also have a ‘deprecated’ folder to renew some the jobs.

The next activity I’d like to focus is to concentrate on further align the data we have with well designed reporting schema that suite our needs to further reduce the load put some of the common data we use.

As always your comments are welcome, and would certainly love to know what you did or tried when your technical team is faced with something similar like ours.

~paul

It was a night on a typical weekday. I relaxed on my couch, and Phyllis my wife trying to upload some photos to Facebook, she had been on it for well over half an hour now.

Haven’t you finished yet? I asked casually.

It’s going, it failed once before, now I just have to make sure I have no other windows open, and it will work.

You sure about that? It’s never failed on me. Naturally like all helpful and somewhat sceptical nerds her response received the typical retort. I leaned over to peer over her shoulder.

Look, it’s all going fine, leave me to it, don’t you touch anything! Phyllis put her hands out, covering the laptop’s screen, at the same time shoots back one of those sharp stares that all sensible husbands ought to know better than to interfere.

This brings me to the topic of today’s post. Yes, why can’t I just trust her to it? Is it paranoia? Is it the fact that I know I can probably do it faster than her, and her reasons to me made no sense? Is it the feeling of hurt that she knows I just can’t trust her with it?

I think conversations like this take place everyday and at a greater scale, conversations like this occurr in boardrooms and stakeholder meetings.

The divide between the IT and business in most organisations are still there. The mistrust is obviously not always ill-founded. We all remember occasions when someone from the business side had this brilliant idea and only to be turned away because the voice of doom (wait replace that with the architect) said ‘you wanted to do what?!’

This attitude has to change on both sides mind you. I draw an example from my current work, where I work very closely with my business stakeholder. He has a report to present to the business, and I am here to prepare that data. What is the best way to present the data? He is best placed to answer that, but he can’t answer that alone, he needs to know what data is available for him to use, and timeliness of it too, and that’s where I come in. We had 2 weeks to hammer this thing right. I released about 4 versions of the report to him in 2 weeks, each time seeking direction, and deciding what to keep and what to lose.

He is informed every 2-3 days, after the presentation of the report, I always touch base again to see how the last report is received with the business. This rapid and regular feedback cycle is key to us bridging that trust gap often seen between IT and the business.

There is clearly benefit in us co-operating, and invariably this helps the business to ‘care’ about how I get things done, because there is direct benefit in getting involved. I do borrow many good practice Agile development in this respect, and part of that Agile story involves a lot of talking and listening. This is part of that journey between business and IT that both side need to partake.

So Phyllis you should trust me with it

~Paul

Recently I was given a number of sites to look at.

I felt the time dial winding backwards. A cursory glance of the source revealed that it complied with HTML 4.0, hang on or was it xhtml 1.0 transitional? Anyway it probably didn’t matter.

There were some standard meta tags in the header, fairly vanilla, I assume to make the search somewhat easier, which means that the search was not based on the content of the pages but just with the infos in the meta tags. (I could almost hear someone say how 1990 is that?)

It was a fairly standard affair as for the layout, a header menu, a side bar of navigation links, a copy-right info down the footer, which are of course done with a minimal of table and div tags. It had no dynamic content as far as I could tell, no javascript doing asynchronous pull of info, no flash widgets no nothing.

The search seems to be calling a CGI routine which seems to pull a standard export of crawled content. Again, no page ranking, no what others have clicked on, no similar pages, no sorting of content by say the newest page to the oldest, FYI the first link was from 1990, while the 2008 link was on the 9th. The search layout and styling is not consistent to the main landing page, in fact I should say it is completely different

And here is the clincher, this site is open to the general public, you would have thought they could make this at least a little bit friendly and a little bit more … well ‘up to date’.

That aside, in this day and age, if you have even but a tiniest presence on the web, then you are your own social media company. Everything that you put up there is a direct representation of you, whether you like it or not. So be it you are a private company, a government agency or a self-starting entrepreneur, you need to be aware of what is going around you. A half-hearted treatment on any site just won’t do. (Ahem, especially those important government sites) Your site is the new social marketing tool for yourself, your agenda, your goals, and social it certainly needs to be.

This blog for example, I had friends telling me what I write up here during casual catch-ups, its content, its presentation, I as myself is in full view of the general public, to their critique, their scrutiny, and most important their support. (Maybe that’s what I think too long before writing anything hehe)

~paul

Today is 1st March 2009, it marks a psychological point in my mind that 2009 has well and truly started. (yes for some of us, it takes a little more prompting than others).

Only last week, the haze of smoke that swept through the suburbia of Melbourne reminded all of us that, lying beneath its stunning beauty, there lies another side of Australian nature, forever untamed and relentless.

Houses burnt, dreams shuttered, but the human spirit soared.

Gripped by fears, jobs lost, factories close, leaders pledge their visions and money, countries forge their pact.

I am trying to spend more time in my garden, growing vegetables, and reconditioning the soil. The ferns are growing well under the shade of the trees in the front yard. Our neighbour helps us with some tomato seedlings. We have had several good harvest of bok-choys and the beans are looking good. I got some sea weed fertilizers putting them deep in the soil, as I listen to Ruper Murdoch’s Boyer Lecture

And I wonder the ramifications the world climate might have on my tiny garden, listening to those postulate on the implications of a climate war on countries.

Somehow the fact I managed to recycle half bucket of water from the weekend lunch for the garden made me feel that little bit better.

All is well and life goes on.

~paul

It’s been just over a year since the Li family has installed the York x202 anniversary cross trainer. It’s still shiny. It still sits near the back corner of our lounge. Please allow me to digress

If the cross trainer was a religion, then it’d have many of its trials and tribulations already. It dealt with the atheist TV that gloats with its big cathode ray tube and stereo sound. It dealt with the house dwellers whom are swinging constantly amongst ‘born-again’ “cross trainer faith holders”, alternative religious sects and non-believers that pay tribute to the atheist TV. (Well you get the picture at least the metaphors do).

Now as I inspect its wonders of modern sports science, its smooth leg struts tracing out perfect ecliptic, the electronic motor power train assembly, wiping the sweat off its heart rate monitor handles I know I have done well with yet another 30 minutes of ‘program 10′ fat burn #2 at the distance of 9.8 km. (Wow, the last time I ran that sort of distance I was at least 10 years younger and going for one of those fitness assessment in the army reserve…)

It has been a bit over 2 weeks, 30 minutes every day. As I pick up the good gospel, I start to notice my weight dropping, sweat flowing, and endorphins taking its sweet time in me. Yes I am a born-again cross trainer believer… (yet again) The sceptics abound as always. The red leather couch is laughing right now “you will be back, just you watch, your f*t arse will be parking right here in no time”

let’s how long this time it lasts… I found the sales receipt for the cross trainer the other day, its 1 year warranty has just expired

~paul

I am looking into enterprise architecture at work, and I saw this rather interesting article from 2006, quite old really by news standards or blog standards

Enterprise Architecture – Lessons from the Laudry room

What the author is trying to say is that the role of an Enterprise Architect is to understand the business impact of those technology decisions that we are grappling with every day. Maybe this is a good way to explain SOA, the Three Letter Acronym that brings out so much passion and hatred at the same time.

~paul

I found this game mod for Team Fortress 2, where the traditional soccer game is combined with the Unreal Tournament’s firepower! pretty funny indeed

The other day as I was driving home in pouring rain (typical Melbourne weather), stuck in not-so-typical bad traffic, I decided to catch up on my overdue pod casts, one of them being the ABC counter point program. If you listens to AM radio at all, I highly recommend taking a look at this program, as at times I have found this very thought provoking.

Last week’s counter point program interviewed Matt Mason, an ex-pirate DJ who recently wrote the Pirate’s Dilemma, The book takes an interesting look at the whole piracy issue in the music industry. And as you’d expect from a veteran pirate radio DJ, he has decided to make the book available completely free. Well… as you check out the book, you can decide how much money to pay him, and if you pay $0, your copy of the ebook shall be aptly named “thepiratesdilemma-pirate_copy.pdf” Nice one Matt

But Matt Mason delves much further than that, tracing back in history early signs of underground movement initiated by the public to go against the huge corporations. He terms this as punk.

Punk in its broad sense defines the underground subversive movement where the individuals defy convention, either by operating outside the existing legal framework or creating an entirely untested territory. And punk capitalists are those pioneers that took advantage of such niche and vacuums where traditional businesses or organisations fail to satisfy or fulfil that demand.

Given his background in the entertainment and music industries, most of his examples are from there. He went into considerable length in describing the activities of pirate radios where local DJs can experiment on alternative music genres. This ultimately can not be stopped by commercial radio stations as they can not cater for all music tastes and instead stick with the fail-safe option of the pop charts. Pirate radios on the other hand allows for specific focus on particular music tastes and become real incubators for up-and-coming musicians and in turn en-mass a considerable number of listeners. Once this has created enough listenership, the mainstream media cannot help but accept defeat in the most fluttering manner, i.e. broadcast music that originally was showcased on pirate radio. In fact many pirate DJs have gone mainstream and finding themselves running commercial radios after they have gained respect and acceptance in the pirate ether.

Matt then further proposes that history has shown such pirate movements before.

…Edison, in turn, went on to invent filmmaking, and demanded a licensing fee from those making movies with his technology. This caused a band of filmmaking pirates, among them a man named William, to flee New York for the then still wild West, where they thrived, unlicensed, until Edison’s patents expired. These pirates continue to operate there, albeit legally now, in the town they founded: Hollywood. William’s last name? Fox.

He obviously isn’t endorsing patent infringements. What’s really important here is that;

1. Technologies like peer-to-peer when used by pirates are ‘game-changing’ to the music and entertainment industry. How can any corporation compete at the scale and value which digital pirates are? The price free, mode of distribution the on-demand torrent network, which is infinitely scaleable and would dwarf any commercial distribution networks.

In that respect, musicians and artists in general are far more adaptable and therefore are abandoning record labels and using the internet themselves to distribute their content. No longer are they tethered by the strict contracts that record labels force them to sign, and they also have far more control over the revenues that their concerts and shows command. Many shows now broadcast freely on youtube, the latest US TV series are now made simultaneous available on the net, on air globally. Alternative revenue streams are sought rather than the traditional TV advertising.

2. Sometimes patents will impede the progress of innovations. Matt shows what’s already happening in our daily lives. Around the world, companies are patenting cure for diseases or genetic codes of 200 plus variety of wheat stocks. Such corporate greed is causing backlash in many third world countries, where its citizens can not afford the exorbitant prices demanded. Many countries have put up provisions in law where the patents concern genetic materials and life-saving drugs.

The idea that capitalism will thrive by individuals pursuing purely for their own self-interests are numbered. Increasing evidence is there that a sense of altruism that exists and prevails. That we do indeed derive satisfaction by helping each other to move forward.

Many open source software are made better by thousands of people around the world devoting free time and effort for the betterment of the general public. Many technologies are now developed and incubated in the open source world, before they get picked up by corporations. Why stop at music and softwares? How about the next battery operated car? People are sharing more than ever before and in the process becoming individual enterprises interacting with the mainstream organisations on level playing fields like never before…

Perhaps this is what Matt meant by punk capitalism. There will forever be punks and pirates, turning what’s happening at the fringes into mainstream, tapping into the trickles of information and transforming that into torrents. May there be a bit of punk in all of us

~paul

Last week, at work we had a number of sales people from Oracle to do a presentation, since I lamented somewhat about Oracle, I was curious to find out from them what exactly are going to be included from the BEA product suite into the Oracle suite. I was told politely to wait for the 1st of July announcement. I didn’t register for the webcast, but I did manage to squeeze in the 2 pod casts on the road with Thomas Kurian. Here are some take-aways for me, and I hope you will find this somewhat useful:

• Weblogic JRockit is here to stay, and existing Oracle products can now running on top of this very efficient JVM and the nowly aptly named Oracle Weblogic Application Server. Some features from existing Oracle App server will be incoporated.

Thumbs up for me on that one, you get no arguments from me here. JRockit is a proven solution, and a crown jewel from BEA. I am not too familiar with the Oracle App Server, maybe others can add their 2 cents here.

• Aqualogic Service Repository will be incorporated into the Oracle Fusion Middleware.

Well, you need a service repository to hold all your artifacts for your service as well as providing governance, and the means to publish such details to it, so the point is sorta moot here.

• Oracle WebCentre Suite (formerly Aqualogic User Interaction) will bring out a whole host of enterprise 2.0 capabilities that are going to be based on the Weblogic Portal servers.

This is very similar to what IBM just announced with its plans on the WebSphere Portal v6.1. I am interested to know what you think about this portal approach. Is it because many of the clients have already bought into portals so it is a way to recap the investment into existing infrastructure?

• Tuxedo is incorporated into as a key offering from Oracle as a transaction processing solution, making it available on more OS platforms, tigher integrations with Oracle database etc etc.

There are no surprises there, given that Tux has been used by a lot of enterprises for mission critical transactions.

• Enterprise Service buses from both AL and Oracle will interoperate and comply to the Service Component Architecture. The Weblogic Event server lives on in the BPM suite, everything else seems to come from the existing Oracle products.

I think AL has been following the SCA path for some time now, may be there are just some re-jig from the Oracle’s fence. That’s all from me, I hope you enjoyed my rant, and your comments are always welcome as always

~paul

I find GetSatisfaction, a Web 2.0 start up that aims to connect companies with consumers quite a refreshing idea. I thought this is a great idea, where users can submit ideas, problems and questions to any companies that care to listen.

In the past user forums have sprung up everywhere focusing on particular products, and those that en-mass enough users do get representatives from respective companies to answer some of the questions/feedbacks/complaints from fellow users. Whirlpool comes to mind, where it started off as a forum and news bulletin dedicated to broadband internet for Australia, very quickly as the user base expanded, so have the forums and the topics, from fridges to the latest Hunda CRV to the best bluetooth headset for your mobile phone… The list well goes on

Now back to GetSatisfaction, many companies have already created accounts where their employees can provide that direct representation. This is a great example where users that participate get immediate value for themselves, either by getting your questions answered, or learning new tricks from others who have used the same products. Needless to say, not all companies will openly embrace this, after all, it takes a great deal of courage from the corporations to meet its customers at the coal face, the consumers market is getting more and more transparent. Judging from the growing list of companies that there, it clearly leans more towards the technology and consumer electronics sectors rather than the traditional businesses… Things will catch on I am sure

I wish GetSatisfaction all the best, for I have already used it to rant about my Treo 680 phone to the company Palm. Now Palm are you listening?

~paul