I have not been up to my usual level of productivity in the blogosphere of late, and here I must claim that the demands of the working world have exercised more than an undo influence on my time and energy.

That now has past. The project I was leading was a technology transfer effort involving the move of 16+ TB of data, 30 users, and 25 new desktops from one environment to another. The difficulty was compounded by the fact that we were also moving from one Windows domain to another, most of the user names were changing, and all of the user desktops were being upgraded. Not, altogether that difficult, but a fiddly effort with many moving parts and a few things that my organization had never had to undertake (moving and developing a cost effective back-up strategy for that 16 TB of data, growing at a rate of 6-10 TB a year).

With a project post mortem (we finished about a week earlier than planned) upcoming, I’ve been reflecting on some of the “facts” about being an IT professional that this project brought once more to my attention. Here are my top five for the current effort:

One: Your Job Isn’t What You Think It Is

Unless you are concretely working for a company whose business is directly tied to software development and support, your job is at its essence a service role, not at IT role. I think now most IT professionals understand this at some level, but often enough our business partners are seen by the technologists as inept in technology, hide-bound in their processes, and just not knowledgeable enough to make the right choices, and the business users consider IT staff as unhelpful, arrogant, and ignorant about the business.

If that rings true to you and your business partners, you have a lot of work to do in understanding what their actual business needs in IT are and how you can work with them to provide them. If you think that the issue is that they “can’t state their requirements”, and always want something “bright and shiny”, then my friend, the fault lies as much with you as with them.

Your job as an IT professional is to understand the business you’re in. Yes, that’s right, actual domain knowledge/experience. It’s up to us to meet them more than halfway, be supportive always and visionary when necessary to help them use the technology we can provide in ways that add value and support the constant improvement of everything that’s done.

Two: “Just in Time” Analysis Beats “Waterfall” in Real Time Events

In projects like my current one that involve a technology transition in the midst of the business continuing to do its actual work, a “waterfall” methodology can be only partially successful because almost everything must be done in parallel. For example, the 16TB of data not only needed to be moved, it needed to be refreshed as it was being added to by the business during the whole period. Not just nightly synchroniztion, but also procedures to ensure 1) the correctness of the copy at the file/directory level, but also 2) at the file integrity level.

Similar issues with permissions, firewall rules, desktop builds: we could plan that these things would happen, but could not know all of the details until the very last minute if we were to keep the business up and running during the whole process. Most users experienced less than a hour of down time on the desktop, and one day of use of a central application — I’d call that a pretty successful effort.

Moral: get the big picture right at the start, figure out not just what you need to know, but also when you need to know it.

Three: You Get It Right the Third Time You Do It

This, as a technology manager and system developer, has long been one of my tenets: by the third iteration of any complex system you have the experience with both the system and, especially, the data, to know how to accomplish what needs to be done. At the end of the day, most data-intensive and data-centric system fail because the actual data is not as the business or business analyst has described or even imagined it.

In this project, the biggest test was to develop software, and then procedures, to ensure that the massive data copy was performed correctly at both the file/path level and at the file level, and these tests needed to be performed on both the origin and target systems.

Even though most of the requisite software had already been developed, we needed those three iterations to feel confident about our work. Moral here: plan for three iterations if you need to develop custom scripts/code to help you accomplish your work.

Four: An “Installed Base” Mentality Can Be Fatal

The biggest obstacle to success can often be the “installed base” mentality on the IT side. That is, what happens when the “here’s what we’ve built/implemented to solve your problem” butts up against a problem that appears to be, but isn’t the actual problem at hand.

In our case, it had to do with the solution central IT offered for data archiving. This was an inflexible, very expensive solution (archiving cost 6 times the cost for spinning disc) put together for a completely different use case. Our business users (a museum) saw their digital assets as being something to be held/maintained in perpetuity, a far different solution than IT was able to offer. While they are looking at cloud storage options, a very attractive choice from the business’s point of view, such a solution is months away, and we will be looking at other interim solutions. Long-term, this is a problem that no one in the library/cultural heritage business has been adequately able to address.

So, for this part of the project, central IT had no acceptable solution, and it took many rounds of conversations for them to realize this and begin to work towards other near-term solutions, solutions which we have yet to decide on and implement

Five: Done is Not Done

Ok, so we’re not done with solving the long-term archival needs of our business partners. This is essential, but will require more work. We’re also not done with getting all of the permissions right, but we’re very close. We think we’re done with the transfer of all of the digital content, but 5TB of the 16TB of data are duplicates, so we need to work through how we address deduplication in a way that makes sense to our users. We think we’re done with the database moves, but each new day seems to bring yet one more thing that didn’t get updated with the move, despite help from the vendor (our efforts have helped them develop their own internal documentation for such future moves by their clients). And then there’s all this other stuff that we now need to take on because it was put into another, follow-on phase. Not to mention all of the new stuff, here way too numerous to mention.

So, not only is there no rest for the wicked, if you’re good in IT, it will only generate more demand. So that’s a good thing, but not a thing on which one can rest.