Scope Verification of Software Projects

A reader over at PM Hut asks, "How [can I] calculate delivery compliance in an engineering/ Software(IT) project?"

From a Project Management viewpoint, I had some difficulty in understanding the term "delivery compliance", so I've taken the liberty of thinking of this as scope verification -- how we determine that what was delivered meets the project requirements. There is then a second part to this inquiry -- how is it measured?

This can be a complex question or issue. We all know from the 70s and the work of Gerald Weinberg (The Psychology of Computer Programming) that testing software does not prove the absence of bugs. Where scope statements require high reliability then, the measurement becomes uncertain. Complex and large projects also create challenges for verification and measurement.

I believe I can put some simple structure around this complex question. During the early years of my software and software project management career, I worked with engineering/software development teams at GE Information Services that were challenged to:

- develop systems that worked with high reliability - 99.999%
- meet the demand that memory and cpu utilization vary no more than 1% from release to release (since clients were billed on this basis)
- provide high client satisfaction -- an "A" average was required on client report cards

This was "RASM" (Reliability, Availability, Scalability, Maintainability) before the acronym was coined! Here's how this was achieved:

Verification
Verification required a robust process to maintain the chain of accountability for the requirements. This process included:

- written SMART requirements (specific, measurable, achievable, realistic, time bound)
- diligent requirements reviews (indirectly more of Weinberg's work)
- diligent design reviews
- coding standards
- peer reviews of all code
- management sign off on all changes
- developer created unit test cases which contributed to future regression tests
- test cases which went through a similar process to the software development process.

Testing was also a critical part of the process. In my opinion, testing these days has become too reliant on use cases. Use cases make fine exemplars or clarifications of requirements, but are poor test cases -- users rarely do what's expected of them 100% of the time. Testing needs to be a good mix of black box, white box, and glass box testing to ensure you are capturing as many issues as possible. The requirements need to become your checklist. Further, in addition to looking at the checklist an item at a time, you need to step back and make sure they meet the big picture (here's a nice role for use cases, as well as additional testing to verify the interaction of component).

Critical components of testing in the environment included:

- a strong, multi- and cross-platform configuration management system to manage builds and their versions
- ability to easily integrate developer unit tests into other test beds through defined test frameworks
- automated testing to drive high test coverage in shorter periods of time
- a measurable regression test bed (ensure no unexpected changes in CPU/memory or functionality)
- load testing (how many simultaneous users could use the system with no database or other system deadlocks)
- stress testing (what happened when the expected limits of any system aspect were exceeded).

Metrics

Now that we've had a look at the intended delivery outcomes and the process to achieve them, let's take a look at the key metrics which were in place and how they were used to assure delivery:

1) Regular reporting of number of open tickets and issues by priority, software system, client, and age
2) Regular reporting on system and application availability. Goal was 99.999% (scheduled down time was considered meeting goal, unscheduled was not)
3) Regular client survey results, requiring a specific average (usually something like A- or better). Clients were proactively called on a regular basis by client services to gather the information.

To further reinforce the compliance of delivery, these three metrics were generally tied to financial compensation and overall employee performance reviews. Everyone in the organization carried a responsibility to help meet the metrics. (Note: See my earlier article Time, Deliverables, or Outcomes? for more on the perspective of the triangular relationship of quality, availability, and high client satisfaction.)

It is also interesting to note that we found client satisfaction directly linked to number of open tickets. It wasn't enough to just close the high priority issues. Initially there was an attempt to ignore lower level issues. As the number of these issues grew, client satisfaction dropped. Once out of compliance, we initiated a period in which only low level tickets were addressed. Client satisfaction increased within two quarters and we then adopted a more balanced approach than just closing high priority, high impact tickets.

After these key metrics, there were other regular reports available including:

- measurement of time on projects
- measurement of time on issues and other non-project work
- measurements of system and application usage on display monitors throughout all facilities, including metrics such as up time, availability time, number of simultaneous users, transaction rates, etc.

Conclusion

Delivery compliance is a complex topic. Its not about a simple metric or measurement. I've attempted to put together a picture of a framework for a development process which drives an organization toward effective, high quality delivery. The process was supported by three key, simple metrics which kept everyone aligned to the highest priorities and values of the company. Additional metrics helped build team work and rally employees around milestones (e.g. the day we reached 10,000 simultaneous users on our private telecommunications network and mainframe processors).

The Power of Checklists and Sign Offs

I usually have to visit a copy shop at least 2-3 times a year to fulfill various needs for PMI meetings and events. This past weekend I had a critical business need to have a small number of copies made and spiral bound for a customer meeting. With FedEx Kinko's being the closest, I dashed off last night to have the copies done. Since I don't frequently visit, this was like a "first visit" all over again.

Fully expecting to pick up my copies on Sunday, I explained what I needed. In under 5 minutes, Kevin, a customer consultant had a copy made and bound for me to review and approve. I was quite surprised when he informed me I could come back in a half hour and pick them up. Being on the way out to dinner anyway, I returned in under an hour to pick up the finished product.

Packed with my copies, I found two similar checklists which read something like this (I've shortened, but maintained the key messages):

- I took your order, repeated your instructions, and offered you enhancements.
- My order was repeated back to me and I am confident my instructions were clearly understood.
- I have reviewed the sample and approve the order.
- I produced the order according to the instructions.
- I quality checked all previous work and finished your order according to your instructions.
- I quality checked the final order using quality standards and the instructions. To the best of my knowledge, this order meets your expectations.

Both lists were cross checked and initialed by Kevin and another employee. My order was very small compared to many (8 spiral bound copies of a 20-25 page document). But the process was followed and the checklist completed. The work was high quality, and my simple document looked impressive. Based on the timestamps, this order was completed by Kevin and another associate in 5-10 minutes.

The important message here is what went through my mind as I left:

- The shop employees are efficient and accurate. They did a great job at setting my expectations.
- They followed the process for even a small order.
- My order looked great and care was taken to make sure my needs and expectations were met.
- This obviously wasn't a bureaucratic process and the checklist captured the key points -- the order was quickly produced and waiting for me on my return trip after dinner.
- I can trust they will complete larger future orders with high quality.

There is an important message here for projects large and small. Having checklists make sure the best practice and process is followed. By initialing each box, I'm assured those completing the work felt good enough to put their name on it. They didn't make me sign or initial the form, but I was more than willing and gave my verbal consent while in the store. But most important, I walked away feeling the project team was able to take on larger challenges while still meeting my needs and expectations.

Do your project sponsors have the same feeling when your projects are completed? Will the next project charter name you as the Project Manager?

PMP® Challenge Question - 9/1

Our 8/15 challenge asked -- in an organization, who is responsible for quality? The answer is the whole organization! The Project Manager has the responsibility to ensure there is a plan for quality of the project, but each team member is also responsible for checking their own work. Senior management has the responsibility for quality in the organization as a whole. Work needs to meet the requirements and testing should be done prior to completion of each deliverable.

For our next challenge, can you name at least 6 communication blockers? See you on 9/15 with the answer.

PMP® Challenge Question - 8/15

Before we look at a new question, let's take a look at the answer to the 8/1 challenge. In it, I stated that many PMI components have some controls or policies in place to prevent preferential treatment to active volunteers at the expense of the general membership. For larger components with more procurement needs, the challenges include providing fair treatment for all stakeholders, including minorities, veterans, volunteers, and members. What is one significant way to conduct the procurement to provide an unbiased opportunity for everyone?

There are several variations we can build into a procurement process to make it more fair, as well as ensure we pick the best vendor. First, this includes developing a good set of criteria, possibly weighted, which will be used to evaluate the responses. There still may be bias if the evaluators know who applied, so a blind evaluation might be the next step. This means removing references to the bidder from the materials so a completely unbiased evaluation can be completed.

Now here's a new question: In an organization, who is responsible for quality?

Reviews are Required

I've noticed a growing trend in a counter-productive behavior -- skipping a review and rushing into execution. The usual excuse is "it slows things down". The price you pay, however, is you may build the wrong thing and really slow things down. Many years ago I worked for a computer manufacturer. They aggressively decided to build a new system in parallel with developing the operating system -- this wasn't the norm in those days. The lack of an appropriate review let a major showstopping issue through -- while the programmers were numbering the bits in one direction, the hardware designers were going in the other. This wasn't uncovered for months, and many more months were wasted reviewing the code and correcting it after the damage was done. In the end, another manufacturer cornered the market and the system never went into manufacturing.

The Siebel Systems Expert Services team used to have at least a dozen different types of reviews available for customers to protect their project timelines and the value of their software purchase. I can recommend three basic reviews to consider for any project:

• Status: Where are we with respect to schedule, cost, performance, and scope? Perform this review periodically (usually weekly) or at least at regular intervals.
• Design: Does it meet spec? User friendly? Can we actually build it? Perform this review at least after the functional requirements are written.
• Process: What are we doing well? What needs to be improved? Perform this review for longer projects or in between short projects.

In fact, each major step should have some form of review if for no other reason than to verify the scope and the quality of the deliverable. If you are not sure how to conduct reviews, I can highly recommend the work of Dan Freeman and Gerry Weinberg. Their original work, published in 1977 was the Ethno Technical Review Handbook (EthnoTech Inc.). In 1990 they updated this work with the Handbook of Walkthroughs: Inspections & Technical Reviews (Dorset House).

So next time you have an important project, as you plan, plan on having some reviews at the right time to ensure your project is successful.

Flying 300 Miles to Correct an Error!

OK ... thought it might be time to repeat this story. I frequently use it in my PM classes as an object lesson in detail and quality. It has a place in my IT classes as well, since it points out an age old issue. I hope you at least get a little chuckle from this one!

This goes back to the very late 70s and early 80s. I was working on a project with an advanced concept for the time. The product we were developing needed to run equally well on any major IBM platform. At the time we selected the 360/370 series running OS-VS, OS-MVS, DOS-VS, DOS-VSE, VM, and the older IBM System 3. Most of the code was in COBOL for ease of portability. Some of the code had to be in assembly language since we were dealing with communications channels specific to the systems. We obviously tried to limit the amount of this code to keep things simpler.

Things went very well during development and initial testing, but the full roll out was rocky to say the least. Finding available equipment to test all these combinations was a major problem. The OS versions were easy -- the company owned the right platforms. For DOS and DOS-VSE, we finally contacted a number of service bureaus and found one that fit our budget. Our local IBM office was kind enough to lease us time on an IBM System 3, so we thought we were home free.

For those of you that aren't familiar with it, the IBM System 3 was a strange beast. Its primary inputs consisted of punch cards and floppy disks. The punch cards weren't very standard, however. They were tiny cards with three rows of punches -- 96 columns arranged in rows of 32. This took a special key punch which made tiny round holes. I bought a special card box to hold the "master" copy we produced at the IBM facility. The local office only supported punch cards, so this became our primary medium.

This scheme worked fine, until we encountered a client in Detroit without a card reader. Our local IBM office connected me with an IBM Data Center in Indianapolis where I could convert our master card deck on to a floppy disk. They even sent one of their local support people with me to assist. We tried running the job several times, but always got a warning on a certain card. Looking at a listing of the COBOL code, we saw a blank enclosed in quotes (" "). Thinking this was nothing out of the ordinary and needing to get over to the client, we packed up the newly made floppy disk and headed out. This was to be a simple, overnight trip.

Next day at the client site, everything installed and compiled smoothly. But getting the software to communicate was another issue. We were unable to connect to the communication channel. Traces showed we sent one message which was immediately rejected. We puzzled over this for hours. Then I remembered the card that kept giving us problems when copying to the floppy. I called up the programmer and had him take a look at the statement. Uh oh! Big trouble! The quotes actually enclosed a non-printable ASCII character which was an essential part of the initial handshake between computers. Had I looked at the actual punch card instead of the listing, I would have seen an odd punch combination it took special pains to make.

So now we had a fix -- the programmer gave us an equivalent statement with the decimal value so it would be visible. But the client had no facility to edit the floppy disks. Next day we hopped on a plane and flew back to Indianapolis (about 300 miles). We fixed up the cards, copied them to floppy again, then went back to the customer. Success! Everything went without a hitch.

Now there are several lessons learned here, so be careful not to miss any! These mistakes cost an extra flight for two and turned an overnight trip into three or four days. First, set a programming standard prohibiting non-printable characters in quotes in the code. These are too easy to overlook. Next, whenever testing text files, always check the file for non-printable characters. I was hit by this one again a few months ago due to a programming error. Dumping the file in hex showed the offending character. Also, look at the actual source, not the copy. Had we looked at the punch card the first time, we might have caught this before leaving the Data Center. And finally, always be prepared for the unexpected! This was so long ago, I don't recall what I did for clothing the extra days -- I'm sure I had to buy an extra shirt or two while on the road.

Get Your "Ticket" to the Show

I was comparing notes with a colleague at a local PMI meeting last week and we agreed there seems to be an evolutionary step many companies go through as they emerge from chaos and move toward an environment supportive of Project Management. For lack of a better name, I'll call it a "ticket" system. They use products such as Bugzilla, TeamTrack, Remedy, SharePoint Lists, and so on, to "organize" their work. This might range from recording service requests, issues, mini projects, and other discrete work tasks they need to complete.

Now there is nothing fundamentally wrong with this type of tool or organization. The problem seems to lie within why it is adopted and the expectations around its performance. A colleague at a major San Diego telecommunications company related this story [I'm paraphrasing] to me a few years ago ...

Our IT Department installed a ticket system to handle their growing volume of work. Within 6 months, at least two new layers of management were installed to "manage" the system. Everyone was going crazy. The project team members were unable to get anything done, because every time they turned around, there was a manager on the phone asking about the status of ticket "x". They needed more managers since the number of unsatisfied tickets just kept growing exponentially.

Now the real issue here is usually a failure to address the "root cause". If these are problems with bugs, defects, or quality issues, sure a ticket system can help in organizing and prioritizing the work, but if the underlying quality issue isn't identified and fixed, the ticket system keeps feeding itself.

Or maybe the underlying issue is planning and resources. In that case, some hard decisions may have to be made about what will and will not be done, or what resources may need to be added to accomplish goals, but throwing in more tickets to develop new features is not going to solve the underlying issues.

There WILL be a breaking point. After a near internal melt down, the telecommunications company realized the situation was not improving and hired consultants to a) improve product requirements so quality and other issues were addressed and b) started to adopt basic project management practices to prioritize work and review resources and schedules. Over time they de-emphasized the significance of the ticket system in favor of creating well run projects.

When I took over my current position, I had 6 pages of Excel spreadsheet of work (around 300 tasks taking anywhere from 8 hours to a week to complete) needed for customers. During my first six months, we triaged and prioritized work, identified roadblocks and planned their removal, and steadily reduced the list. During the second six months, we analyzed resources, looked at some trade-offs, and continued to build tools. Today, there is little or no chance this list will become unmanageable again unless something goes horribly wrong. Why? Some of our clients are now trained and would rather take on the work themselves than wait in a larger request pool, we have trained more people to perform these tasks so we shouldn't easily run out of resources, and we've dramatically improved the tools and techniques we use to carry out the work. The root causes were successfully identified and addressed.

So if you are at the point that a "ticket" system is called for, step back and take a hard look at the root cause. The ticket system itself will often disappoint if you believe the organization it offers is the only thing that needs to change. Be sure you are planning to address the real root cause and not just institute a tool where tons of tickets will accumulate without resolution. The meltdowns may be entertaining for some, but they rarely lead to productive situations.