Some Common Problems Noted In Computer-Related Audits

The Legislative Division of Post Audit

Problems With Project Planning

• Often agencies started a computer development project without adequately assessing their needs. This led to time-consuming and costly modifications to the projects during their development.
• When planning projects, agencies did not sufficiently involve the end users of the systems in the decision making process.
• When agencies did a needs assessment, they often relied on software vendors who had a product to sell, rather than using an independent consultant.
• Significant costs--such as the time of State staff involved in the project--often were overlooked when establishing the budget and presenting it to the Legislature.

 Contracting Problems

• When private contractors have been used, poorly written contracts have allowed vendors to be paid for systems that didn't work, or for work that was not satisfactory.

 Staffing Problems

• Agency staff assigned to systems development projects didn't always have the necessary expertise.
• Agency staff were supposed to be provided to work on systems development, but often were not provided in sufficient numbers or for enough time for the project to be successfully completed.
• Agency staff, including project managers, often were expected to continue with their other duties and work on complex computer projects "on the side."

Management and Oversight Problems

• Project managers sometimes were given responsibility for a project, without being given authority to make crucial decisions or to commit the necessary resources to the project.
• Top management did not always lend their full endorsement to the project which led to disagreements and in-fighting among lower-level managers about what the system was supposed to do, what was to be done, and by whom.
• On many projects there was no quality-control review to assess the quality of the work vendors had provided before the vendor was paid.

Some Suggestions for Good Practices to Follow When Developing or Acquiring a New Computer System

The following list of good practices comes from recent audits of State computer systems, suggestions from State officials, and various references dealing with the planning and implementation of new computer systems. These good practices can and should be followed when developing a system in-house or when contracting with a private vendor, but sometimes have been lacking in computer systems developed by State agencies. References that may be helpful when developing a computer system are listed at the end of this document.

 
PLANNING THE PROJECT

First, do a needs assessment. Clearly defining what you need will reduce the number and magnitude of project changes and revisions after the project is under way. A needs assessment involves examining current operations and deciding exactly what you want the new computer system to do. A good needs assessment includes an evaluation of the existing agency manual and computer business processes; new automation may be just one way to improve those processes.

 
If possible, have the needs assessment done by someone who is not trying to sell you a specific product. It is best to use an independent consultant, or in-house staff if they have the expertise, rather than a computer vendor. This can help ensure that the needs assessment is objective. Also, while computer vendors may give reasonable assessments, their knowledge may be limited to their own products and may not take into account other vendors' products that are more suited to your agency's needs.

Extensively involve those who will be using the computer system in all stages of planning and development. Administrators and managers often don't have a good sense of the day-to-day needs of the people who actually use the system. In the past, important pieces have been left out of systems because users were not involved.

Consider whether an existing software package will fit your needs. Computer vendors often have software packages that they have developed for agencies in other states. These packages often can be modified to fit your needs for less than it might cost to develop a system from scratch. Again, always involve the users in this assessment. Only opt for custom software, with its higher development and maintenance costs, when there is a clear business case for doing so.

Ensure that the system's infrastructure is adequate. Network design and capacity, data security, user help, data backup and storage, and disaster recovery all are important considerations. Also, system designers should develop standards for acceptable response times and other service levels. Those standards should be approved by the system's users.

SETTING THE PROJECT BUDGET

Establish a budget for each phase of the project and a budget for the total project. Always prepare a budget that considers staff time to be provided as well as dollars to be spent. A working budget provides a comparison of staff time spent and actual costs with planned time and expenditures. This tool provides accountability for staying within the budget, and a systematic basis for deciding whether to modify activities to control costs.

Be sure that all costs are included in the budget. Costs such as salaries and benefits of State personnel assigned to the project and costs of training agency personnel frequently are forgotten.

STAFFING THE PROJECT

Assign a full-time manager to the project with clearly defined responsibilities. Computer development projects are very complex and difficult to manage. The manager should not be expected to work on other agency projects while managing the development of a computer software project.

Give the project manager sufficient authority over the people and other resources provided for the project. A project manager cannot successfully complete a project if he or she has to compete with other managers for limited staff resources. The manager needs to know what resources have been committed to the project, and that they can be used as needed.

Ensure the in-house staff assigned to the project have the expertise needed to effectively carry out the role assigned to them. In the past, assigning inexperienced staff to projects has been a major reason why system implementation was delayed. In addition, on joint development projects, if an agency does not supply people with the necessary expertise to do certain tasks, contractors can claim that this affected their ability to complete the project on time.

Define the work and responsibilities of all project staff. This helps to make sure that tasks are not forgotten, and that everyone who works on the project knows what his or her responsibilities are.

Make sure project staff are able to devote the required time without being expected to fulfill their other full-time duties. A frequent mistake is to expect staff to work on a computer development project and fulfill their regular duties at the same time. This generally causes poor results in both areas.

MAKING SURE THE AGENCY AND THE CONTRACTOR KNOW
WHAT WILL BE PROVIDED AND WHEN

Write detailed project specifications in your request for proposals and the final contract. In the past, some agency contracts have not contained sufficient detail to specify exactly what the contractor was to provide. Because the contract language was so vague, the contractors were paid even though they had not provided a workable product.

Evaluate proposals against the agency's needs assessment. A common mistake is to compare proposed software systems against each other, instead of trying to get the best match with the agency's needs.

Contract for completed products rather than "time and materials." This helps control project costs and ensure that you receive the product you wanted. There have been cases where projects cost millions more than expected and the agency still did not receive a computer system that worked.

When a contractor is used to develop software, make sure agency staff are familiar enough with the system to be able to maintain it. This may eliminate the need to contract with the vendor to make minor modifications or adjustments to the system.

Create a quality assurance review process. The review process ensures the contractor delivers a product that meets the contract criteria before releasing payment. This step could save time and money that would be required to fix a non-working system later.

Create project milestones and timeframes to be met by the contractor before releasing payment. This helps ensure that the project stays on schedule.

Specify realistic penalties for contractor non-performance or delays. This also will help ensure that delays are kept to a minimum.

Create a joint management team of outside consultants/vendor officials and State employees to make important policy decisions. This combination of viewpoints and knowledge helps ensure that the best possible decision is made.

OVERSEEING THE PROJECT

 Involve top agency management. Without top management support, the project may not have adequate staffing, or it may become overwhelmed by disagreements between departments.

Require the project manager to make regular progress reports. These reports should go to a management team or to the top agency managers. It is important that frequent reviews of the project progress be made so problems can be identified quickly, or avoided.

Address problems in software development or system configuration before moving on to the next step. Making sure each stage of development is done completely and correctly prevents costly time delays spent in diagnosing problems and rewriting programs at the end, when the system is much more complicated to unravel.

Top management should hire or assign a Project Architect to monitor the overall design of new systems. The Architect ensures that the system is designed to specifications and that it works with the existing agency structures. The Architect is a senior technician who prepares a plan for the entire scope of the project, including operations, networking, security, user help, and so forth. The Architect also monitors compliance with the plan.

Additional References

Information Technology: A Model to Help Managers Decrease Acquisition Risks; United States General Accounting Office, August 1980.

 Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology; United States General Accounting Office, May 1994.

 Information Resources: Summary of Federal Agencies' Information Resources Management Problems, United States General Accounting Office, February 1992.