How to determine if an organization’s Quality Management System (QMS) is effective?

In last month’s article I discussed the properties of a ‘good’ QMS as well as its relationship to various Critical-To-Quality (CTQ) objectives organizations can use to perform quality assurance and quality control. For example, an organization is more likely to be successful when the objectives it has for its QMS are aligned with what I feel are the most important attributes and characteristics of a ‘good’ QMS:

In this article I discuss how to determine if an organization’s QMS is effective. Specifically, by examining the actual practices and results of the organization’s management review and risk-management activities, you will gain accurate insights into the effectiveness of that organization’s QMS.
What should you look at first when trying to determine the effectiveness of the QMS?

Many people feel that an examination of an organization’s non-conformance handling process and results will give a comprehensive indication of the effectiveness of its QMS. For example, insights into the effectiveness of the organization’s QMS might be seen in the:

• classification and trending of the root-causes of the organization’s non-conformances/escaped defect incidents;   
• amount of rework and scrap that occurred to yield the intended and conforming results.

On the other hand, non-conformance management process activities are practiced upon the occurrence of a non-conformance; i.e., this process is executed re-actively and as such will not yield a pro-active indication of the effectiveness of the organization’s QMS. 

The circumstance of examining non-conformances to determine the effectiveness of the organization’s QMS may be further corrupted by staff choosing to not accurately document that non-conformances exist because, in doing so, they suspect it will have a negative consequence towards their own career-development path or even the compensations they are afforded (if their own performance does not meet or exceed the organization’s goals for process or product conformity).

So, where else can you look to determine if the organization’s QMS is effective? Well, instead of looking at lagging processes/process indicators, let us first consider examining the leading ones within an organization’s QMS. To that very point, two of the leading processes/process indicators are those for the management review and risk management activities and results. This is because these two processes are directly tied to the ‘Plan-Do-Check-Act’ methodology encouraged by QMS standards like ISO 9000. For example, these two processes are purposed to proactively identify opportunities as well as serve as the continual focal point of the dissemination of and the decisioning needed to identify, prioritize, and communicate awareness to the necessary actions that will allow the organization to achieve its objectives.

Why are Management Review activities and results the first indicator of the effectivity of the QMS?

Well let us start with the fact that a management review process is purposed to assess the organization’s:

• ability to meet customer expectations. For example, management must decide which requirements as well as which statutory and regulatory obligations are in scope.
• actual performance. For example, management must determine what, if any, opportunities to improve exist based on the measured performance of the organization’s products/services, processes, people, and otherwise practices selected to meet its C-T-Q objectives.
• ability to grow or improve to satisfy the customers with which it intends to do business.

In other words, when an organization practices management review, it is by those very purposes trying to use its QMS effectively.

Next, one of the most crucial factors on whether an organization is going to be successful is having the genuine commitment and support of the persons in charge. This is another reason an examination of the management review process activities and results is going to be an exceptionally good indicator of the effectiveness of the organization’s QMS. For example, the management review results contain the decisions and plans that are intended to enable success for all the organization’s resources as well as empower them to make changes for improvement. Without those decisions and plans success is not possible; furthermore, a successful organization is one more likely to be an effective one.

Next, another reason the assessment of management review activities and results will be a good indicator of the effectiveness of the organization’s QMS is that management review is not just something done for some of the organization’s top-level objectives or its most significant products, processes, and people. Rather, it is applicable to every objective, product, process, and role in the organization. Thus, an effective QMS is one that is minimizing, if not eliminating, the uncertainty and unpredictability with any facet of the organization’s strategy, tactics, and operation.

Why are Risk Management activities and results the next best indicator of the effectivity of the QMS?

When you examine the organization’s management review results, and they tend to include decisions and actions that primarily are re-active in nature then that organization’s risk management activities also need to examine. 

Specifically, the whole point of a risk management process is to gain the identification of mitigations pro-actively as to prevent obstacles (in any manifestations of the organization’s effort to successfully met its client, supplier, employee, regulatory and statutory obligations) from happening in the first place. 

That is, the management of a risk (whether by avoidance, reduction, sharing/transferring or even acceptance/retention) is one of the best mechanisms for organizations to use to effectively:

• prioritize the use of their limited resources
• accomplish their objectives
• increase stakeholder confidence and satisfaction

Another reason an examination organization’s risk management process and results will provide insights into the effectiveness of its QMS is because QMS standards like ISO 9001 stress the importance of understanding the intended interaction of the organization’s key processes as well as the contribution expected of each role implicated by those processes. To that point, the organization’s risk management activities are integral to its ability to proactively and clearly understand issues as well as to determine and prioritize the recommendations for how risk opportunities can be managed. For example, within an effective QMS, the risk management activities will:

• identify the processes and objectives implicated by a risk
• the extent of actions the organization can plan/take to manage the risk
• re-assess the actions that have been taken to manage a risk as to determine if yet newer (e.g., residual) risk still exists and what if any newer plans to mitigate those newer risks now need to exist

QMS standards like ISO 9001 are also correct in pointing out that before an organization accepts a contract it should first consider the risks associated with it. In other words, hierarchically speaking QMS standards position risk management as a process very much key to the effectivity of the organization’s management review process and results.

Please see Annex-1 for characteristics of a ‘good’ risk management process.

Summary and Conclusion

In summary, an effective QMS is one in which the management review and risk management processes are:

• recognized as being integral to driving the organization’s success and growth. For example, the management review process must be the primary mechanism for assessing and deciding on what, if any, changes to courses of action are needed for the strategic, tactical, and operational activities going on within the organization. Many of the significant inputs to the organization’s management review process will come from its risk management process.
• two of the most well practiced and monitored activities in the organization. For example, beginning with how the human resources are hired and trained, to how the culture of quality is defined and practiced with the organization, emphasis must be based on having defined QMS objectives as well as decisions that are based on facts or a well formulated and vetted analysis.

To conclude:

• an effective QMS is one that pro-actively enables the organization to make ‘good’ decisions. Those decisions are more likely to be made when the organization is focused upon customer satisfaction, compliance with its objectives and an awareness to the ever-changing requirements of or recommendations for product/service specifications and implementations as well as the statutory or regulatory obligations and the best practices of the marketspace(s) implicated.

Please see Annex-2 for a description of the characteristics of a ‘good’ decision.

• although QMS standards like ISO 9001 identify a comprehensive list of the inputs and outputs of the management review process, they are a not so clear in their narrative on the frequency required or recommended for executing that very process. That is, after reading ISO 9001 some people are too quick to conclude that the management review process needs to be executed only once per year. Whereas in most business circumstances that will not be a road to success because the only thing constant is ‘change’ (in opportunities and when necessary, the objectives and the respective plans and/or implementations for achieving success).

Please see Annex-3 for my own guidance on the frequency of management reviews.

• an organization’s risk management activities must be dynamic, iterative, and responsive to change. This is because it is those very activities that facilitate continual improvement within an organization. 

Please see Annex-4 for some best practices for an organization’s risk management process.

And now, a sneak peak into my next article

In my next article I will discuss ‘e-QMS’ capabilities and user best-practices an organization may wish to have in its overall Enterprise Resource Planning (ERP) systems. For example, the e-QMS tooling must be designed and implemented to provide improved results in how people in the organization can collaborate and detect risk. The e-QMS tooling must also provide accurate and real-time indications to the organization’s perception of its customer’s satisfaction as well as to the actual progress staff are making on the work tasks that have been assigned to them.

Annex-1: The characteristics of a ‘good’ risk management process

The characteristics of a ‘good’ risk management process are:

• a risk-rating rubric that is used for all the organization’s processes and locations. 
• the risk management plan also needs to continuously assess the timeframe in which the risk implicated is expected to occur; i.e., risk management is ongoing. It is not just attempted when an objective is first set or when a project first starts.
• the relationship of a risk to the key processes implicated must be known/identified. This will allow risk management practitioners to measure the effectiveness of the actions they take to manage the risk.

Remember, risk-rating calculations (e.g., consequence*likelihood, impact*probability) are one-thing but so is sustaining an understanding of the timeframe in which the opportunity will occur. The following table reinforces the idea that risk management activities must describe the impact, probability, and timeframe of each entry in a risk registry as to ensure the organization is demonstrating it can and will manage the risk. Further to this point, a ‘good’ risk management process will use an evaluation method that helps to ensure that the Risk Rated Number (RPN) of a risk is not too superficial OR too complicated in use of the practitioner’s time to quantify let alone manage this risk.

Annex-2: The characteristics of a ‘good’ decision

The characteristics of a ‘good’ decision are that it is:

• readily traceable to the facts as well as the organization’s mission and vision statements
• supportive of the customer as well as the organization’s goal to increase customer delight
• clear and concise as to be readily understood and implemented by the resources implicated
• timely
• obtained by consensus

Since the decisions made in a management review are intended to create/alter any of its strategic, tactical, or operational objectives or their respective courses of action to be taken, then it should also be recognized that each and everyone of the management review practitioners must demonstrate commitment, courage, and consistency. That is, an effective management review process is one in which the implicated decision makers first understand the characteristics of a ‘good’ decision as well as continuously commit themselves to making such ‘good’ decisions. 

The characteristics of a ‘good’ management review meeting are that:

• weaknesses identified with any of the organization’s products and processes are seen/discussed as candidate opportunities to increase the amount of customer satisfaction or the ability of the organization to exceed its defined objectives/goals. 
• the practitioners of management review (and the analysis data/information gathered in its supporting risk management activities) have the necessary skills/experience and training to be able to focus themselves on the issues for which the organization has reasonable control. Or more practically speaking, the registries for risks and action items must not contain entries that are ill-defined, ranked or prioritized.
• during management review meetings questions are both asked and answered on the organization’s opportunities and the response plans proposed for them. Open, candid, and transparent discussions will help stimulate the innovations and/or improvements to where and how the organization understands itself to have the reasonable means and controls to manage its risks.

Annex-3:  The recommended frequency for management review

Here is some guidance that I feel QMS standards like ISO 9100 should have with respect to the frequency of management review. That is, because organizations have multiple levels of management, and each is intended to manage diverse types of objectives to success. it is more than likely that an “annual” approach to management review will fail to make the organization successful.

Annex-4: Some best practices for a risk management process:

In short, try to implement risk management mechanisms that: 

• define and manage risks based on the objectives held by each level of an organization’s management; i.e., think about management’s current objectives for strategy, tactics, and operations before you assign a risk its values for its likelihood of occurrence and consequence.
• describe and categorize risks so they can be quickly recognized for their applicability and re-use in future circumstances encountered within the management review proceedings.
• ensure the mitigations proposed for a risk are ones the organization can attempt and complete.

Next, remember that risk management activities (and tooling) must occur at every phase of the product development and release methodology used by an organization. That is because an effective QMS prevents the risks associated with the bid, development, release, and maintenance activities that the organization commits to from impeding/diminishing its intended success (e.g., ROI/budget, schedule, product reliability).

Depending on the complexity of the product and the regulations for it, the risk management process will have multiple preventative measures in place including but not limited to:

• Safety Hazard Analysis (e.g., for medical devices, this is a function of the device’s class; for an aerospace avionic product, this is a function of its Design Assurance Level (DAL)).
• a comprehensive and appropriate design methodology (which can include modeling and simulation, DFMEA, Halt/Hass testing).
• a comprehensive new product introduction and manufacturing methodology (which can include PFMEA, Test Equipment Maintenance and Control Programs, Production Part Approval Process).

Next, the management review process must use the results of the risk management process. For example, assessing business goals to actual performance, addressing impact of adverse/unplanned events as well as monitoring perception of customer satisfaction (via awareness to customer complaints, field failures/returns, and product quality alerts and, if necessary, product recall incidents) must remain aligned to and commensurate with the risk/opportunity management plans.

Finally, here are some exemplary questions to ask yourself to identify mitigations for two common risk-filled circumstances often encountered by organizations:

Risk Circumstance #1 … Growing the business by assessing a contract with a brand-new client

The risk assessment of this kind of circumstance needs to determine the suitability of the contract versus the likelihood of its ability to bring success. So, when looking to identify the possible risks in engaging a brand-new client, ensure you think about:

• What is the opportunity and the client’s terms and conditions?
• Does the organization have the funding/money to meet the client’s objectives (from start to finish)?
• Has this organization been successful with this type of work before? If not, does it have mentors to help lead it to success?

Risk Circumstance #2 … Growing/sustaining the workforce by hiring a new employee

This risk assessment of this kind of circumstance needs to validate whether the candidate employee is likely to be successful within the organization. So, when looking to identify the possible risks in hiring a candidate employee, ensure you think about:

• How is the candidate qualified for the job?
• Will the candidate fit into this organization’s culture/environment?
• Does the candidate really want the job?