8.5 Performing Measurement System Analysis (MSA) for CTQ(s)

You will see some variation whenever you measure a process’s outputs because any process always produces some variation. Besides, methods for taking measurements cannot be perfect. In the Measure phase, you and your team need to look for potential errors in the measurement system itself. These errors fall under two categories:

1. Accuracy: the difference between recorder measurements and actual values.

2. Precision: variation in measurements when a device takes repeated measurements.

Precision consists of reproducibility and repeatability. Reproducibility is variation caused by the measurement systems. You observe it when different employees measure different parts using the same device. Repeatability is variation caused by the measuring device when the same employee measures the same part with the same device repeatedly. You need to collect data on the CTQ of interest. If there is significant variation due to the way you take measurements (measurement error), you may end up making wrong decisions.

For a simple example, imagine you are producing the infamous peanut bags for onboard airliner service. You are using 12 grams bags. You decide to test the weight of the bags using a scale. You could encounter three situations. First, you could use a scale that measures the bags precisely—there is little variation in the measurements. However, the scale is not accurate. The measurements for a sample of three are 13.2 g, 13.33 g, and 13.13 g.

Second, you could use a scale that is accurate, measuring three samples at 12.02 g, 11.74 g, and 12.43 g, but not precise. In this case, the measurements have a larger variance, but the average of the measurements is very close to the target value of 12 g. Finally, your scale could be such that your measurements could be all over the place, where your scale would not be accurate or precise. You need to ascertain whether your measurements and instruments are good. Figure 8.2 illustrates these points.

You use measurement system analysis (MSA) studies to examine the capability of your measurement system for a critical CTQ to accurately deliver data to the improvement team. The fundamental question here is whether the variation of your measurement system is too large to study the current level of process variation. The MSA’s purpose is to identify errors of accuracy within data collection tools. Anything that can compromise repeatable and reproducible data may produce unreliable outcomes. And as the old adage goes “garbage in, garbage out.” Bad data drives project failure.

The primary contributors to measurement system error affect the spread of your data distribution. Besides, they describe the measuring system’s precision—repeatability and reproducibility. As these two characteristics are essential types of errors, there is a specific study to examine them, the gage R&R. This study is a statistical tool that helps you measure the amount of variation in the measurement system caused by the measurement device and people.

A Gage run chart can assist you in determining whether your measurement system is consistent in its measurements of a particular item and whether it agrees on the measurements of a particular item. An important point here is that the data used in the Gage run chart needs to represent the full range of conditions the system experiences. For instance, the most senior person performing the task alongside the most junior one should repeatedly measure each item selected in the study. Besides, you need to collect the data in random order to avoid adding biases.

Setting Up a Variable Gage R&R Test

You can use two to three appraisers and at least five to ten outputs you will measure to set up an R&R test. You have each appraiser measure each sample two or three times, randomizing the order in which they appear to avoid appraisers remembering the measurements results. You then record all data on a template such as the one shown in Table 8.2. The list of basic steps to perform a gage R&R consists of the following items:

1. Calibrate the measuring device.

2. Have each appraiser measure all the samples once in random order.

3. Repeat step 2 for the required number of trials.

4. Use a spreadsheet or software to determine the statistics of the R&R.

5. Analyze the results and determine follow-up actions.

You can then perform a statistical analysis using Excel or Minitab to obtain the following:

1. % Study variation

2. % Tolerance

3. % Contribution

4. Number of distinct categories

You want to see each of the four elements within the “safe” ranges. Typically, each element comes with a scale for safe, cautions, and failure zones. If you observe one of the elements within the caution zone, you may conclude that the measurement system is sufficient. If increasing the accuracy of the measurement system is costly, you may accept having most or even all of the scores in the caution zones. However, if you observe any score in the failure zone, you should fix or replace the measurement system. Table 8.3 shows typical criteria to assess each element of the R&R.

The number of distinct categories represents the number of groups your measurement system can distinguish from the data itself. For instance, when the number of categories is 3, the data can be divided into three groups: high, medium, and low. As you can see from the criteria table, the higher this number, the better chance the system has discerning one part from the other. Figure 8.3 shows a Minitab printout. Observe the number of categories equal to 24.

Collecting and Analyzing Baseline Metric

A critical deliverable coming off the Measure phase is the baseline metric. You will execute your data collection plan by obtaining baseline data to assess the stability and capability of each CTQ. Keep in mind that, in some circumstances, you may already have baseline data. Otherwise, you will need to collect it. Your goal is to examine how the process is performing now and what measurement you will use to compare current to post-improvement performance.

It is good practice to present the baseline metric graphically. You can display discrete data on Pareto charts, covered in Topic Six, and continuous data via run charts. A run chart is a type of line chart where you plot data over time. You can use run charts to graphically display measurement data. Most lean six sigma run charts also feature a line representing the median of all data points for visual reference. You can use run charts to monitor your process, but they are not as effective in the monitoring function as control charts you will see in a separate topic.

Creating a Run Chart in Excel

Software such as Minitab creates all elements of a run chart automatically, but you can always use essential Excel functions for the same job. Let's look at using this standard software to create a run chart by assuming you are working on an improvement project for a line of products sold at Amazon. Your leadership team is concerned with the rate of returns. After taking the first steps into the process, you defined your measure of interest as the rate of returns per sales. You decided to show them a visual representation of the baseline metric.

1. Create a data Table. The procedure to create the run chart is pretty simple. First, you create a data table containing time labels and the attribute measurements in separate columns. For instance, for the returns project, your data table would look like the one in Figure 8.4. In this example, the data labels are in months and appear in column A. The attribute measurements appear in column D, and you obtain them by dividing the number of returns by the number of sales: (D = B ÷ C) × 100).

2. Calculate the median of the metric you will chart. Use Excel to calculate the median of the metric you will chart in a column using the formula =Median (8.24, 12.29, ...). In this case, the metric is Returns per Sales in percentage, and the median is 12.17. After inserting the formula in the first cell, copy it down the column (in this case you would copy down the following: =MEDIAN($D$2:$D$13).

3. Plot the chart. Highlight in Excel the data labels (in this case column A), the metric of interest (in this case column D), and the median (in this case column E). In the ribbon Charts, select Line. You can then adjust, copy, and paste your chart in another document.

You can use the final chart, shown in Figure 8.5, to present graphical representation of the baseline process performance in documents or during the Measure tollgate review. While there is specialized statistical software that can produce a run chart for you, sometimes you may not have access to this software. Excel, however, is typically available at any organization.

At this point, you want to see whether the baseline data for the CTQ exhibit any patterns over time. If the CTQ is not stable over time, you and your team want to examine where the causes of abnormal variation taking place are. You will see in later topics that you can use control charts to identify where and when special causes of variations occur. However, you do not use them to identify the causes themselves. To examine the causes, you can use tools covered in previous topics such as brainstorming and cause and effect diagrams.