In multiple regression model What are the characteristics of a good predictor variable

11.3.5.3 Multiple regression analysis of discussion evaluation

Multiple regression analysis was conducted to examine the effects of three factors (decision-making strategy, group to which participants belonged to, and type of agenda) on individuals’ evaluation of the discussion process, evaluation of the discussion results, and overall satisfaction with the discussion. Specifically, multiple regression analysis was conducted with the evaluation of the discussion process, the evaluation of the discussion results, and the overall satisfaction of the discussion as the dependent variables, and the decision-making strategy, the group to which the participants belonged to, and the type of agenda as the independent variables. The models were evaluated by AIC. The models examined were a model predicting by intercept only (model number 1), a model predicting by intercept and decision strategy (model number 2), a model predicting by intercept and participant's group (model number 3), a model explaining by intercept and agenda type (model number 4), a model predicting by intercept, decision strategy, and participant's group (model number 5), a model predicting by intercept, decision strategy, and agenda type (model number 6), a model predicting by intercept, group participants belong to, and agenda type (model number 7), a model predicting by intercept, decision strategy, group participants belong to, and agenda type (model number 8), an interaction between the intercept and the decision strategy, the group to which the participant belongs, the decision strategy and the group to which the participant belongs (model number 9), an interaction between the intercept and the decision strategy, the type of agenda, the decision strategy and the type of agenda (model number 10), and an interaction between the intercept and the group to which the participant belongs, the type of agenda, the group to which the participant belongs and the type of agenda (model number 11). We compared the partial regression coefficients of the decision strategy in the model with the lowest AIC among the six models that included the decision strategy.

Multiple regression analysis was conducted to examine the influence of the three factors of decision-making strategy, the group to which the participants belonged to, and the type of agenda on the evaluation of the discussion process. As a result of comparing and ranking the AIC of each model, the model with the lowest AIC was the model that predicted the evaluation of the discussion process by the interaction of the decision strategy, the group to which the participants belonged to, and the decision strategy and the group to which the participants belonged to (model number 9), with an AIC of 2767.89. In other words, among the 11 models examined, the model that predicts the evaluation of the discussion process by the interaction of the decision strategy, the group to which the participant belongs to, and the group to which the participant belongs to with the decision strategy can be judged to be the model with the highest predictive ability.

The partial regression coefficients for the decision strategies in this model are shown in Table 5.6. The partial regression coefficients for DIS as the reference were −0.07 for WAD and −0.02 for LEX. There was no significant difference between WAD and DIS for either WAD or LEX [WAD: t(1140)=−0.63, n.s.; LEX: t(1140)=−0.18, n.s.]. There was no significant difference between WAD and DIS [WAD: t(1140)=−0.63, n.s.; LEX: t(1140)=−0.18, n.s.]. This suggests that when comparing WAD and DIS, and LEX and DIS, the ratings of the discussion process did not change, respectively.

Multiple regression analysis was conducted to examine the influence of the three factors of decision-making strategy, the group to which the participants belonged to, and the type of agenda on the evaluation of the outcome of the discussion. The lowest AIC model predicted the evaluation of the discussion process by the interaction of the decision-making strategy and the group to which the participants belonged to (model number 9), with an AIC of 3439.48. In other words, among the 11 models examined, the model that predicts the evaluation of the outcome of the discussion by the interaction of the decision strategy, the group to which the participant belongs to, and the group to which the participant belongs to with the decision strategy can be judged to be the model with the highest predictive ability.

The partial regression coefficients for the decision strategies in this model are shown in Table 5.6. The partial regression coefficients for DIS as the reference were −0.19 for WAD and −0.06 for LEX. There was no significant difference between WAD and DIS for either WAD or LEX [WAD: t(1140)=−1.21, n.s.; LEX: t(1140)=−0.41, n.s.]. There was no significant difference between WAD and DIS [WAD: t(1140)=−1.21, n.s.; LEX: t(1140)=−0.41, n.s.]. This suggests that when comparing WAD and DIS, and LEX and DIS, the evaluation of the results of the discussion did not change, respectively.

Results of Multiple Regression Analysis (MRA)

Recall that MRA is a statistical procedure that assesses the relationship between a dependent variable and several predictor variables. The estimates generated by MRA are called coefficients. Using MRA, we can calculate the amount of variance in the dependent variable that is accounted for (= explained) by the variation in each of the independent variables. This calculation shows the relative importance of each independent variable to the relationship.

It is beyond the scope of this book to provide a detailed treatment of MRA as a statistical technique. For a basic interpretation of MRA results in economics, consult Greenlaw (2009). For advanced information on MRA and other statistical techniques, you may wish to consult Tabachnick and Fidell’s Using Multivariate Statistics.

In an MRA study, the following information generated by regression software is usually reported.

The size and sign of regression coefficients. The size of regression coefficients shows how much each predictor variable contributes on its own to the variance in the dependent variable after the effects of all the other predictor variables in the model have been statistically removed. In their standardized form (as β), regression coefficients are a measure of the importance of each variable, allowing researchers to compare the relative importance of the predictors. In economics and public policy, the sign of regression coefficients is also important and it is discussed in comparison with the expected (or hypothesized) sign predicted from theory: Do the explanatory variables have the expected sign?

Statistical significance for each estimated coefficient, which is determined by comparing the p-value (or significance probability) associated with a coefficient with the chosen level of significance. If the p-value is smaller, the coefficient is interpreted as being statistically significant; if it is greater, the coefficient is interpreted as being nonsignificant, or as not being significant. There are many variations in the reporting and interpretation of null hypothesis significance testing in public policy and economics. For example, in economics, three significance levels are commonly used: 1%, 5%, and 10% and results are often described as being “statistically significant at the 1% (or 5%, or 10%) significance level.” The 10% significance level is uncommon in other disciplines, for example, in sociology or education, where results with p-values that are greater than .05 (5%) are interpreted as being nonsignificant.

Alternatively, when reporting statistical significance, researchers may simply indicate whether the generated p-values are smaller than the level of significance. In this case, authors indicate statistically significant values with asterisks—a single asterisk (⁎) for p < .01, a double asterisk (⁎⁎) for p < .05, and a triple asterisk (⁎⁎⁎) for p < .1—and use a note under the table to show what the asterisks refer to.

In some research areas, authors may provide the exact p-values (e.g., p = .58). Providing the exact p-values is especially common in psychological and educational research, but it is fairly uncommon in economics. In some areas, confidence intervals are commonly used to indicate significance levels.

Because of the great variability among disciplines in reporting statistical significance, it is important to find out what is common in the particular area you are working in and report statistical results using the conventions of your field.

“Goodness-of-fit” statistics. These statistics show how well the model you are testing explains the data: How much variance in the dependent variable is explained by the combination of the predictors? The F-statistic is used to determine if all the coefficients in the model are statistically significant, whereas R2 (or adjusted R2) is used to determine the overall amount of variance in the dependent variable that is explained by all the predictor variables in combination.

Greenlaw (2009, p. 217) gives good advice for interpreting R2: “R2 for cross-section data is generally less than R2 for time-series data. Econometricians typically consider a time-series regression to be “good’ if it results in an R2 of 0.8 or higher. By contrast, a cross-section regression is considered “good” if it has an R2 of only half that: 0.4 or above.”

Regression results are always presented in table form. A typical regression table includes the following information: regression coefficients, standard errors (in parentheses), statistics indicating significance, and goodness-of-fit statistics. It is important to stress here that regression tables that are included in a paper are always constructed and never copied directly from regression output provided by the regression software. Later in this section, I give suggestions for formatting tables in a quantitative study.

11 Multiple Regression

To run the multiple regression analysis, follow these steps:

Start Excel and open the example model Risk Simulator | Example Models | 01 Advanced Forecast Models.

Go to the Regression worksheet.

Select the data area including the headers or cells B5:G55 and click on Risk Simulator | Forecasting | Regression Analysis. Select the Dependent Variable as the variable Y, leave everything else alone, and click OK. Review the generated report.

Exercise Question: Which of the independent variables are statistically insignificant, and how can you tell? That is, which statistic did you use?

Exercise Question: How good is the initial model’s fit?

Exercise Question: Delete the entire variable columns of data that are insignificant and rerun the regression (i.e., select the column headers in Excel’s grid, right-click, and delete). Compare the R-Square and Adjusted R-Square values for both regressions. What can you determine?

Exercise Question: Will R-square always increase when you have more independent variables, regardless of their being statistically significant? How about adjusted R-square? Which is a more conservative and appropriate goodness-of-fit measure?

Exercise Question: What can you do to increase the adjusted R-Square of this model? Hint: Consider nonlinearity and some other econometric modeling techniques.

Exercise Question: Run an Auto-Econometric model on this dataset and select the nonlinear and interacting option and see what happens. Does the generated model better fit the data?

Methodology

A quantitative paper states a hypothesis and tests it using statistical tools (e.g., multiple regression analysis) to produce generalizable results. If you are writing a quantitative proposal, explain what kind of data you will use and where the data will come from. Explain your empirical methodology, also what model you will use, what variables you will include, and how you will measure them.

A qualitative empirical paper explores a phenomenon or a process using multiple sources of information including in-depth interviews, documents, and observation. If you are writing a proposal for a qualitative study, explain why and/or how you selected your case(s), what data you plan to use, and how you will collect the data.

Which Approach Is Prevalent in Public Policy Programs?

Since the 1970s, public policy literature has been characterized by the pervasive use of quantitative methods of data collection and analysis including survey research, quasiexperimental research, multiple regression analysis, cost-benefit analysis, and economic modeling. Although public policy research became more diversified in the 1990s (Radin, 2000) and began to include qualitative studies, quantitative research remains prevalent in public policy and, especially, policy analysis, both in journal publications and in educational curricula. For example, in a review of educational curricula of 44 programs in public policy and policy analysis taught at leading public policy schools in the United States, Morçöl and Ivanova (2010) found that quantitative courses constituted an overwhelming majority of courses taught at both the master's (88%) and doctoral (79%) levels. They also found that the most frequently taught method of data collection was survey and the most frequently taught method of data analysis was multiple regression analysis. This emphasis on quantitative methods is also reflected in the predominantly quantitative types of papers that students in public policy programs are often required to write.

Quantitative vs. Qualitative Data Analysis

There are many different strategies and techniques for data analysis in public policy and economics; some are more common in a particular research area than others. The choice of the particular strategy for data analysis is dictated largely by the purpose of your study—by its research question—and by the type of data that are used.

Quantitative research questions usually ask about relationships among multiple variables, and data are usually observational rather than experimental. By far, the most common tool used to analyze such data is multiple regression analysis. Multiple regression analysis allows researchers to assess the strength of the relationship between an outcome (the dependent variable) and several predictor variables as well as the importance of each of the predictors to the relationship, often with the effect of other predictors statistically eliminated.

It is important to point out, however, that multiple regression analysis is a statistical technique, not a research design, and as such, it does not establish causation. This is because multiple regression builds on correlation, which shows mere associations between variables. To infer a causal relationship, re- searchers need to eliminate bias resulting, for example, from variables that cannot be observed. This can be done by design—through experimental manipulation of variables, or by using statistical controls. The second option is much more common in studies of public policy and economics. Various approaches can be used to minimize bias due to reverse causality and omitted variables. Panel regression with fixed effects is one example of a commonly used approach in economics research. However, panel regression requires the use of panel data, which may not always be available, and they, too, have limitations. It is, therefore, wise to keep in mind when interpreting results, that even under the best of circumstances, statistical controls are never fool-proof.

Qualitative data analysis in public policy depends on whether the study is data-based or literature-based. In data-based studies (e.g., studies based on data collection through interviews, focus group discussions, or participant observation), data analysis involves transcribing and coding participants’ responses and/or the researcher’s notes by identifying certain themes or patterns in the data that help answer the research question(s). In many ways, qualitative data analysis is an attempt to reduce a very large amount of qualitative data—participants’ responses and comments—to a few themes. For example, if your study has looked at how poor women in rural areas cope with violence, you may want to analyze the women’s responses to identify the strategies that they have used. You would have to make many subjective decisions about what the women’s responses really mean and you would need to be very clear about how you made those decisions. Using multiple sources of data (e.g., interviews + documents + observation) in a qualitative study is one strategy to reduce subjectivity.

In literature-based studies, there are usually no data and a paper may be based solely on summarizing an often arbitrary selection of studies or other documents. In such studies, it is common for authors to explain how the literature was located, how the specific studies and documents were selected, and how they help answer the research question(s).

Multiple Comparisons

When making many statistical comparisons, i.e., performing multiple hypothesis tests, a certain fraction of the test statistics will be statistically significant even when the null hypothesis is true. In general, when a series of tests is performed at the α significance level, approximately α × 100% of tests will be significant at the α level even when the null hypothesis for each test is true. For example, even if the null hypotheses are true for all tests, when conducting many independent hypothesis tests at the 0.05 significance level, on average (in the long term) 5 of 100 tests will be significant by chance alone. Issues of multiple comparisons arise in various situations, such as in clinical trials with multiple end points and multiple looks at the data. By doing multiple tests, you naturally increase your chances of making a type I error if no adjustment is made to the usual testing framework for a single test statistic. Pairwise comparison among the sample means of several groups is also an area in which issues of multiple comparisons may be of concern. For k groups, there are k(k – 1)/2 pairwise comparisons, and just by chance some may reach significance. Our last example is with multiple regression analysis in which many candidate predictor variables are tested and entered into the model. Some of these variables may result in a significant result just by chance. With an ongoing study and many interim analyses or inspections of the data, with no adjustment for performing multiple comparisons, we have a high chance of rejecting the null hypothesis at some time point even when the null hypothesis is true.

There are various approaches to the multiple comparisons problem. First, consider if multiple comparisons is actually a problem. If we ask multiple questions we expect multiple answers. If we ask related questions we expect related answers. Looking at the totality of the evidence when interpreting results is far more useful than overzealous correction for multiple comparisons (or ignoring all but the single significant p-value out of 50). One rather informal approach to multiple comparisons is to choose a significance level α lower than the traditional 0.05 level (e.g., 0.01) to prevent many false-positive conclusions or to “control the false discovery rate.” The number of comparisons should be made explicit in the article. More formal approaches to control the “experiment–wise” type I error using corrections for multiple comparisons have been proposed. An example is the Bonferroni correction, in which the type I error rate is taken as α/n, where n is the number of comparisons made. Another class of methods has been developed to correct for multiple comparisons that result from monitoring trial results during the trial. Interim monitoring methods that control the type I error rate are available for various study designs and are discussed further in Chapter 27.14 The classic reference by Hochberg and Tamhane provides a broader discussion of methodology to adjust for multiple comparisons.15

It is best to address the issue of multiple comparisons during the design stage of a study. One should determine how many comparisons will be made and then explicitly state these comparisons. Studies should generally be designed to minimize the number of statistical tests at the end of the study. Ad hoc solutions to the multiple comparisons problem may be done for exploratory or epidemiologic studies. Multiple comparison adjustments should be made for the primary analyses of definitive studies (such as phase III confirmatory studies) to rigorously maintain the type I error rate, i.e., the probability of falsely rejecting any null among those tested, at the chosen α level. Studies that focus on a single primary outcome and data analyzed at the end of study avoid the issue of multiple comparisons. The topic of multiple comparisons is expanded in Chapter 27.

13.4 Results

To examine the factors that cause irrational meetings, we created eight videos of meeting scenes and conducted an experiment. Then, analyses were conducted to examine the effects of the outcome of the meeting decision, the emphasis of the risk, and the compliance with the rules on the desirability of the meeting decision and process. Specifically, comparison of means, analysis of variance, analysis of covariance, and analysis of correlation were conducted.

13.4.1 Analysis of the desirability of a meeting decision

First, we calculated the means of the desirability of the decision in each video. In Fig. 13.6 the first left item 1 indicates the low risk and risky outcome (Matsuzaka beef) condition. The other numbers of items are follows: Item 2 for the high risk and risky outcome condition, Item 3 for the low risk and riskless outcome (Imported beef), Item 4 for the high risk and riskless outcome condition, Item 5 for the rule noncompliance and risky outcome condition, Item 6 for the rule compliance and risky outcome condition, Item 7 for the noncompliance and riskless outcome condition, and Item 8 for the rule compliance and riskless outcome condition. For clarity, when the final decision result was Matsuzaka beef, the bar graph was set to “Matsuzaka” and colored black, and when the final decision result was imported beef, the bar graph was set to “Import” and colored white.

Figure 13.6. Mean and standard deviation (SD) for the desirability of decision in each condition.

Fig. 13.6 shows the mean and the SD for the desirability of decision regardless of the emphasis of the risk or the compliance with the rule; the desirability of the decision of the meeting tended to be higher at the level where the outcome of the decision was imported cattle. Next, we conducted a two-level analysis of variance for each of the two between-subjects factors separately for risk intensity and decision outcome and for rule compliance and decision outcome.

To examine the influence of the degree of risk and the decision of the meeting on the desirability of the decision, we conducted a two-level analysis of variance for each of the two between-subjects factors of risk (high risk vs low risk) and decision outcome (Matsuzaka beef (risky outcome) vs imported beef (riskless outcome)). As a result, a significant main effect was found for the decision outcome in the meeting (F(1,59)=21.25, P<.001). Regardless of the emphasis of the risk, the final decision for imported beef was evaluated as more desirable than that for Matsuzaka beef.

Next, to examine the effects of rule compliance and decision outcome on the desirability of the decision, we conducted a two-level analysis of variance for each of the two between-subjects factors of rule compliance (rule compliance vs rule noncompliance) and decision outcome (Matsuzaka beef vs imported beef). As a result, a significant main effect was found for the decision outcome in the meeting (F(1,57)=4.98, P<.05). Regardless of whether the rule was adhered to or not, the final decision for imported beef was evaluated as more desirable than that for Matsuzaka beef.

In the multiple regression analysis of the desirability of the decision outcome of the meeting, since it was not possible to analyze the three factors together due to the experimental design, the decision outcome (Matsuzaka beef vs imported beef), risk (high vs low), and compliance with the rule (control vs rule compliance vs rule noncompliance) were used as independent variables. Multiple regression analysis was conducted to predict the desirability of the decision from these independent variables. The control condition for the presence or absence of rule compliance was used as the low risk condition, in which there was no conflict over the rule in the majority vote. The results showed a positive partial regression coefficient that was significant at the 0.1% level (t=4.81, P<.001). In other words, it was recognized that the outcome of the meeting was more desirably evaluated when the final decision was made for imported cattle. There was a significant negative partial regression coefficient at the 5% level in the noncompliance condition (t=−2.07, P<.05). In other words, the condition in which the decision was overturned due to noncompliance with the rules was rated as significantly less desirable than the control condition of rule compliance.

We conducted an analysis of covariance with the decision outcome as the predictor variable, the desirability of the process in the meeting as the covariate, and the desirability of the decision in the meeting as the dependent variable. The results showed that the desirability of the meeting decision was significantly higher in the condition in which the decision outcome was imported beef than in the condition in which the decision outcome was Matsuzaka beef (t=5.18, P<.001).

Next, we conducted an analysis of covariance with risk intensity as the predictor variable, desirability of the meeting process as the covariate, and desirability of the meeting decision as the dependent variable. The results showed that there was no effect of risk intensity on the desirability of meeting decisions (t=1.30, n.s.).

13.4.2 Analysis of the desirability of the meeting process

Fig. 13.7 shows the mean and SD of the desirability of the meeting process for each video as a bar graph. The numbers of items indicate the same conditions as mentioned in the previous section.

Figure 13.7. Mean and SD for the desirability of meeting process in each condition.

Unlike the desirability of the decision, no difference was found between the decision results of Matsuzaka and imported cattle. In addition, regardless of the final decision result, the evaluation tended to be higher in the rule compliance condition than in the rule noncompliance condition. Next, we conducted a two-level analysis of variance for each of the two between-subjects factors, dividing the results into the emphasis of risk and decision outcome, and the presence or absence of rule compliance and decision outcome.

To examine the influence of risk emphasis and decision outcome on the desirability of the process, we conducted a two-level analysis of variance for each of the two between-subjects factors: risk (high vs low) and decision outcome (Matsuzaka vs imported beef). The main effects of risk (high vs low) and decision outcome (Matsuzaka vs imported beef) on the desirability of the process in the meeting and their interactions were not significant (F(1,59)=0.02, n.s., F(1,59)=0.40, n.s.).

Next, to examine the influence of rule compliance and decision outcome on the desirability of the process, we conducted a two-level analysis of variance between subjects for rule compliance (adherence vs nonadherence) and decision outcome (Matsuzaka beef vs imported beef). A two-level analysis of variance was conducted for each factor. As a result, no factor was found to affect the desirability of the process in the meeting.

As in the multiple regression analysis of the desirability of decision-making, we conducted a multiple regression analysis to predict the desirability of the process in the meeting from the three factors of decision outcome, risk intensity, and rule compliance. As a result, no independent variable was found to influence the dependent variable (for all t <2.00, n.s.).

An analysis of covariance was conducted with the rule compliance factor as the predictor variable, desirability of meeting decisions as the covariate, and desirability of meeting process as the dependent variable. The results of the analysis of covariance showed that there was no effect of the factors of noncompliance (t=−1.39, n.s.) and compliance (t=0.69, n.s.) on the desirability of the meeting process.

2.4 Types of Variables × Number of Categories and Scales of Accuracy

Qualitative or categorical variables can also be classified based on the number of categories: (a) dichotomous or binary (dummies), when they only take on two categories; (b) polychotomous, when they take on more than two categories.

On the other hand, metric or quantitative variables can also be classified based on the scale of accuracy: discrete or continuous.

This classification can be seen in Fig. 2.11.

4.2 Microeconomic Modeling