| Sign In to gain access to subscriptions and/or personal tools. |
Determining the Explanatory Potential of Variables
1 (PCT 117) Department of Preventive and Community Dentistry and Pedodontology, University of Nijmegen, Postbus 9101, 6500 HB Nijmegen, the Netherlands; Correspondence: *corresponding author, D.Abraham{at}dent.kun.nl
A meta-analysis may not be the most appropriate method for the extraction of information on independent variables from the scientific literature. A formal quantitative review method is proposed for reducing a literature review variable list to a manageable list of explanatory variables. The proposed method synthesizes the information in a standard literature review into two steps. First, the hypothesis "independent variable ‘X’ had no effect on utilization" is tested by means of binomial, one-sided testing. If it is rejected, then the hypothesis "The effect of independent variable ‘X’ is as likely to be in the same direction as it is to be in the opposite direction" is tested, by an extended version of the binomial formula. In two steps, the presence of a consistent direction of significant effects is calculated. This method is tested on dental utilization publications. It is concluded that the method is an effective tool for the creation of a manageable variable list.
Key Words: policy evaluation • literature review • explanatory variables
In 1995, the Dutch government reformed the public health insurance system (den Dekker, 1995; Peddemors, 1995; Swinkels and Maessen, 1996; Ziekenfondsraad, 1996, 1997). The reform affected approximately 65% of the Dutch population covered by public health insurance. An evaluation was planned to study the effects of this insurance reform. This evaluation study regarding the role of dental insurance in dental utilization required a variable list comprised of independent explanatory variables. Because of the numerous publications on utilization in dentistry, the researchers were confronted with the problem of over-information. An existing theory for modeling dental utilization that includes dental insurance as an explanatory variable would have provided a basis for selecting the other independent variables for the study. Several such dental utilization theories exist, but there is no consensus regarding the most appropriate model. And even the most frequently chosen model, by Andersen and Newman (1973), has limited explanatory power (Kiyak, 1987). Thus, there is no forceful theoretical argument for restricting the potential explanatory variables for the evaluation study to the variables used in these models. Therefore, another method was needed for the selection of independent variables and the determination of whether they have significant explanatory potential. The method of meta-analysis was not feasible, because many publications did not meet the requirements for this type of systematic review (Petitti, 1994; Chalmers and Altman, 1995). First and foremost, there is a lack of (randomized) controlled trials that include dental insurance as an explanatory variable. Weighing the available survey-type studies on the basis of sample size presents a dilemma, since the existence, not the magnitude, of the explanatory power is the focus in the proposed review method. Second, the application of a meta-analysis would imply the a priori deletion of independent variables used in the studies not considered suitable for the review. Finally, most publications report on studies that use extensive independent variable lists. It would therefore lead to hundreds of meta-analyses, one for each independent variable, to assess the influence of that variable on dental utilization. Making use of the literature review instead would have avoided the a priori deletion of independent variables; however, this review method offers no possibility for an objective filtering of the variables with explanatory potential. The present article describes an approach that was developed to create an independent variable list based on explanatory potential. The research question is: Does the use of a formal method make it possible to create a manageable list of explanatory variables that is based on an extensive independent variable list resulting from a literature review? A quantitative method was developed that introduces objectivity into the process of selecting explanatory variables. The main prerequisite was that the method had to make possible the statistical testing of combined results. The proposed method was tested using the results of a literature review on utilization in dental care. The method is first described, after which the application of the method to dental utilization literature is presented.
The computerized databases MedLine and Current Contents, with Silver Platter as an addition to these databases, were used to search for publications on dental utilization published after 1984. In 1985, Manning's paper on the RAND Health Insurance Study was published, marking the start of a period of increased interest in dental utilization research and dental insurance reform (Manning et al., 1985; Arinen et al., 1996; Schwarz, 1996a,b). The period covered is 1985 to 1998. The search used the following MeSH terms and key words for the description of dental utilization: access, care delivery, consumption, demand, expenditure, and utilization. The numerous synonyms for dental utilization made it necessary to use different key words. The automated search was supplemented with articles in footnotes, reference lists, and bibliographies (Louden, 1978). No abstracts or books were included. Limiting conditions were introduced, including language (Dutch or English) and subset (dental or dentistry). A total of 5467 articles matched these criteria. It was found that many publications did not report on demand or utilization but on oral health as the outcome measure. Therefore, the definitions for demand and utilization based on the work of Stoddard and Barer (Barer et al., 1998) and proposed by Grytten (1991) were applied as a filter. Demand is defined as "a request by a patient for care", and utilization is "the amount of care received". For studies based on the dentist's perspective, demand and utilization were defined as "the number of patients requesting care", and utilization as "the amount of care provided per patient" (Grytten, 1992). Finally, two criteria were used to filter the publications further. First, the papers had to report on populations over the age of 18; second, statistics had to be multivariate. Only 144 articles met these criteria (reference list available on request). The 144 articles included 143 survey-type studies and 1 controlled trial. The independent variables in these publications were recorded, which resulted in a list of 538 variables. None of the studies included specified whether the independents were variables or co-variables. The independent variables were grouped into patient, dentist, and system variables. Synonyms were merged into one term. These included variables mentioned in both English and Dutch and variables in different wording, such as dental fear and dental anxiety. The process of excluding synonyms was repeated independently by the senior researchers, and consensus was reached on the resulting list of variables. There were several variables which, although not synonyms, appeared to be the practical interpretation of more abstract variables. These abstract variables, such as "Dental Health Attitudes", were considered concepts. Variables were thus grouped according to concept, resulting in 24 patient and 20 dentist concepts. The articles were then re-read, so that we could record the significance of the statistical effects reported for each concept. In the analyses, it was assumed that the results presented in the 144 selected articles were of comparable reliability and validity. The first phase in the analyses was to determine, for each variable, the overall statistical significance of the measured effects. The point of departure was the binomial, one-sided testing of the hypothesis that "Independent variable ‘X’ had no effect on dental utilization". This binomial formula was the basis for the calculation of the total likelihood P (the number of significant effects reported in the publications or more—but at most the total number of reports in the literature), given N studies reporting on independent variable X. In binomial testing, the likelihood of "success" requires specification. In this proposed review method, the chance of success is a significant effect of a specific variable reported in one study. In research, we accept a possibility of 5%—referred to as the alpha value—that a study will report a significant effect when in fact there is no such effect. Thus, the chance of success was set at 5%, given the hypothesis of no effect. The number of studies reporting a significant effect was abbreviated to Nsgf and represents the sum of significant effects in the same direction as the independent variable ‘X’ (Npe) and significant effects in the opposite direction as the independent variable (Nne). With Ntotal studies reporting on X, Nsgf of which report a significant effect and Nns of which report no significant effect, the test was based on Bin(Ntotal, Nsgf, 5%). The binomial testing resulted in a list of variables for which the hypothesis that the variable had no effect was rejected. To put it simply, the statistically significant effects of these variables on dental utilization were determined. Next, the directions of the effects of these variables were tested. Again, N is the total number of studies reporting on the effect of the variable under consideration. The effect of independent variable ‘X’ is recorded based on the following categories:
The consistency in the direction of the significant effects was calculated in a manner analogous to binomial testing, with a generalization of this formula known as the multinomial formula. The hypothesis tested read "The effect of independent variable ‘X’ is as likely to be in the same direction as it is to be in the opposite direction". Given this second hypothesis, the likelihood ‘p’ of finding an effect in the same direction (Npe) or an effect in the opposite direction (Nne) can be calculated:
The total likelihood of the event (Npe, Nne, and Nns, given ‘p’) can be calculated by means of the following generalization of the binomial formula:
The independent variables are shown in the first column and the dependent variable in the second column of Table 1  . Columns 3 and 4 show the frequencies of significant (Nsgf) and not significant effects (Nns). Table 1 shows that most independent variables have a significant effect on both dental demand and dental utilization. For example, "Patient Treatment Cost" was included in 23 studies that focused on dental utilization. The binomial test answers the question whether this distribution of effects—15 significant and 8 not significant—is exceptional enough to warrant rejecting the hypothesis "Independent variable ‘Patient Treatment Cost’ had no effect on dental utilization". The probability of this proportion occurring (or more significant effects, i.e., between 15 and 23 out of 23) is 0.0%. When the standard alpha value of 5% is applied, the hypothesis is rejected. In this way, the hypothesis "Independent variable ‘X’ had no effect on dental utilization" was tested for all independent variables.
The direction of this relation between ‘X’ and ‘Y’ was determined for those variables that were significantly related to dental utilization according to the binomial test. Continuing the example with the independent variable ‘Patient Treatment Cost’: The proportion of non-significant effects is 8/23, and the proportion of significant effects is 15/23. With regard to the hypothesis, the likelihood of finding a significant effect in the same direction is equal to the likelihood of finding a significant effect in the opposite direction. This likelihood is (15/23)/2. The probability of finding a number of 12 significant effects or more significant effects in the opposite direction in 15 studies is 0.3%. Thus, the hypothesis is rejected. On the basis of the 23 available studies, we can conclude that ‘Patient Treatment Cost’ has a significant influence on dental utilization. This significant influence was found to be consistently in the opposite direction, indicating a decrease in the use of dental care as the treatment cost billed to the patient increases.
The method presented in this article is applied to dental utilization literature. The example demonstrates its usefulness in the creation of a list of variables with explanatory potential. The binomial testing results in a reduced list of variables that are significantly related to dental utilization. The presence or absence of a significant effect is not an absolute argument. The conclusion drawn depends on the types and numbers of studies included in the analyses. This is why the clarification of the publication selection process and the treatment of the independent variables are essential. Regardless of the formal conclusion, it may be of importance to include an independent variable in an evaluation study simply because an effect is expected in that specific circumstance or the power of the studies used in the analyses is questionable. The second hypothesis, "the effect of independent variable ‘X’ is as likely to be in the same as it is to be in the opposite direction", is often accepted for dentist and health insurance system variables. This indicates that, while the effect has been found to be significant in step one, the direction of that effect on dental demand or utilization cannot be decided to be in the same or the opposite direction. This phenomenon could indicate that local factors have a significant impact on the relation between independent variables and dental utilization. This makes the formulation of a hypothesis regarding the effect of an independent variable on dental utilization more difficult, but should not be a direct reason for exclusion of the variable from a study since the effect has been concluded to be significant. In the exercise reported on in this article, it was possible to establish the direction of the effect on dental utilization of most patient variables. At present, a systematic review is considered the gold standard for synthesizing the results of multiple experimental research efforts. The meta-analysis method is used to calculate the aggregate effect of intervention studies. It is also used to assess the importance of individual independent variables in etiological studies. Even though the method has been applied to non-experimental studies, it may not be the best course of action. Meta-analysis has been found to present several problems (Petitti, 1994; Chalmers and Altman, 1995), including such issues as:
Unfortunately, most of these issues—especially the multivariate relationship between the dependent and independent variables—come into play when a meta-analysis is used to study the utilization of dental care. There is no valid and accepted theoretical framework that explores dental utilization. Thus, a variable list cannot be derived from such a theory. In the absence of a theoretical framework, the literature on dental utilization is overwhelming and heterogeneous, thus increasing the risk of "garbage in, garbage out". There are publications that meet the prerequisites for inclusion in meta-analysis. However, there are also many publications that cannot be synthesized through meta-analysis. A great many valuable studies, especially those examining multivariate relationships, would be excluded. The proposed method aims at synthesizing information from publications which are not necessarily suitable for meta-analysis, but which report on sound research that should not be disregarded. Of the 144 studies selected, only one was dealing with a controlled trial (Manning et al., 1985). All others were survey-type studies. The assumption of the equivalence of the results of the 144 papers made in the "MATERIALS & METHODS" section of this paper was based on this lack of randomized controlled trials reported on in the literature. The size of the samples reported on could be an alternative argument for the selection of specific studies. The sample size determines the potential of a study (β) to calculate accurately the effect of an independent variable on the dependent variable. However, the size of a sample has an optimum when the focus is the existence of a statistically significant effect on the dependent variable, and not the precise prediction of the explanatory power of the independent variable. Independent variables with limited explanatory power are detected by larg(er) studies. They contribute little to the overall explanatory potential of the independent variable list. Therefore, weighing studies according to sample size is not efficient in the context of the proposed method. Neither the meta-analysis nor the method of hypothesis testing proposed here can effectively deal with the problem of publication bias (Petitti, 1994). As in a meta-analysis, the number of available studies that use a particular independent variable influences the probability that a conclusion can be reached regarding the effect (if any) of that variable. In the reported review, publications focusing on dental demand and studies including dentist and dental-system variables were found less often. The proposed method aims to introduce reproducibility and objectivity into the interpretation process to a greater degree than the traditional literature review. It limits the subjectivity of the selection process by minutely describing that process and by having it repeated independently by other researchers. The added value of the proposed method is that it does not rely solely on the scrutiny of the researchers in selecting publications and interpreting the information but uses a formal quantitative method for selecting explanatory independent variables. In this publication, the criteria for the selection process and the limits of the search are listed. Both the criteria and the limits used should be open to discussion. However, the value of debating them in great detail is directly related to the need for the determination of a standard or minimal set of criteria for reviewing literature in dental utilization research.
We gratefully acknowledge the financial support of the Dutch Dental Association in the funding of the 1995 insurance reform evaluation study. This study has been conducted at the University of Nijmegen (the Netherlands) by the Department of Community and Preventive Dentistry and Pedodontology, in cooperation with the Department of Social Sciences Research Methods and the Department of Health Organization, Policy and Economics at the University of Maastricht. Received for publication June 4, 2001. Revision received November 27, 2001. Accepted for publication February 12, 2002.
Journal of Dental Research, Vol. 81, No. 4,
289-294 (2002)
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|
 |
|
Sign In to gain access to subscriptions and/or personal tools.
TOP
ABSTRACT
INTRODUCTION
