Requirements for development and adaptation of complex measurement tools in sociology: reliability, validity and trustworthiness
The article dwells upon main characteristics of sociological research quality (reliability, validity and trustworthiness) as well as general logic of their testing. Complex measurement instruments divided into scales and indexes are in the focus. Scales aim to measure certain qualities that are not explicitly observed (latent variables) and determine characteristics, which underlie the indicators of the measurement technique. When using integral indices, latent variables or social phenomenon are determined by the factors based on which indicators are formulated. Reliability is viewed as repeatability of measurement or absence of random errors.
Authors distinguish such approaches towards reliability testing as a test-retest (level of rigidity of the measurement in some time), parallel forms (level of equivalence of two forms of the same scale), and internal consistency (consistency of results of different indicators). In the most general perspective, validity is viewed as correspondence degree of the research process results to the reality. Integrated approach towards validation of measurement scales and integral indices includes two stages – theoretical and empirical validation.
Theoretical validation implies reaching the sufficient level of correspondence of theoretical construct to phenomenon under study, while empirical validation implies testing the accordance of measurement technique and results that were obtained while using it with the theoretical construct. Thus, theoretical validity is achieved first, and then empirical validity is tested. Trustworthiness of measurement is viewed as a respondent’s capability and desire to report information in which the researcher is interested. Usage of reliable and valid measurement technique does not guarantee obtaining required data when it is not possible to assure trustworthiness of measurement.
It is recommended to use index of subjective reliability to test trustworthiness. Recommendations regarding general logic of validity and reliability testing of complex measurement techniques and requirements towards design and sample size of the survey are presented in the article.
DeVellis, R. (2012). Scale Development: Theory and Applications. Los Angeles; London; New Delhi; Singapore; Washington DC: SAGE
Alreck, P.L., Settle, R.B. (1985). The Survey Research Handbook. Homewood, IL: Dow Jones-Irwin
Carmines, E.G., Zeller, R.A. (1979). Reliability and Validity Assessment. Beverly Hills, California: SAGE Publications
Paniotto, V. (1986). Quality of sociological information. Kyiv: Naukova dumka [in Russian]
Trochim, W. Types of Reliability. URL: http://www.socialresearchmethods.net/kb/reltypes.php
Mueller, R. (1996). Basic Principles of Structural Equation Modeling. An Introduction to LISREL and EQS. New York: Springer-Verlag New York, Inc.
Johnston, J., Pennypacker, H. (1980). Strategies and tactics of human behavioral research. New Jersey: Erlbaum
Dembitskii, S. (2014). To the problem of statistical verification of theoretical constructs. Sotsiologiya: teoriya, metody, marketing – Sociology: theory, methods, marketing, 2, 190-194 [in Russian]
Dembitskii, S. (2016). Theoretical validation in sociological research. Moscow: LENAND [in Russian]
Dembitskii, S. (2012). Theorethical validation of measurement scales. Sotsiologiya: teoriya, metody, marketing – Sociology: theory, methods, marketing, 2, 53-65 [in Russian]
Yin, R. (2009). Case Study Research: Design and Methods. London; Thousand Oaks; New Delhi: Sage
Tilli, Ch. (2009). Coercion, Capital, and European States, AD 990–1990. 1990–1992. Moscow: Territory of the Future [in Russian]
Anderson, B. (2001). Imagined Communities: Reflections on the Origin and Spread of Nationalism. Moscow: Kanon-press-Ts: Kuchkovo pole [in Russian]
Derogatis, L., Fitzpatrick, M. (2004). The SCL-90-R, the Brief Symptom Inventory (BSI), and the BSI-18. In M.E. Maruish (Ed.), The use of Psychological Testing for Treatment Planning and Outcomes Assessment. Vol. 3: Instruments for Adults (pp. 1–42). New Jersey; London: Lawrence Erlbaum Associated Publishers
Golovakha, E., Panina, N. (1997). Integral index of social well-being (IISWB): design and application of socio-logical test in surveys. Kyiv: Institute of Sociology of NAS of Ukraine [in Russian]
Campbell, D., Fiske, D. (1959). Convergent and Discriminant Validation by the Multitrait-multimethod Matrix. Psychological Bulletin, 2, 81-105
Volovich, V. (1976). Reliability of information in a sociological research. Problems of methodology and techniques. Kyiv [in Russian]
Shmelev, A. (1996). Fundamentals of Psychodiagnostics. Rostov-na-Donu: Feniks [in Russian]
McNabb, D. (2014). Nonsampling error in social surveys. Los Angeles, London, New Delhi, Singapore, Washington DC: SAGE
Golovakha, Ye., Gorbachyk, A., Lyubyva, T., Panіna, N., Sereda, V., Ursulenko, K. (2008). Subjective Reliability: Theory and Method of Measurement (ISR). Sotsіolohіia: teorіia, metody, marketynh – Sociology: theory, methods, marketing, 1, 166-188 [in Ukrainian]