Empirical Research Methods - in Electronic Commerce
Notes following the Bled Conference and Uni. Linz Presentations
9 and 13 June 1994
Introduction
This presentation was given at the Electronic Commerce Research & Teaching
Symposium in Bled on 9 June 1994, and again at the University of Linz on 13
June 1994. It was generally well-received, even by the most experienced and
influential academics present. These included Profs. Milt Jenkins (Baltimore)
and Doug Vogel (Arizona), who wrote two of the primary references in the area,
and Ron Lee (Erasmus), and Dr Juliet Webster (East London). Several, including
Doug and Juliet, requested copies of the slides, because they provide a tight
synthesis and checklists which can support a short tutorial, a longer workshop
or a panel discussion.
The slides are modular, so that they can be applied to IS research generally,
or to research in a particular topic, in this case electronic commerce. The
first slide lends itself to improved graphics, beyond the hand-drawn one I
used. But colour is useful. It would be handy to have both a series of
overlay slides to build up the story (for tutorials), and a single complete
slide (for mature audiences).
Underlying Philosophy
The underlying philosophy is that students preparing to undertake research
should be educated at two levels. They should learn about the philosophy of
science (underpinned by a small amount about the history of science), and about
research methodology. They should learn the process of conventional scientific
endeavour. They should then undertake their research within that framework, as
an exercise or discipline. They should, in effect, suspend their disbelief,
and strive for a (largely unattainable) level of objectivity. They should,
however, finish their graduate work with the insight and the intellectual
capability to constructively criticise the shortcomings of the work of others,
and of themselves.
A future version of this document could be useful as pre-reading for COMM8201,
and/or as accompanying documentation for IS students undertaking COMM8013.
Conventional Scientific Method
- There is a real world, comprising objects and processes. This real world
cannot be directly understood by humans, nor 'captured' into human artefacts.
However it can be observed. On the basis of observation of the real world,
humans form theories as to how it came to be the way it is, and how and why the
processes take place. In an applied discipline such as Information Systems, it
is common to depend on theories prepared by 'reference disciplines'.
- Theories should ideally be expressed in deductive form, such that a set of
axioms or postulates, operated on by conventional deductive logic, lead to
inferences being drawn. A scientific theory, á la Karl Popper, is one
which is capable of generating inferences which are, at least in principle,
refutable by reference to the real world. For example, religious and
ideological theories generally result in inferences such as "through grace, a
believer is saved" (Christian liturgy) and "only a virtuous ruler can survive"
(Machiavelli), which cannot be subjected to such testing.
- In order to test whether the theory has the capability to describe,
explain, and especially to predict, the behaviour of the real world, it is
necessary to construct tests of the inferences arising from a theory.
Inferences are generally expressed at a conceptual level, however; for
example, "experienced auditors are better judges of the extent to which
substantive tests of controls should be undertaken". In order to actually
perform tests, it is necessary to express them in 'operational' terms, in the
form generally referred to as hypotheses, e.g. "the proportion of auditors who
select the same extent of substantive testing as recommended by a panel of
experts will be highly correlated with their number of years of auditing
experience".
- The outcomes of hypothesis testing provide feedback to the theory. If the
hypotheses are not confirmed, then evidence exists that the theory may be
wrong, and if it can be reproduced by others, then at least some aspects of the
theory should be regarded as refuted. If the data support the hypotheses, then
there is justification for increased or sustained confidence in the theory's
effectiveness, or at least continued use of the theory. As findings accumulate
which support a wide range of hypotheses generated from the theory, it tends to
be regarded as authoritative within some range of applicability.
- Empirical research is that which is based on observations of the real
world. Empiricism is relevant at two points in the conventional cycle:
- during the theory formation stage, relatively passive observation is
undertaken. It is certainly influenced by the observers' particular
perspectives, but tends not to be directly guided by any particular theory;
- in gathering data to test hypotheses, on the other hand, relatively active
observation is undertaken, guided by the theory, and gathering precisely that
data which the researcher considers necessary to test the hypotheses.
- It is important to appreciate the role of theory. It is perfectly
feasible to create hypotheses directly from a small collection of observations,
without any statement of underlying principles and logical derivation of
inferences. The weakness of this approach is that the outcomes of the research
can merely refute, or provide conditional support for, those specific
hypotheses; there is no accumulation of knowledge. On the other hand, where
the hypotheses are derived from a body of theory, the results arising from the
research accumulate, and can be used again by the researcher and others to
flesh out knowledge.
- In summary, 'normal science' consists of extracting new hypotheses from an
existing theory, testing them, and adding the results to the pool of knowledge.
- But Information Systems is an applied discipline, and lacks a solid body
of existing theory. In practice, therefore, it is uncommon in IS to have a
theory readily available. It is most common to have a partial theory which
requires embellishment, and/or to use a theory from a 'reference discipline'
such as organisational behaviour, management accounting or computer science.
- In the various reference disciplines, many theories are available. Many
of them, however, were developed for purposes related at best tenuously with
IS. Some which appear relevant are in fact quite inappropriate, e.g. because
they are specific to a particular culture (often the U.S.A. or some part
thereof), or the unit of analysis is individuals in a loose-knit community
rather than individuals acting within an organisational context (important
instance: Diffusion of Innovations theory).
- Where no appropriate theory can be found as a basis for research, it may
be necessary to undertake 'exploratory research'. This involves open-ended
study, unguided by theory and intended to provide a new body of empirical
knowledge from which theories might be able to be postulated.
Post-Positivist Perspectives
- Conventional research method is based on 'rational positivist' thought.
This includes the presumptions that there is a 'real world', that data can be
gathered by observing it, and that those data are factual, truthful and
unambiguous. The 'post-positivist', 'interpretivist' philosophy, on the other
hand, asserts that these assumptions are unwarranted, that 'facts' and 'truth'
are a chimera, that 'objective' observation is impossible, and that the act of
observation-and-interpretation is dependent on the perspective adopted by the
observer.
- Interpretivists criticise even the physical scientists for the narrowness
of their assumptions, but their criticisms strike home particularly in the
social sciences, where the objects of study are influenced by so many factors,
and are incapable of being isolated and controlled in experimental laboratory
settings.
Empirical Research Techniques
- A wide range of techniques exist whereby hypothesis testing can be
undertaken. 'Scientific' techniques are distinguishable from 'interpretevist'
techniques on the basis of their degree of inherent 'rigour', in particular
whether they generate data whose quality can be tested, e.g. by way of
replication or statistical analysis.
- Empirical testing techniques involve observation of the 'real world'.
Many of them are 'scientific', but some interpretivist approaches are also
empirical: the test is not the degree of rigour, but the extent to which the
real world is observed. [These slides need to be augmented with the more
formal qualitative techniques, e.g. protocol analysis, 'grounded theory'?]
The Quality of IS Research
- The IS literature contains many expressions of concern about the quality
of IS research.
- Until the mid-1980s, 'conceptual' (i.e. non-empirical) papers dominated
the literature, even in the most highly-regarded journals. Despite this, there
has been, and remains, a shortage of theories, and even an absence of a
'cumulative tradition' which would lead to the establishment of theories.
Where pre-theoretical 'research frameworks' have been established, they have
not been used to any great extent.
- There has been, and remains, a shortage of validated constructs (i.e.
conceptually-expressed inferences for which operational definitions have been
established and tested in a variety of settings), and of validated survey
instruments.
- The quality of statistical understanding and analysis demonstrated by
researchers has been relatively low.
- There has been inadequate application of the principle of 'triangulation',
i.e. the gathering of data from multiple sources, and the comparison and
cross-analysis of that data. Of particular importance has been the inadequate
exploitation of the complementariness between quantitative and qualitative data.
- Case study research has often been applied inappropriately, or undertaken
with insufficient rigour.
- The study of time-variant phenomena necessitates the use of longitudinal
rather than cross-sectional studies, but longitudinal studies are uncommon.
Electronic Commerce
- On the basis of the preceding discussion, the specific difficulties
confronting empirical research into electronic commerce can be examined. I
used the research designs in my own Research Programme in Supra-Organisational
Systems (specifically GPIE, ITEA, Biel Smart-Card and on-line trading programs)
as a basis for discussion.
Conclusion
- I intended to finish with a slide providing a set of criteria for a
research design, but the current draft is a bit banal compared to the other
slides.
RAC - 14 June 1994
Navigation
Go to
Roger's
Home Page.
Go to
the
contents-page for this segment.
Send
an email to Roger
Last Amended: 11 October 1995
 |
These community service pages are a joint offering
of the Australian National University (which provides the infrastructure), and
Roger Clarke (who provides the content). |
 |