Drawing on the literature review, this paper now moves towards an explanation of the evaluation exercises carried out so far on Infocentros, or ‘an evaluation of evaluations’. An interpretivist approach – associated with the ‘understanding zone’ – it analyses the evaluation processes through considering context, is used for the analysis. In particular by using Pettigrew’s contextualist approach (1985), whose components of analysis include the context, the content and the process, this study explains the interactions between the processes of evaluation of Infocentros and the context within which they have taken place.
According to Pettigrew (1985), the contextualist approach is useful in early approaches to organisational change because it focuses on two levels of analysis – horizontal and vertical level –and the interconnections between the two over the time. The successive interconnections of events in historical, present and future time are referred to as the horizontal level. Similarly, the vertical level refers to the environment, including the national and international environment within which change takes place.
The contextualist framework provides a theory for research into change, which can also guide practice. It helps to examine change as a process in an historical and contextual way. Furthermore, the approach makes a distinction between the object of change – ‘the what of change’ or the content -, ‘the why of change’ or the context which refers to the issues or factors in the environment in which the change occurs, and the process – ‘the how of change’- which analyses how the change process unfolds in a temporal manner (Pettigrew, 1985).
Pettigrew’s ideas have been considered for the specific case of IS evaluation for Serafeimidis and Smithson (Serafeimidis et al, 1996a; Serafeimidis et al, 1996b), who purposes a content-context-process model which the content of evaluation answers what is measured or monitored by considering the criteria and values. The context (internal and external) responds the questions: why is the evaluation carried out?, who is involved?, are there conflicting aims or interests?, and what level of change is aimed at? Finally, the process considers how the evaluation is done?, how the issues are perceived? when the evaluation is done?, how often is it done?, how the results are to be made available?, and how are they linked to organisational learning? The analysis, in conclusion, will utilise a W-framework approach based on the ideas discussed above for the assessment of each evaluation of Infocentros, and a posterior analysis of the changes and evolution of the whole evaluation practices in the Infocentros Programme. A summary of the W-framework is provided in Table 1.
Table 1- W-Framework
When considering the ‘WHAT’ component of evaluation– the content-, it has been argued that different types of IS require different types of evaluation frameworks (Farbey, et al, 1995). As a consequence, it was found necessary to complement the W-framework with a more specific telecentre and ICT-for-development analysis tool. Specifically by expanding the W-framework with ‘The Guiding Principles of Telecentre Evaluation’ (Gómez, et al, 1999a). In sum, the ten criteria for telecentre evaluation are incorporated into the analysis as a means to complement the general W-framework’s Context-Content-Process evaluation approach. We intend to provide an analysis of the different evaluation processes which allows a concurrent comparative analysis of the evaluations of Infocentros carried out so far.
‘The guiding principles for evaluation of telecentres’ document was built from discussions among telecentre practitioners and researchers, and primarily gives guidance for telecentre evaluation to be “useful, financially responsible, builds local capacity; and enables shared learning” (Reilly, et al, 2001). A list with the principles is shown in Appendix 3.
Reilly and Gómez (2001) used these principles for a regional comparative study between evaluation practices used in Latin America and in Asia. In this study, these principles will be used to compare approaches used in Venezuela but in the different evaluations of Infocentros Programme. A summary of the guiding principles is presented in Table 2 (Reilly et al, 2001).
Table 2- The guiding principles for evaluation of telecentres
An interpretive, exploratory case study approach has been selected as the most appropriate for the purposes of this paper. One of the reasons behind this selection is that the main research question concerns the way telecentres are evaluated in practice. Case studies offer the ideal research strategy to answer this kind of questions (Yin, 1989). In addition, Yin (1989) argues that case studies are adequate to address new phenomena in realistic circumstances; telecentres are a relatively new phenomenon in Venezuela. Moreover the problem of telecentre evaluation needs to be studied in context. Such an approach accords with the Information Systems research approach which considers the correlation between contextual factors and the use of ICTs in development – that is, situated studies (Avgerou and Madon, 2002).
The research was conducted within the National Centre of Information Technologies (NCIT), the Venezuelan government institution in charge of the Infocentros programme. The empirical work in this research was conducted mainly through qualitative semi-structured interviews (Appendix 4), using open-ended questions in order to ensure a high degree of veracity, and additionally, to provide useful insights into the way the evaluation process is performed. Directors of the NCTI, including its President, Executive Director and Project Manager, and staff in charge of evaluation directly supervised by the Project Manager, were the people interviewed to gain an insight into their criteria for telecentre evaluation and their opinions about the programme. Notwithstanding that there is a new division in the NCIT called Infocentros, it was not possible to work in conjunction with directors and staff of the department because at the time of this study, they were “in an intense period of openings”. Academics and practitioners related to the ICT and development sector in Venezuela were also interviewed in order to obtain viewpoints from those unrelated to government.
Given the senior positions in government of many of the respondents and due to the importance of avoiding any biased responses, full confidentiality of the data provided was promised to the interviewees. Informal discussion sessions were also performed in an attempt to verify the evidence supported in the formal interviews.
Within the interviews, several main themes were pursued. First of all, general aspects about the Infocentros programme were addressed such as its evolution, impacts and managerial issues. Secondly, Infocentros evaluation mechanisms including: opinions about earlier evaluations, evaluation strategy or methodology, teams and criteria for samples and pilots were examined. Finally, it was the intention of the study to establish the impact of past evaluations in terms of influence on improvements to the programme and as learning exercises for later evaluations.
Relevant official documents such as the past evaluation reports of the Infocentros Programme and NCIT reports about the creation of the Infocentros programme were secondary sources also examined. Some Infocentros in Caracas were visited for the purpose of contributing to the current situation of the programme in terms of its evaluation and its evolution and to see to what extent recommendations from past evaluations were considered. Similarly, brochures, official gazettes, NCIT’s, and Infocentros’ web sites were examined in terms of information on policies and procedures implemented during the period.
Data for this study was collected across a six-week period, beginning at the end of June 2004. All data collected was analysed and used to answer the research questions outlined above. Evidence from questionnaires, relevant documents and websites was also incorporated to offer comprehensive answers to the primary and secondary research questions (Yin, 1981).
The methodology on which this paper is based mainly draws on the application of the theoretical framework above and the analysis of primary and secondary sources, interviews, observations, and web site examination. Interest in this initiative derives from its unique context and importance, and that, as a Venezuelan citizen, the author is interested in the efforts which have been implemented to improve Venezuela’s political, economic and social problems.