Evaluation of Infocentros
Although it is not the purpose of this paper to analyse the process of administrative reform in Venezuela during the Chávez administration, it is opportune to mention briefly the sequence of the implemented reforms in this period in order to understand the context and nature of the government which created the Infocentros Programme.
During the presidential campaign Mr. Chávez proclaimed that his main purpose was to clean up the mess that 40 years of corrupt administration had left behind; he would govern with new legislation, new staff, and new institutions to rebuild the country. After two referendums, Venezuela had a new constitution, new ministries, a new congress and the country has a new name “República Bolivariana de Venezuela” rather than “República de Venezuela”. In the same vein, many ministries and institutions changed their names and citizens supporting the president’s ideas were hopeful that these reforms would bring about prosperity (D Sylvia and Danopoulos, 2003).
Creating the Science and Technology Ministry and the NCIT in 1999, Venezuelan officials pursued the establishment of guidelines and projects promoting the use of ICTs in the public sector. The Venezuelan government implemented “the Platino platform”, a centralised model of citizen assistance whose main purpose is the improvement of the relationship between citizens and the State. One of the most important Platino objectives – in terms of digital divide reduction - is the promotion and consolidation of an ICTs culture across the nation and in Venezuelan society.
In terms of policies for the diffusion of technologies, the government established the use of the Internet in all public entities as compulsory. (Official Gazette, 2000) Additionally, the legal framework for contract bidding recognises the use of electronic means as a valid method for sending and receiving information related to the process. Initiatives promoting the citizens’ use of ICTs include the Infocentros project – the implementation of 240 telecentres in the country - and a public schools programme through the Bolivarian Telematic Centres.
In 2000, a Telecommunication Law was approved, regulating the sector and creating funding mechanisms in order to promote research and development in information technology. Legislation concerning the transmission of electronic messages has regulated electronic signatures since 2001, providing security for electronic transactions. Likewise, certification authorities were created alongside the correspondent superintendence.
It can be noted that this was a dynamic period in terms of administrative reform, however one may temper this view in the light of events some three years later, arguing instead that these reforms were merely superficial. Support for the President began to diminish with the apparent failure of the administration. New ‘names’ were not enough to improve the country’s key conditions, and institutional changes fell short of expectations.
Despite the country’s oil wealth, a product of recent oil price recovery in the international markets, 80% of Venezuelans are still poor in 2004. The opposition maintained a campaign to see Chávez ousted and a referendum was activated after the collection of signatures from 20% of the population triggering a recall mechanism inserted into the Venezuelan constitution by Mr Chávez in 1999. After a combative referendum process in which claims of fraudulent activity were offset by authentication of the result by international observers, Mr. Chávez was confirmed as President with 58% of votes. Although the country remains polarised in terms of support for Mr Chávez, he will be president until January 2006. Some analysts argue that Mr Chávez has won people’s hearts and current support with extensive school, health and technology programmes – called missions – and that these programmes were intensified just before the referendum. (BBCi, 2004) For example, telecentres were used for Mr Chávez’s campaign with some advertising exhorting people to riot “… let them steal your benefits, defend your Infocentros” and just before the referendum, telecentre facilities were used for checking citizens’ electoral information through an e-government application. Although the future of Venezuela after the referendum cannot be predicted, it can be argued that this administration’s social programmes are important government tools at least as a means for gaining political support.
In 1997 the Venezuelan government launched the “Cabinas de Acceso Publico a Internet” project - Internet access public cabins - which was the first programme which intended to provide mass access to information technologies to all social classes. Following the basic model of a franchise, which had been implemented in Latin America and other developing countries, users were asked to pay a minimum fee for the services offered in the cabin such as printing and scanning of documents, access to the internet, and usage of computers for word processing. With this scheme the financial sustainability of the project was viable and three cabins were implemented in each public library under this model.
It was then, during 2000, that the President declared that additional services would be charged to people but internet access and the usage of computers in cabins would be free services. In this context the programme name changed to Infocentros and, with this presidential declaration the financial sustainability of the project is questioned. (Barrera, 2002)
Infocentros was one of the programmes included in the New Information Technologies National Plan 2000, which also included The Intelligent Portal, Thematic databases for health, environment, commerce and science and technology, and a dot COM incubator. (NCIT, 2000) All these projects were linked to the philosophy of the Infocentros because they promoted shared knowledge when this is produced through the use of ICTs.
With actions and objectives defined, the NCIT and the Ministry of Science and Technology (MST) began the planning and management of the project which resulted in the Infocentros strategy. Infocentros was a pilot project for the initiation of a global strategy oriented toward the development of a national connectivity platform which allowed the immediate achievement of the objectives of the democratisation of new ICTs.
With a defined strategy for the programme, the first evaluation of the project was carried out in order to establish a telecentre model. The evaluation concluded that the most relevant model for a telecentre was the old Venezuelan public cabin with some additional concepts coming from other countries’ experiences. As a result, a unique telecentre model for Venezuela was defined in accordance with its social characteristics. (NCIT, 2001)
As a result of this evaluation of the programme, an Infocentro was defined as “spaces with the minimum infrastructure, equipment and ICTs – in terms of its parallel and distributed computational architecture – which will be configured according to the context in which it is going to be installed physically…”(NCIT, 2001). This meant that each Infocentro was going to foster the creation of the particular services required by each particular community.
In philosophical and social terms, an Infocentro was “a space open to the general public, for providing services to institutions and individuals and which offers many opportunities for handling information through new technology…” ”an Infocentro should encourage training in ICTs, the opening of spaces and alternative practices which allow the linking of the community and with this, it can contribute to the improvement of its conditions”. (NCIT, 2001)
In sum, the implementation of the Infocentros programme was conducted in seven basic stages:
Promotion campaign and diffusion to authorities and other interested parties
Reception of proposals, inspection and validation of proposed spaces
Adaptation of infrastructure in approved spaces
Furnish by awarding contracts to bidding companies
Installation of Infocentros
Training of personnel
Connection and start up
During the initial period of Infocentros, no formal evaluations were carried out. The government and polling companies provided only limited statistics, showing the evolution of Infocentros usage and Internet penetration in Venezuela. (CAVECOM, 2003) Notably, these studies did not measure the impact of Infocentros on social and economic development. Officials interviewed stated “we were in an intense period of installations, we needed to install 240 Infocentros and we did not care for evaluations at that moment”…”The project had to become known and after that we cared about assessing it”.
Analysis The Genesis
Following what is common in initial phases of telecentre projects, a pre-operation planning or “start-up” evaluation was carried out during this period. A proposal or idea for the project was evaluated after the presidential resolution of “free Internet access for the people”. The programme had defined objectives which had to be implemented through a telecentre model.
There is no evidence outlining those involved in this evaluation. Official documents however state that principally directors of MST and NCIT participated, but responsibilities were not indicated. Indirectly, Venezuela’s president initiated the evaluation with his declaration of free internet services, which entirely altered the direction of earlier policies and telecentres implemented thus far. Apart from officials who read and signed the official reports, citizens were informed of the results of the evaluation through a presidential press conference.
The purpose of the evaluation included the definition of a telecentre model which was adapted to the presidential declaration and as a consequence, to the recently defined telecentre strategy and objectives. Moreover, with the provision of free services, the financial sustainability of the project came under threat, thus a financially sustainable model was needed which would support this new schema. A new implementation strategy, which allowed short-term implementation of pilots was also necessary as a product of these changes. In terms of construction and technical specifications, the intended model had to be relevant to Venezuela’s situation. One could speculate that this evaluation did not precede a decision making process, on the contrary, it can be argued that the decision to include a specific model of telecentre with free services had already been taken by the president, and even if this model was not the most appropriate, it became the schema was to be implemented. Consequently, it can be also argued that this position had implications for the evaluation since its outcomes had to reflect the president’s plans.
MST and NCIT staff evaluated the programme in terms of technical feasibility, technical infrastructure and equipment. Furthermore, the managerial feasibility of the project was also evaluated in order to implement the project in the short term. Designing the project required the creation of a model which in addition to technical aspects, also included theoretical-conceptual and methodological concerns in order to develop Infocentros according to appropriation, quality, participation and economic sustainability criteria. Although there are no specific details about who for, how long, when, and how the evaluation was carried out, it was noted that the methodology included research which assessed evaluations and telecentre experiences in other countries, with the purpose of establishing the most relevant aspects in terms of the construction of a model telecentre. The official documents and people interviewed did not provide details about the criteria used to choose countries and experiences to be examined nor whether the process of evaluation included additional activities. In terms of the project’s development life cycle, it is clear that this evaluation can be considered a pre-operations planning or “start-up” evaluation, which is commonly carried out in the first stages of a project.
The fact that, the methodology included the assessment of other countries’ experiences indicates that at least there was the intention of learning from other projects’ successes and failures. Despite this, there is no information to indicate what information from these projects was analysed, and if these experiences influenced the evaluation results. These latter, expressed as the new telecentre model and the implementation strategy, are available in official documents and also on the websites of Infocentros and NCIT. Data collected in this study did not provide evidence to determine who decided: where, what, how and when were the results going to be available.
Evaluation UNESCO 2002
After a year of operation, the authorities recognised that Infocentros had, in some way, to be linked to the community and that this depended on particular cultural contexts and the socio-economic needs of communities. Furthermore, project co-ordinators warned that evaluation of the management of Infocentros was needed as were comprehensive sustainability studies and the implementation of mechanisms for linking communities to Infocentros. (Barrera, 2002) A new Infocentros evaluation – the first formal one conducted was then carried out.
Conclusions stated the that Infocentros project was found to be very fragile in its concepts and operation, and a sustainability framework was proposed in order to correct and improve these aspects. Similarly, it was suggested that a higher degree of participation and linking with communities was needed, and that some positive impact was being achieved in terms of improvement of ICT literacy. Aspects of social and financial sustainability were also mentioned by the report, giving special emphasis to the definition of short and long term objectives and adaptability to changing social, economic and cultural circumstances of communities. Additionally, a business plan was proposed in order to define more clearly how sustainability of the project was to be achieved. The conclusions of the UNESCO evaluation are presented in Appendix 5.
Analysis Evaluation UNESCO
This first formal evaluation intended to assess the performance of the infrastructure installed, services and staff, along with early impacts that the programme had or was likely to have. Aspects of financial sustainability were also considered as were the way in which these were related to the impact of the programme on communities.
In the different stages of the evaluation several groups of stakeholders were involved. In terms of initiators, the evaluation began because NCIT directors and Infocentros programme co-ordinators were worried about management, financial sustainability and linking to community issues. A proposal for a new sustainability scheme (which considered the free nature of the services offered by Infocentros), a business plan, and recommendations for the improvement of the management and appropriation of Infocentros were necessary. Moreover, in terms of schedule, the programme was nearly two years old and a formal evaluation was needed. Additionally, a proposal of mechanisms for linking Infocentros to communities was required. The planning division of the Information Society of UNESCO was hired by NCIT for the designing and carrying out of the study, NCIT staff were also involved in designing tasks. UNESCO, on the other hand, sub-contracted an academic, who had managed the telecentres project in Panama in 2000.
It was found that the study was omitted or not known of by current officials in NCTI. Even staff in charge of the planning and designing current evaluation practices did not know of this evaluation. During interviews and informal sessions the author informed current officials about the UNESCO evaluation. It may therefore be argued that only the NCIT directors who ordered the study had access to its results. Moreover, current officials –mainly directors - were not interested in commenting on UNESCO conclusions. On the other hand, staff working on evaluation 2004 of Infocentros showed more interest and asked to have access to the document. One could argue that these omissions and ignorance exist because previous NCIT directors ordered this study. Similarly, the report’s conclusions were critical, especially in terms of the sustainability of the programme – a problem at that time - and perhaps this aspect was not a ‘convenient’ issue to release to the public or to admit to.
The function of the evaluation followed Hudson’s (1999) criteria, which distinguishes between formative and summative functions of evaluation. Formative evaluation of Infocentros was carried out with a focus on how well the project was working, changes and improvements that should be made and lessons learned. Infocentros’ summative evaluation focused mainly on the degree of Infocentros goal achievement – impacts on community in terms of development or a difference made by the project. (Hudson, 1999)
The evaluation methodology included the use of questionnaires to collect the data needed. These questionnaires were based on relevant telecentre studies such as Proenza’s sustainability study (Proenza, 2001) and Whyte’s assessment methodology (Whyte, 2000). They were intended to identify needs, levels of satisfaction and, to a minor degree, the impact of Infocentros on communities. Time available and limited mobility were mentioned as the main limitations of the study because the time frame did not allow more Infocentros in rural areas to be visited. This was considered important because in some cases Infocentros were managed by entities which apparently had more roots in the communities and these could have enriched the recommendations provided by the study. Additional data collected included: NCIT reports, web site examinations, interviews, questionnaires and discussion sessions.
Data collection from questionnaires was carried out with the Infocentros’ operators, the process lasting one week. A psychologist was in charge of the development of the questionnaire that was going to be applied in UNESCO’s evaluation and periodically after. Appendix 6 shows the utilised questionnaire. Interviews and observations were carried out by UNESCO staff. There is no evidence however, which provides information on how long these sessions and observations lasted. With observations and interviews of operators it was intended to characterise Infocentros’ context, space distribution and technical infrastructure. Similarly, the quality of service offered by the operators and their enthusiasm were assessed and also whether they followed procedures established in the operation manuals. Operators were interviewed about peak hours in the centre, average number of users per day and also busiest day(s) of the week. Connectivity quality – speed –and performance and the quality of printing and scanning services were assessed.
The study was done considering two pilots Infocentros, which were selected by NCIT, in order to fulfil the purposes of the evaluation. Additionally, UNESCO staff visited other Infocentros and public internet access centres, in both rural and urban areas by using comparison with the pilots as a benchmark. When assessing other Infocentros and centres not included in the pilots, applied questionnaires were modified, for example when examining the Infocentro in a women’s penitentiary centre, income questions were eliminated in order to avoid any discomfort among respondents.
Results were available in a report, although presumably they were presented to NCIT directors during a formal presentation session. Contrary to earlier evaluation results, the report is not available on any official website related to Infocentros. For example, the Infocentros website has a webpage that shows programme evaluation reports but the UNESCO evaluation is not available there. One may argue that results presented in this study had negative implications for NCIT officials because they were not in accordance with expectations.
This situation may also support the argument that the evaluation was not linked to organisational learning. Just one document referencing the UNESCO evaluation was found in official NCIT papers. This describes a new ICT and development programme called “Gerencia Social del Conocimiento a traves de Infocentros” – Social Management of Knowledge through Infocentros – and it mentions some recommendations concerning the mechanisms for linking Infocentros to the community that were proposed by UNESCO.
Interesting recommendations and results were provided in this evaluation but many of them have not been considered or implemented thus far. For example, the study recommended the creation of an Association of Infocentros to establish a body that allowed the sharing of experiences between the centres, and also the recommended sustainability framework for the project, which was given as one of the purposes of the evaluation. NCIT argued that strong political rivalry was the reason for infrequent collaboration between Infocentros, because some centres are located in areas where local authorities belonged to political parties which opposed the current administration.
Evaluation IDC 2003
After changes to NCIT personnel, a second formal evaluation of Infocentros was conducted in 2003. A mixed team including staff from the International Data Corporation (IDC) of Venezuela, NCIT and consultants was constituted. IDC gave its opinion about the sustainability of the programme and argued its effectiveness was acceptable. Nonetheless, it identified the need to revise and analyse management because there was a risk to medium and long-term sustainability. Despite these viewpoints, it was argued that the potential sustainability problems were reduced by the relationships already established between Infocentros and the communities, which had been assessed as more than acceptable (IDC, 2003).
Results presented in the study began with information about the context of Infocentros included in the sample, indicating also the economic activities and behaviours generated in the communities. Results also offered information about Infocentros’ services and new activities produced through the implementation of Infocentros, and also people’s opinions about scarce resources in the centres, which affected the performance of services. The study offered statistics on Infocentros’ user profiles and attitudes. The most important part of the study – Evaluation of social Impact of Infocentros – analysed the data collected according to three defined areas.
To conclude, the study offered recommendations for increasing the impact of Infocentros and positioning them as the most important social programme implemented by this administration. Among the general recommendations presented was periodic monitoring of the programme through the annual updating of data offered by the study. The identification, measurement, comparison of this data and furthering studies of indicators of progress and delays were strategies suggested to correct problems, financially support the programme, and extend and maintain the programme over time. Specific recommendations related to evaluation of Infocentros are included in Appendix 7.
Analysis Evaluation IDC
The second evaluation of Infocentros aimed at identifying impacts that the programme had or was likely to have in social, educational, technological and economic areas. This resulted, at least in definition, in a consideration of a ‘wider’ information system, which involved software, hardware, people, context and interactions. Aspects of sustainability were also assessed in a broader context because sustainability was considered not only in financial but social terms.
With the purpose of evaluating social, educational, technological and economic impacts of Infocentros, NCIT hired the International Data Corporation (IDC) of Venezuela. The main purpose of the study was the provision of information that allowed the identification and maintenance of achievements attained by Infocentros so far (IDC, 2003). The evaluation had the political role of giving credit to the presidential administration because at that time it was being ‘evaluated’ by citizens, and implemented social programmes required “positive” evaluations. It can be argued then, that evaluation was indirectly initiated by the president and explicitly by NCIT directors. Negative outcomes were not permitted and team, methodology, and evaluation results, were designed ad-hoc to achieve that purpose. The results of evaluation were to be read primarily by officials and after “they were going to be known by people in order to confirm the outcome of the [Chávez] revolution”. Additional objectives of the study included the formulation of recommendations for maintaining and incrementing achieved impacts, and the definition of evaluation needs of the programme.
The process lasted four months and the evaluation methodology included four stages: Conceptualisation, Elaboration of Indicators, Collection of data and Tabulation and Interpretation of results.
Indicators defined for the study considered statistical variables, context characterisation, and impacts on context and users. (See Appendix 8) Once indicators were defined, five different questionnaires were designed: Community and Context, Infocentros, Co-ordinator / Operator, Users, and Person of context. (See Appendix 10) For the assessment of impacts of Infocentros, indicators were classified according to a model for evaluating and measuring the achievements of Infocentros, which was designed by IDC with the collaboration of NCIT staff, and defined three areas of study in order to fulfil the evaluation objectives.
Management: measured the quality and effects of management of Infocentros, especially those related to sustainability.
Relations and Co-operation: measured Infocentro ability to co-operate in context, responding to needs, and generating relations between different elements of the community.
Position: measured the perception of Infocentro in terms of the community served, and how users, operators and co-ordinators, and people of context are affected according to the social benefits perceived by them.
Each area was related to questions in the instruments of data collection for all samples, and the three areas together constituted a unique block of indicators of impact.
The levels of impact were qualified in the following areas:
The changes produced by the incorporation of new elements and habits in the community.
Levels of usage of Infocentros
Levels of Management
Levels of capacity from lack of services
Levels of users’ attention
Co-operation relationships between Infocentros, NCIT and local technical service and maintenance companies.
Samples included in the study were defined according to the number of Infocentros and staff, people in context, and user population. For the collection of data, two phases were needed, the first collected data from five Infocentros pilots – designated by NCIT - in order to validate and perfect the instruments for data collection. The second consisted of the application of adjusted instruments to the rest of the sample according to observations from the first phase. For data collection, questionnaires were supplied around the country by a team of trained pollsters.
Similarly, the study included an evaluation of the effectiveness of Infocentros that resulted from the combining of the Management and the Position areas through the indicators ‘service offer’ and ‘number of visits’. The general impact of the programme was then described from two perspectives, a quantitative one, defined by three of the areas mentioned, and a qualitative one, defined by the sectors that had an effect on educational, social, economic, technological and political impacts.
This study has been extensively used by NCIT directors as a tool for the promotion of the programme in different presentations i.e. UNDP’s meetings and national and international summits. After publication in June 2003, other social programmes were created as a product of many of the recommendations, for example “Social Management of Knowledge in Infocentros” programme.
During formal and informal meetings with NCIT staff and Directors, a wide knowledge of this study was expressed, and in many cases considered the only evaluation of Infocentros carried out so far. Among the positive aspects, evaluation staff mentioned that it provided “a pile of useful information” to NCIT, it offered an insight into the programme that was new at that time, and it outlined the situation of Infocentros. Similarly, evaluation was considered positive because it fulfilled its objectives and purposes, and it was useful in showing results to the media and entities related to the programme.
Negative aspects of the evaluation enumerated that results, effects and impacts of the programme had not been correctly established, impact results were abstract, and the evaluation did not show all impacts of Infocentros. It was also mentioned by NCIT staff that UNESCO’s evaluation was not considered as an antecedent, or was not known of by the staff participating in the study. Similarly, some staff currently working on future evaluations of Infocentros mentioned that the methodology did not consider any of the frameworks and studies in the telecentre evaluation literature.
Future Evaluation 2004
After some discussion sessions and interviews, it was established that NCIT was planning a new evaluation of the project corresponding to the year 2004. Although this evaluation is only a proposal, it presents interesting issues which can contribute to a better understanding of the evaluation of Infocentros. It is also important to note that this evaluation strategy was presented to NCIT directors in May 2004 but at the time of this study, the Project Management division, in charge of evaluation, had not received any answer to the proposed evaluation plan.
Recognising the weaknesses of past evaluations in terms of indicators, it is intended that a new process which considers telecentres, their context, and their interactions can correctly establish the various impacts of Infocentros.
Since Infocentros does not have a specific evaluation schedule, an objective of the 2004 evaluation is the establishment of periodic evaluation and monitoring processes of Infocentros. It has been stated that the learning of lessons, obtained from these evaluations will be the foundation for evaluation strategies of other ICT and development programmes at NCIT. The hope is the definition and implementation of policies and strategies for the evaluation and monitoring of all projects managed by the NCIT’s Project Management division.
While the evaluation team is enthusiastic about this study, NCIT directors do not express interest in the evaluation planned nor Infocentro’s evaluations in general. According to the NCIT’s President “for an accurate evaluation is needed to know what you are evaluating, and at this moment objectives and philosophy of Infocentros is changing. We care more about the evolution of the programme, if it is evolving is because it is responding to the changing needs of communities”. Evaluation staff expressed their opinions about the directors’ attitudes and questioned whether they understood the objectives of the proposal or whether the presentation sessions and materials were not clear enough. On the other hand, other directors, recognising the weaknesses of earlier evaluations, are aware of the need for evaluation and show more enthusiasm for this new evaluation project. It can be argued that for senior Directors this evaluation does not have a ‘political’ objective and usefulness, “enough” positive results of the Programme were obtained in the IDC evaluation.
Recognising the lack of support from directors for the new evaluation and the large investment usually involved in the evaluation of large-scale projects, evaluation staff worry about the future of the proposal and have started to think that the plan is too ambitious.
Initiated by NCIT’s Project Management Division, the evaluation team of the division has been working on a plan since 2003. The evaluation team is still at the negotiation stage. However, the plan includes a research team to be formed with a sociologist, two statisticians, and an economist included. The incorporation of PhD students and NCIT staff is also planned for the implementation of this evaluation strategy, and the development of tools and pilots. Apart from NCIT staff, there is thus far no explicitly stipulated audience or intended recipient of this process of evaluation.
The evaluation strategy for 2004 is the product of a compilation/adaptation of frameworks in the telecentre evaluation literature, which have been created by international development organisations and academics. Among these frameworks consulted are, for example, the World Bank’s impact evaluation framework, IDRC’s framework and Olistica’s impact assessment tools.
The initial plan defined by the evaluation team will be applied first to a sample of pilots, and afterwards, to all Infocentros. Once the proposal is approved by directors, a stage of information gathering will begin – on small and medium scales- followed by implementation of defined monitoring and assessment strategies across the national Infocentros project.
Despite there being no complete and detailed information about this new evaluation exercise, the suggested categories for the analysis in the 2004 evaluation were mentioned by the NCIT staff and these include: results, effects and impacts of Infocentros, and analysis of the national context and the internal community factors. A complete specification of these categories is included in Appendix 9. Moreover, there is not yet information available to establish what, how, where and when evaluation results will be available.