Understanding Telecentre evaluation frameworks through the Venezuelan Infocentros programme abstract

Download 319.92 Kb.
Date conversion15.05.2016
Size319.92 Kb.
1   2   3   4   5   6   7   8


1 - Internet Penetration in Venezuela

Source: CAVECOM, Study of usage and Internet penetration in Venezuela, 2003

2 - Evolution of Infocentros usage 2000-2003

Source: CAVECOM, Study of usage and Internet penetration in Venezuela, 2003

3 - Guiding Principles for telecentre evaluation

Guiding Principles of Telecentre Evaluation (Gómez and Hunt, 1999):

The following synopsis is based on a brainstorming session involving all meeting participants, held at the conclusion of the gathering. It represents the group’s collective determination of basic principles to guide our work in telecentre evaluation. Ideas were generated in response to this query:

What basic, guiding principles will ensure that telecentre evaluation:

(a) is useful;

(b) is financially responsible;

(c) builds local capacity; and

(d) enables shared learning.
Guiding Principles:

People-focused, participatory evaluation design, sensitive to gender and the local context

(Focus on people, Participatory, Horizontal participation, Users and non-users, Contextually sensitive, Involve stakeholders early in the process, Build up from local cultural paradigms, Identify target groups, Enhance women’s participation)

Design the evaluation process to ensure quality and sustainability

(Dynamic costing arrangements, Viable business, Constant quality of service with diminishing external support, Ensure that cost vs. resources are realistic)

E=L2: (Evaluation = listen and learn)

(Use available expertise, Listen!)

Both successes and failures will be analysed through the evaluation process

(Learn from failure and innovation)

The evaluation result will be disaggregated and used to conduct training workshops

(Oriented towards strengthening and empowering practitioners and intermediaries, Builds capacity tools).

Clarify purpose

(Make sure all stakeholders share a vision of what questions the evaluation needs to answer

and how this information will be used, Focus on critical issues)

Use adaptable and replicable methods

(Replicable research design, Easy to do, Evolving Design for local context and project, Variety of methods, Adaptable, responsive frameworks, Transcend project boundaries)

Ensure the process is open and transparent

(Make it public!)

Design an Effective Mechanism to Communicate Results

(Craft evaluation for intended audiences, Strategic partner impact, Report findings in appropriate multiple formats, Have appropriate learning incentives, Proactive feedback, Participation in regional international telecentre evaluation)

Include evaluation from the start and in all stages

(Build evaluation into project design, Consider accessibility and equity considerations, Start from the objectives of the evaluation)

4 - Infocentros Evaluation Questionnaires

I. General questions (NCIT Directors, Project Manager)

  • Project evolution and factors triggering these changes in the Infocentros Programme

  • Differences between your administration in NCIT and the earlier ones

  • Positives and negative aspects of Infocentros

  • Why is the Infocentros Programme an independent division inside NCIT?

  • How is the Programme related to other current and future Science and technology programmes and /other projects?

  • In your opinion, what are the most interesting/successful cases or projects using ICTs for development in Venezuela?

II. Infocentros Evaluations (NCIT directors, project manager, current evaluation team, conversations with ICT-and-Dev people)

  • How many evaluations of the Programme have been carried out so far?

  • How often is the Programme evaluated or monitored? Has the evaluation plan or agenda been followed?

  • Differences between the evaluations carried out so far? Team? Methodology? Time frame? Stakeholders? Sampling criteria? Etc

  • Positive and negative aspects of the different evaluation exercises

  • Models and Methodologies taken into account when designing evaluations

  • Indicators used in evaluations

  • Have other evaluations of the programme being carried out by outsiders?

III. Infocentros General issues (NCIT directors, project manager, current evaluation team, conversations with ICT-and-Dev people)

  • Sustainability problems vs. Evaluations exercises – are there a relationship between the two?

  • Apart from Central government and NCIT’s guidelines, are there external bodies, organisation, NGO, particular company, academic groups or persons whose opinions have been considered when evaluating Infocentros?

  • Actions after evaluation? Monitoring of these actions or/and after-action-reviews? Can you consider evaluations as learning exercises?

  • Are the contents evaluated? Are contents created according to communities’ need for information?

  • Has any association or network grouping Infocentros for collaboration and support, or sharing of experiences been created? If negative why?

  • Is the project sustainable? What is your concept of sustainability? Financial? Social? Cultural? Etc

  • Is there a Business Plan for the project?

  • How is the performance controlled? SW tools? Other mechanisms?

  • Usage of Infocentros statistics? Collection of this data, methodology, team, etc?

  • How is the programme in terms of flexibility or adaptability to community needs? Improvisation exists? Reactions to new situations?

  • Are there plans for the development of new contents or the offering of new training courses, SW, HW, etc?

5 – Conclusions of UNESCO evaluation

Conclusions about the general performance of Infocentros included:

  • They (NCIT) started off eagerly and began to ‘drift’ without considering all aspects and details involved in large-scale national projects. Despite qualitative measurements not being carried out, the project was expanded to the country with the installation of more Infocentros.

  • Diffusion and promotion campaigns did not include direct contact with communities and leaders.

  • Financial sustainability was considered while the Public Cabins project existed, however after the creation of Infocentros with free services, sustainability studies were not included in order to assess if the project was self-sustainable under this new scheme.

  • Antecedent of evaluations - during the public cabins - collected only information on user profiles and usage. Records of economic activity from Infocentros did not exist, nor evaluation of impacts, acceptance and management of Infocentros. The Infocentros plan did not include periodic or continue evaluation and UNESCO’s evaluation was to evaluate for first time these aspects alongside the establishment of a model for permanent evaluations.

  • A psychologist was in charge of the development of the measurement tool that was going to be applied periodically and in UNESCO’s evaluation.

  • Infocentro’s operators were not properly trained. They could not solve basic technical problems in the hardware area. This caused a delay in services because Infocentro depended on the availability of regional technical staff. Since on many occasions even the regional staff were unable to resolve problems NCIT staff had to visit the area and find solutions. The FAQ section in the Infocentros web site was going to be expanded to achieve additional technical resources for staff.

  • Daily activity statistics were not registered by Infocentros and financial performance of centres were not known by NCIT.

Recommended aspects of social and financial sustainability were based on different experiences collected from international development organisations, such as UNESCO, the World Bank, IDRC, Inter-American Development Bank, and others. As part of the proposed sustainability framework, the monitoring of financial and social impact aspects of Infocentros was suggested.

In proposing mechanisms for linking Infocentros to communities, several issues were raised, the most important being: the creation of a community committee in each Infocentro, the fostering of a continuous relationship between Infocentro and the community, the assessment of community demands, education of community including ICTs literacy and finally, the creation of an Association of Infocentros.

6- UNESCO questionnaire

Questions included in the questionnaires intended to collect information concerning:

  • Socio-economic information of users: Occupation, sex, income, age, etc

  • Availability of computer and internet access at home

  • How people knew about the Infocentro – mechanism of promulgation and spread

  • Access and journey length to Infocentro

  • Transport used to reach to Infocentro

  • Frequency of visits per user

  • Why and how people use the Infocentro – purpose of use and Information consulted during the visit

  • Impact of Infocentro was measured with some questions concerning level of satisfaction on: environment, speed of access, software available, technical expertise of operators, staff availability, Infocentro’s timetable and training facilities available for users.

  • Impact of Infocentros was also measured in terms of improvement of computer skills and internet usage skills of users. It was a subjective indicator but at least showed a level of impact of the Infocentro.

  • Additional services and training courses desired by users, and the acceptance of charges for these additional services and courses.

7 - IDC Recommendations

Specific recommendations related to evaluation of Infocentros included:

  • Designing and carrying out a monitoring plan of Infocentros following the methodology of evaluation defined by the Inter-American Development Bank and also incorporating experiences that can be applied to Venezuelan.

  • Definition of effectiveness and efficacy indicators.

  • Improvement of NCIT databases through better instruments of data collection.

  • Design and implementation of an information system of indicators that works under data warehouse or data mining schemes.

  • Standardisation of evaluation mechanisms including regional definitions, human resources, training plans and better practice.

8 – Indicators used in evaluation IDC





  • users by age

  • users by education

  • users by occupation

  • users by Infocentro

  • total number of visits per Infocentro

  • number of courses

  • number of scheduled courses per month



  • population

  • number of schools

  • number of universities

  • number of hospitals

  • number of clinics

  • number of surgeries

  • number of fire stations

  • number of churches

  • school fees

  • economic activity

  • number of cyber cafes

  • number of telephone lines

  • number of PCs

  • number of cable TV subscribers

  • number of Internet subscribers

  • primary sector activities

  • secondary sector activities

  • tertiary sector activities

  • number of governmental organisations



  • variation of users

  • satisfaction

  • Infocentro frequency of use

  • Variation of internet use habits

  • Number of people trained by Infocentro

  • Percentage of use of additional services

  • Percentage of Internet use

  • Measurement of service appropriation

  • New elements in the community


  • users with mobile phones

  • users by income

  • employed users

  • unemployed users

  • users by internet interest

  • users willing to pay for Infocentro services

  • users with home internet access before Infocentro

  • users with home internet access after Infocentro

  • reasons for using of Infocentro

9 - Suggested categories for the analysis in the 2004 evaluation

1. Results of Infocentros: How the communities use new technology provided by Infocentros and the direct relationship with the equipment and capacities that the community can develop. Equipment, newly created skills and different aspects of usage of Infocentros – how it is used, what it is used for, frequency of usage – are some of the aspects that will be considered in this area.

2. Effects of Infocentros: How Infocentros modify the processes of communication, information and generation of services through the incorporation of technology in communities.

  1. Impacts of Infocentros: when measuring impacts, the new evaluation strategy proposes the assessment of changes in the dynamics of communities in processes of information, communication and services. Equitable access, usage with purpose, and social appropriation are included as aspects to be measured.

  1. Analysis of National context: Because the national context influences the organisational behaviour of each community, the evaluation will include the assessment of:

    1. Structural factors: Human development Index, Intern development programme

    2. Sectorial factors: degree of opening

    3. Other public policies

    4. Provision of services, direct and indirect ICTs

    5. Environment and national attitude

    6. Other factor which can inhibit or encourage the use of ICTs

  1. Internal Community factors: environment, attitudes and internal knowledge of the communities influence how people confront the changes that ICTs bring to their daily lives. Change agent, needs of technology, technology interest, knowledge of technology potential, knowledge of changes produced by the adoption of ICTs and computational status of the community, are factors included in this evaluation section.

10 - IDC questionnaires

COMMUNITY and CONTEXT Questionnaire
Here is presented a summary of the content in the different sections of IDC questionnaires, the original versions in Spanish follow.
Section 1: Community socio-economic data (population, educational institutions, health services, and telecommunication services)

Section 2: Community Economic Information (main local economic activity)

Section 3: New economic activities produced by Infocentros

Section 4: Appropriation of community (open-ended question)
PEOPLE in context Questionnaire
Section 1: Personal Information (i.e. age, gender, education, and occupation)

Section 2: Awareness of Infocentro (time using it, home computer /internet access)

Section 3: Needed Skills to use the Infocentro

Section 4: Performance Infocentro

Section 5: Lacks of Infocentro

Section 6: Perception of Infocentro’s commitment with community

Section 7: New elements introduced by Infocentro
INFOCENTRO Questionnaire
Section 1: Address and Date of Installation

Section 2: Performance (Daily visits, new users per day, target population)

Section 3: Operators, Host (number of operators/hosts, services, additional services)

Section 4: Services (Popular services, costs of electricity, water, etc., demand of new services)

Section 5: Training (Courses, Frequency, monthly income for courses)

Section 6: Opening times

Section 7: General Comments
OPERATOR Questionnaire

Section 1: Personal Information (i.e. age, gender)

Section 2: Education (languages, education, acquired skills while working in Infocentro)

Section 3: Services (NCIT support, technical services)

Section 4: Infocentro Services

Section 5: Telecommunication context

Section 6: New elements in the community and context caused by Infocentro

Section 7: Training and Courses (Popular ones, demand of new services and courses)

Section 8: Professional benefits of working there

Section 9: Weaknesses of Infocentro – lack of what?

Section 10: is Infocentro committed to Community?
Section 1: Personal Information (i.e. age, gender, and occupation)

Section 2: Education, Income

Section 3: Awareness of Infocentro (frequency of use, time using Infocentro, How knew)

Section 4: Services (Use, frequency, and demand of new services)

Section 5: Internet usage (information, entertainment, web sites visited, priorities)

Section 6: Training and Courses (satisfaction, use)

Section 7: Was your First contact with the Internet in Infocentro?

Section 8: ICT literacy (SW, HW, internet)

Section 9: Knowledge and experience acquired after Infocentro

Section 10: Service satisfaction (operator services, Infocentro services, lack of what?)

Section 11: Willing to pay for services

Section 12: Impact of Infocentro (new elements in context or community)

Section 13: User’s telecommunication context (mobile, telephone, Computer or internet access at home)

Section 14: Is Infocentro committed to Community?
1   2   3   4   5   6   7   8

The database is protected by copyright ©essaydocs.org 2016
send message

    Main page