The approach to evaluate CTEC, its grantees, and their affect on rural health included collecting and analyzing qualitative and quantitative data, which took into account cultural, social, economic, and political contexts for data evaluation, in order to provide an accurate assessment of processes and outputs. The evaluation process focused on developing data collection tools. The tools track 1) changes in healthcare services access; 2) changes in grantees' technological capacities; 3) grantees' involvement in networks; 4) organizational changes brought about by changes in technological capacities; 5) ability to sustain telemedicine programs and factors that promote sustainability; 6) stakeholders' satisfaction with the use of technology in delivering services; and 7) gaps in health-care services and barriers to using telemedicine.

The Endowments program was designed to mitigate the loss of medical services in rural areas due to movement of medical professionals away from rural toward urban areas and the lack of incentives for professionals to move to rural areas to practice. The evaluation has helped determine how and to what extent The Endowment's investment in the development of eHealth networks has improved access to and utilization of health services in California. In addition, the evaluation of existing eHealth networks examined how well the program supported network development within the larger framework of existing partnerships and extended eHealth to rural and underserved communities by linking with the state's rural service providers. The evaluation focused on the following five areas: Network Development, Technical Assistance Survey, Education and Training Services, Policy Development and Systems Change, and Community Assessments.

The approach to evaluate CTEC, its grantees, and their affect on rural health included collecting and analyzing qualitative and quantitative data, which takes into account cultural, social, economic, and political contexts for data evaluation, in order to provide an accurate assessment of processes and outputs. The evaluation process focused on developing data collection tools. The tools track:

• Changes in healthcare services access
• Changes in grantees' technological capacities
• Grantees' involvement in networks
• Organizational changes brought about by changes in technological capacities
• Ability to sustain telemedicine programs and factors that promote sustainability
• Stakeholders' satisfaction with the use of technology in delivering services
• Gaps in health-care services and barriers to using telemedicine

The evaluation goals included assessing both: 1) how and to what extent services, capacities and resources were put into place where eHealth access did not previously exist; and 2) how currently existing telemedicine efforts were further developed. The review process included an assessment of the development of CTEC policies and practices, and how current strategies regarding the development of eHealth networks have been implemented. For example, this process included an assessment of the extent to which the current development strategy has resulted in the expansion of the number of rural and eHealth sites or services. This approach has helped facilitate a comprehensive assessment of patterns for success or failure in the development issues faced by each program component. WRMA developed the evaluation tools to match each program's goals to outcomes, including methods of evaluating success of goals and objectives, changes to be made to further goals, and common experiences/barriers to implementing best practices. Designing targeted evaluation tools included not only the framework for describing both processes and outcomes, but also identify opportunities for developing policy changes which may expand eHealth services to rural and underserved communities as well as increasing access to care statewide.

Using data from the evaluation of each grantee's program, WRMA designed an action plan for facilitating collaboration between CTEC, their grantees, and public and private agencies at both the state and local level using information and communication technology. WRMA served as a key resource for program integration and information management to increase access to and awareness of the need for ongoing collaboration in eHealth expansion and implementation. In addition, assessing current online information system interoperability capabilities, training, and awareness-building resources provided a basis for strategies to expand eHealth delivery.

Additional goals of this evaluation included developing specific evaluation criteria for underserved persons with disabilities, increasing access to services, and specific measurement protocols to ensure cultural competence, with special attention paid to difficult-to-serve populations. Data collection efforts include:

• Data which clarify the outcomes and impacts of the program, including the extent to which these outcomes expand beyond the individual project to the policy arena as a whole;
• Data pertaining to the contextual factors that influence the success or hindrance of the program, including policy, economic, socio-cultural and environmental factors; and
• Data related to issues and/or themes which emerge across grantees and elements of the program, including similar trends that emerge from grouping grants by content area, geographical region, or institutional context.

TThe indicators and findings from these analyses are the basis for policy and practice recommendations to develop, sustain, and expand eHealth service delivery networks. By collecting and evaluating both quantitative and qualitative data and defining common indicators and outcomes, this evaluation revealed how best to overcome the barriers to implementing telemedicine programs and the policy-based best practices in designing and implementing successful Telemedicine services and partnerships.

TThis effort involved not only assessing the effectiveness of the current network with regard to delivery of telemedicine on a state-wide basis, but also developing and helping to implement strategies for expanding eHealth services to rural and underserved populations. WRMA research staff evaluated the existing network in terms of organization of service delivery networks, service improvement strategies, client perception of program availability and accessibility, accomplishment of operational targets, and identification of “best practices” among a number of similar program sites. The project team designed and helped to facilitate the development of expansion strategies to enhance the accessibility and effectiveness of collaborative, culturally competent telemedicine service delivery throughout the network, enabling both increased access to healthcare as well as sustainable collaborative structures and continuing development beyond the completion of the project.