| 
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • Work with all your cloud files (Drive, Dropbox, and Slack and Gmail attachments) and documents (Google Docs, Sheets, and Notion) in one place. Try Dokkio (from the makers of PBworks) for free. Now available on the web, Mac, Windows, and as a Chrome extension!

View
 

Evaluation Panel

Page history last edited by Angela Siefer 10 years, 3 months ago

Session Description: Evaluation consultants for four BTOP-funded sustainable broadband adoption and public computing center projects will describe their work, and lead an open-microphone Q&A on project evaluation strategies and goals.  Don't forget to add your notes below!

 

Location: Ballroom

 

Time: Monday 10:15-11:15

 

Moderator: Francine Jefferson, Department of Commerce

Panelists: Karen Mossberger, Univ of IL at Chicago. Nicol Turner-Lee, Joint Center for Political Studies. Samantha Becker, Univ of Washington. Blanca Gordo, Berkeley

 

Notes:

Categories of evaluation
1. Personal outcomes
2. Interpersonal outcomes
3. Organizational outcomes
4. Community -level outcomes
-city-wide surveys
---barriers to broadband use, technology use barriers

 

Joint Center
Working with OneEconomy to connect public housing
Works with pew studies polling services
GOALS
- determine how well the program meets need
- evaluate how well program extends broadband use
- assess and document the shift in the culture of technology use
- comparative research design

 

Evaluation approach uses a comparative model > test cities vs. Control cities

CHALLENGES
1. Identifying a stable population
2. Consistent data
3. Anticipating lag time between participation and service connection
4. Disentangling test cities from control cities
5. Capturing racial,ethnic, and language diversity of constituents between test and control cities

 

SAMANTHA BECKER - U.S. IMPACT Study

Assist PCCs in doing their own evaluation ( to keep costs down)

Framework
Context analysis > situated logic model > outcome measurement > validation and reflective practice

Step 1
Policy issue mapping - what is the most pressing issue in the community
Step 2
Identify other stakeholders concerned with policy goals
Step 3
Develop a policy or stakeholder logic model
Inputs > activities > outcomes > impacts
Step 4
Bridge the logic model to show how the CTC supports larger policy goals

SideBar
What is a measurable indicator?
--cost effective
--specific, unique, unambiguous
--understandable
-- relevant
-- time-bound

Step 6
Measuring the outcomes

Step 8
Validate outcome measures

WA BTOP's reporting system relates to a community technology logic model

IMPACT TYPES
Comp skills enhancement
Employment skills and opportunities
Education enhancement
Access to information and services
Life skills and social inclusion

 

Francine Jefferson - "We should have a series of webinars about how you operationalize variables....or create a research/evaluation portal".

 

Integrate survey questions into the program. Be clear with program participants that the program funding is dependent upon knowing the program is working (ie. evaluation data).

 

There is a blog post at the following:  http://somewhereonthebridge.wordpress.com/

 

 

Presentation Slides:

Becker_CBAIS_06_27_2011.pptx evaluatingBB_CBAIS.pptx OE_BBOC BTOP Evaluation Presentation.pptx

Comments (20)

Tom.Miller@onecommunity.org said

at 11:29 am on Jun 27, 2011

Suggestion made to aggregate/share survey/research templates for SBA/PPC programs.

Connect Your Community said

at 11:36 am on Jun 27, 2011

Seems like polling is a common method for gathering data samples! but incentivizing is expensive - post youu ides for alternative low coast methods here!!!

Connect Your Community said

at 11:39 am on Jun 27, 2011

Contact the Urban Planning department at your local college or University to partner on evaluation!

Bartholomew Gonzalez said

at 11:39 am on Jun 27, 2011

surveymonkey.com

Angela Siefer said

at 11:42 am on Jun 27, 2011

Need location for sharing surveys.

Amy Luckey said

at 11:51 am on Jun 27, 2011

Absolutely. Rather than waiting for the potential federal evaluation portal - which would be great, can we at least get the 4 speakers' tools posted on this wiki now?

Tom.Miller@onecommunity.org said

at 11:58 am on Jun 27, 2011

thinking digitalliteracy.gov is the right place

Amy Luckey said

at 11:58 am on Jun 27, 2011

I think this is the evaluation study, including survey questions in appendix - that was referenced by Samatha Becker: http://www.cityofchicago.org/content/dam/city/depts/doit/supp_info/DEI/Digital_Excellence_Study_2009.pdf

Connect Your Community said

at 12:00 pm on Jun 27, 2011

YES !! Feel free to post any evaluation materials you might have to this wiki page! If we get enough - I will create a new page just for evaluation resources!

Angela Siefer said

at 12:02 pm on Jun 27, 2011

I emailed the evaluation panelists and asked them to post their evaluation materials, particularly survey questions, this this session page.

Amy Luckey said

at 12:03 pm on Jun 27, 2011

Public Policy Institute of California Californians and Information Technology survey funded by ZeroDivide and CETF was released last week. PPIC worked with Pew to develop some of the questions: http://www.ppic.org/main/publication.asp?i=985

Connect Your Community said

at 12:11 pm on Jun 27, 2011

There are many of us who will likely lode funding in a year or two and would like to continue impacting out communities

Can you comment on the notion of sustainability ( at the CTC level) in how evaluation strategies can be used to extend programming by helping to identify community and organizational partners?

Jemarius Moore said

at 12:21 pm on Jun 27, 2011

How can it be said that there are multiple factors used to evaluate the effect of the program when the only statistic seemingly used is how many new broadband connections in an area?

Tom.Miller@onecommunity.org said

at 12:25 pm on Jun 27, 2011

tell your stories at tomorrow's Strategic Community Impacts - K-12 workshop!

Amy Luckey said

at 12:25 pm on Jun 27, 2011

Would be great for BTOP grantees to work together with the expert evaluators to ID of the 4-5 survey questions that could be asked across most evaluations. Also, development of of a menu of survey questions that are likely relevant to most evaluations.

Samantha Becker said

at 1:38 pm on Jun 27, 2011

The U.S. IMPACT Public Library Study (Opportunity for All: How the American Public Benefits from Internet Access at U.S. Libraries) is available here: http://cis.washington.edu/usimpact/us-public-library-study.html
The survey instruments are included in the appendices.

Amy Luckey said

at 1:42 pm on Jun 27, 2011

Thank you!

mdavis@matrixhs.org said

at 3:02 pm on Jun 27, 2011

Having asked the initial session question regarding the need for a template for online surveying SBAs on their meaningful use experience, I was hoping that our main supporter (BTOP) would offer a set of general survey questions for use by the CYC agencies that provided data reporting for all levels of the program (CYC agency, One Community, BTOP, etc.). Special, localized survey questions for individual agencies or CYC could also be added at their appropriate levels. Online surveying should be the initial medium for post-SBA data capture. Initial intake forms currently in use should provide the necessary baseline data for the SBA. Any hope on getting this type of survey template real soon, as new program funding needs are rapidly approaching?

LeeAnn Shreve said

at 12:39 pm on Jun 28, 2011

I have uploaded several of our surveys we use for evaluating our program. We also use the results of this survey to determine what services are needed in particular communities. They are called: Baseline Survey - Future Generations, Business Survey - Future Generations and Online Survey - Future Generations.

If you have any questions, please email me at leeann@future.edu. I'd be happy to discuss our research and evaluation process with you!!

Colin Rhinesmith said

at 3:03 pm on Jul 8, 2011

I've been collecting resources related to BTOP evaluation. I've posted them on my blog at Community Media in Transition, here: http://cmediachange.net/2011/06/29/evaluating-btop-projects/

You don't have permission to comment on this page.