Session Description: Evaluation consultants for four BTOP-funded sustainable broadband adoption and public computing center projects will describe their work, and lead an open-microphone Q&A on project evaluation strategies and goals. Don't forget to add your notes below!
Location: Ballroom
Time: Monday 10:15-11:15
Moderator: Francine Jefferson, Department of Commerce
Panelists: Karen Mossberger, Univ of IL at Chicago. Nicol Turner-Lee, Joint Center for Political Studies. Samantha Becker, Univ of Washington. Blanca Gordo, Berkeley
Notes:
Categories of evaluation
1. Personal outcomes
2. Interpersonal outcomes
3. Organizational outcomes
4. Community -level outcomes
-city-wide surveys
---barriers to broadband use, technology use barriers
Joint Center
Working with OneEconomy to connect public housing
Works with pew studies polling services
GOALS
- determine how well the program meets need
- evaluate how well program extends broadband use
- assess and document the shift in the culture of technology use
- comparative research design
Evaluation approach uses a comparative model > test cities vs. Control cities
CHALLENGES
1. Identifying a stable population
2. Consistent data
3. Anticipating lag time between participation and service connection
4. Disentangling test cities from control cities
5. Capturing racial,ethnic, and language diversity of constituents between test and control cities
SAMANTHA BECKER - U.S. IMPACT Study
Assist PCCs in doing their own evaluation ( to keep costs down)
Framework
Context analysis > situated logic model > outcome measurement > validation and reflective practice
Step 1
Policy issue mapping - what is the most pressing issue in the community
Step 2
Identify other stakeholders concerned with policy goals
Step 3
Develop a policy or stakeholder logic model
Inputs > activities > outcomes > impacts
Step 4
Bridge the logic model to show how the CTC supports larger policy goals
SideBar
What is a measurable indicator?
--cost effective
--specific, unique, unambiguous
--understandable
-- relevant
-- time-bound
Step 6
Measuring the outcomes
Step 8
Validate outcome measures
WA BTOP's reporting system relates to a community technology logic model
IMPACT TYPES
Comp skills enhancement
Employment skills and opportunities
Education enhancement
Access to information and services
Life skills and social inclusion
Francine Jefferson - "We should have a series of webinars about how you operationalize variables....or create a research/evaluation portal".
Integrate survey questions into the program. Be clear with program participants that the program funding is dependent upon knowing the program is working (ie. evaluation data).
There is a blog post at the following: http://somewhereonthebridge.wordpress.com/
Presentation Slides:
Becker_CBAIS_06_27_2011.pptx evaluatingBB_CBAIS.pptx OE_BBOC BTOP Evaluation Presentation.pptx
Comments (20)
Tom.Miller@onecommunity.org said
at 11:29 am on Jun 27, 2011
Suggestion made to aggregate/share survey/research templates for SBA/PPC programs.
Connect Your Community said
at 11:36 am on Jun 27, 2011
Seems like polling is a common method for gathering data samples! but incentivizing is expensive - post youu ides for alternative low coast methods here!!!
Connect Your Community said
at 11:39 am on Jun 27, 2011
Contact the Urban Planning department at your local college or University to partner on evaluation!
Bartholomew Gonzalez said
at 11:39 am on Jun 27, 2011
surveymonkey.com
Angela Siefer said
at 11:42 am on Jun 27, 2011
Need location for sharing surveys.
Amy Luckey said
at 11:51 am on Jun 27, 2011
Absolutely. Rather than waiting for the potential federal evaluation portal - which would be great, can we at least get the 4 speakers' tools posted on this wiki now?
Tom.Miller@onecommunity.org said
at 11:58 am on Jun 27, 2011
thinking digitalliteracy.gov is the right place
Amy Luckey said
at 11:58 am on Jun 27, 2011
I think this is the evaluation study, including survey questions in appendix - that was referenced by Samatha Becker: http://www.cityofchicago.org/content/dam/city/depts/doit/supp_info/DEI/Digital_Excellence_Study_2009.pdf
Connect Your Community said
at 12:00 pm on Jun 27, 2011
YES !! Feel free to post any evaluation materials you might have to this wiki page! If we get enough - I will create a new page just for evaluation resources!
Angela Siefer said
at 12:02 pm on Jun 27, 2011
I emailed the evaluation panelists and asked them to post their evaluation materials, particularly survey questions, this this session page.
Amy Luckey said
at 12:03 pm on Jun 27, 2011
Public Policy Institute of California Californians and Information Technology survey funded by ZeroDivide and CETF was released last week. PPIC worked with Pew to develop some of the questions: http://www.ppic.org/main/publication.asp?i=985
Connect Your Community said
at 12:11 pm on Jun 27, 2011
There are many of us who will likely lode funding in a year or two and would like to continue impacting out communities
Can you comment on the notion of sustainability ( at the CTC level) in how evaluation strategies can be used to extend programming by helping to identify community and organizational partners?
Jemarius Moore said
at 12:21 pm on Jun 27, 2011
How can it be said that there are multiple factors used to evaluate the effect of the program when the only statistic seemingly used is how many new broadband connections in an area?
Tom.Miller@onecommunity.org said
at 12:25 pm on Jun 27, 2011
tell your stories at tomorrow's Strategic Community Impacts - K-12 workshop!
Amy Luckey said
at 12:25 pm on Jun 27, 2011
Would be great for BTOP grantees to work together with the expert evaluators to ID of the 4-5 survey questions that could be asked across most evaluations. Also, development of of a menu of survey questions that are likely relevant to most evaluations.
Samantha Becker said
at 1:38 pm on Jun 27, 2011
The U.S. IMPACT Public Library Study (Opportunity for All: How the American Public Benefits from Internet Access at U.S. Libraries) is available here: http://cis.washington.edu/usimpact/us-public-library-study.html
The survey instruments are included in the appendices.
Amy Luckey said
at 1:42 pm on Jun 27, 2011
Thank you!
mdavis@matrixhs.org said
at 3:02 pm on Jun 27, 2011
Having asked the initial session question regarding the need for a template for online surveying SBAs on their meaningful use experience, I was hoping that our main supporter (BTOP) would offer a set of general survey questions for use by the CYC agencies that provided data reporting for all levels of the program (CYC agency, One Community, BTOP, etc.). Special, localized survey questions for individual agencies or CYC could also be added at their appropriate levels. Online surveying should be the initial medium for post-SBA data capture. Initial intake forms currently in use should provide the necessary baseline data for the SBA. Any hope on getting this type of survey template real soon, as new program funding needs are rapidly approaching?
LeeAnn Shreve said
at 12:39 pm on Jun 28, 2011
I have uploaded several of our surveys we use for evaluating our program. We also use the results of this survey to determine what services are needed in particular communities. They are called: Baseline Survey - Future Generations, Business Survey - Future Generations and Online Survey - Future Generations.
If you have any questions, please email me at leeann@future.edu. I'd be happy to discuss our research and evaluation process with you!!
Colin Rhinesmith said
at 3:03 pm on Jul 8, 2011
I've been collecting resources related to BTOP evaluation. I've posted them on my blog at Community Media in Transition, here: http://cmediachange.net/2011/06/29/evaluating-btop-projects/
You don't have permission to comment on this page.