To measure meaningful impact, beyond ratings and “likes”, requires a flexible, multi-faceted approach. Many projects use a blend of quantitative and qualitative assessment tools. Qualitative methodologies are gaining greater notice in recent years, as they can provide opportunities for more diverse and inclusive levels of assessment that might countermand the inherent biases of quantitative data. Impact can be assessed at many levels of direct distribution, such as individual, societal, organizational, as well as the analysis of discourse (online or offline) before and after the film is presented. Analyzing and understanding what the status quo thinking is around a film’s subject matter informs how a film can influence the ecosystem of advocacy efforts in conversations, social media, news, even speeches. Dr. Karen Hughes explains in her article Measuring the Impact of Viewing Wildlife, ” the best indicator of a program’s effectiveness is whether or not it increases (viewers’) commitment to, and involvement in, conservation actions.” More investment is not necessarily better. The key is a well-defined impact assessment strategy, implemented at the outset of the project, that has clear goals.
In addition to some better-known tools, we list some approaches that while are not typically used in film assessment, might be helpful in evaluating long-term media impact.
To evaluate the full impact of a film, it is necessary to look at all of the elements from the early development of the film idea, to contextual conversations in potential communities where the film might be shown, to the final product and how it performs during distribution and outreach. Assessment is not just in terms of numbers, but in reactions and actions. Impact is measurable against a current snapshot of the state of the conversation and the level of awareness of the topic subject of the film. Measuring impact is a fluid and flexible process, rather than a procedure, and evolves over time according to the goals set and the constant validation and reconsideration of them.
In this first phase of our research, we identified several measurement tools that could be useful for impact evaluation. Several of them are included in both Assessment of Social Issue Documentaries: Research Methods & Future Considerations , and The Impact Field Guide and Toolkit . Others used in educational arenas for training evaluation and instructional design. It can be useful to try different frameworks and combinations of tools.
(Four Level Approach)
A multi-level framework originally designed to evaluate the effectiveness of training programs, the Kirkpatrick approach follows up with trainees to evaluate different elements and outcomes over time.
The Kirkpatrick approach is useful for the creation of surveys, and to help evaluate the core messaging of a film during production, distribution and outreach.
Used for adult learners, the ADDIE framework provides a simple and effective pathway to assess how a learning resource, such as a movie, is effective in achieving intended objectives and generating knowledge.
The framework and model could be useful in the development and pre-production stages to evaluate how messaging and impact goals can be further reinforced.
A mobile web-based solution that allows filmmakers to assess impact of the media “on-the-fly”, by providing the end viewer a set of simple tools to express their attitudes about what they are watching. The interactive elements can be paused briefly during the media in order to ask a targeted question.
Useful for focus groups and within controlled screenings of a film. Its output informs effectiveness of messaging and measurement of the perception “in the moment”. This tool is most useful when accompanied with a long-term assessment tool that looks to validate views and attitudes identified in this early stage.
Created by ITV$, OVEE is a cinematic social TV experience. Working as a live platform, it allows teams to hold screenings with live Q&A and guests, and provides a remote screening experience in high-quality.
The platform allows interaction and communication among the virtual attendees. Interactions can be assessed as people watch the film, and several other assessment tools can be included in the experience.
Text mining and discourse analysis
This online platform processes high amounts of text and interventions in online social media platforms, or other types of qualitative data. It identifies common trends in conversations around a topic or given question and can be used for discourse and sentiment analysis.
This particular program can be useful in assessing the current state of affairs on a given topic, core messaging, or context for the film.
Explorer / Topic Mapper / Source Mgr.
An open platform from MIT that provides media analytics and an overview of digital discussions around particular topics.
The MIT Media Cloud tools allows teams to better understand contextual discussions while developing a film and its impact strategy.
in person or on line supported "Collaborative Filtering"
The establishment of randomized controlled trials (RCTs) in an isolated laboratory environment is an excellent way of to evaluate audience reaction to a film. There are also online tools such as Dev-CAFE, DocScale.
RCTs provide a comparable approach to validate a film’s reactive effect, and is most useful for immediate assessment, rather than for long-term follow up analysis.
*Human Mounted Displays (VR or AR glasses)
This approach can be done with an eye-tracker device mounted on a screen, or with the Steam-VR plug-in from Adobe Premiere Pro. This exercise can inform the post-production/testing phase of a VR or 360 experience to identify the areas of interest in the virtual environment.
The assessment tool informs beyond reactions to the placement of elements. It reveals areas of interest and how the user is consuming and exploring both the media and the media format.
An essential compilation of guidelines and best practices for environmental and science related films.
This compilation of tools offers a comprehensive guide to evaluate every stage of an environmental and nature, science-related film project.
Up Next