Open Source for MERL Workflows
Mala Kumar, Stephanie Coker, Vidya Mahadevan
Last updated in April 2021
Background
In the social sector, we use Monitoring, Evaluation, Research and Learning (MERL) to understand if programs achieve their intended impact and through ongoing monitoring of activities, allow managers to provide course correction during the program lifecycle. The MERL lifecycle is designed to encourage a continuous feedback loop and facilitate ongoing program adaptation. While there are many ways to depict a MERL lifecycle, this post will focus on how the MERL lifecycle relates to MERL technology workflows and the corresponding functionality of open source tools.
A MERL technology workflow (including open source) is defined as a series of discrete tasks using digital technology that together form an orchestrated and repeatable pattern of activity. The MERL lifecycle - below - is continuous in nature, and MERL technology workflows can support multiple stages of the MERL lifecycle. Understanding MERL workflows is key to integrating open-source tools’ functionality in the MERL lifecycle.
MERL practitioners describe three main groups of MERL technology workflows:
- Data Collection and Management (DCM)
- Data Analysis and Visualization (DAV)
- Reporting (Rep)
Where
- Data Collection and Management are MERL technology workflows that include but are not limited to:
- Creating, configuring or using a technology platform (e.g. mobile app, website) to gather quantitative or qualitative data, such as survey responses
- Storing quantitative or qualitative data, such as survey responses, on a database or spreadsheet
- Creating, modifying, enforcing permissions models to access quantitative or qualitative data through a technology platform
- Preparing data stored for further analysis using data relationships and models
- Creating governing mechanisms for stored data including data catalogs, metadata and data lineages
- Data Analysis and Visualization are MERL technology workflows that include but are not limited to:
- Inputting or importing raw data into a spreadsheet
- Cleaning raw quantitative or qualitative data in a spreadsheet or with a script (code)
- Generating a visualization using all or some of cleaned data
- Modifying the design or inputs of a visualization
- Reporting is MERL technologies workflows that include but are not limited to communicating research or learnings on a technology platform (e.g. blog, website, social media)
In a typical MERL lifecycle, these workflows occur at each stage and often happen simultaneously.
Figure 1 - Typical MERL and Program Implementation Lifecycle
For example, data visualizations can inform the design, implementation and learning of a project’s MERL lifecycle. However, the workflow of creating a data visualization requires a set of tasks to input (cleaned) data into a tool that produces a visual output. An open source tool’s functionality refers to the tasks it can do as part of a MERL technology workflow.
The below table illustrates one stage of the MERL lifecycle mapped to a MERL technology workflow and example tasks/functionality.
Example - MERL Lifecycle vs MERL Technology Workflow
| Portion of MERL Lifecycle | Corresponding MERL Technology Workflow | Example Tasks/Functionality |
|---|---|---|
| Communicate, Research | Data Analysis and Visualization | Input or import raw data into a spreadsheet |
| ” | ” | Clean raw data |
| ” | ” | Select section of raw data for visualization |
| ” | ” | Generate visualization |
| ” | ” | Modify design of visualization |
The following table provides example Open Source Software (OSS) - see footnote - tools for different MERL technology workflows and tasks/functionality. Note that this table is a starting point for exploring OSS tools for MERL and is not meant to be comprehensive. Links to explore more MERL open source tools are listed below the table.Some OS tools listed in the provide functionality for more than one MERL technology workflow, but the majority do not. Consequently, it becomes easier to identify OSS tools if you are able to identify the MERL technology workflow and the output needed, such as an evaluation matrix, spreadsheet or dashboard.
MERL Technology Workflows and Examples of OSS Tools
| MERL Technology Workflow | Tasks and Functionalities | MERL Output | Examples Open Source Software or Tool |
|---|---|---|---|
| Data Analysis and Visualization | Create visual representations of theories, Create visual representations of frameworks, Track KPIs | Theory of change | Theorymaker (GitHub repository) |
| ” | ” | Evaluation matrix | Kashana (GitHub repository) |
| ” | ” | Key performance indicators (measurement) framework | Kashana (GitHub repository) |
| Data Analysis and Visualization | Create visual representations of MERL activities, Project management | MERL Gantt/Project Timeline | OpenProject (GitHub repository) |
| ” | ” | Roles and responsibilities for MERL org chart | diagrams.net (GitHub repository) |
| Data Analysis and Visualization | Create [visual] representations of theories and frameworks | Theory of change | Theorymaker (GitHub repository) |
| ” | ” | Evaluation matrix | Kashana (GitHub repository) |
| Data Collection and Management | Create surveys, Create and store quantitative data, Create and store qualitative data | Survey data | Kobo Toolbox |
| ” | Loads forms and collect data offline | Survey data | ODK Collect |
| ” | Design and manage forms and its submissions | Survey data | ODK Central |
| ” | ” | Interview recordings/transcriptions | oTranscribe (GitHub repository) |
| ” | ” | Database management | PostgreSQL (GitHub repository) |
| Data Analysis and Visualization, Reporting | Perform statistical analyses, Perform text analysis, Perform content analysis, Generate data visualizations | Statistical analysis (descriptive and inferential) | The R Project (GitHub repository) |
| ” | ” | Text and content analysis | Coding Analysis Toolkit |
| ” | ” | Geospatial analysis | QGIS (GitHub repository) |
| ” | ” | Documents - Word processing | LibreOffice (GitHub repository) |
| ” | ” | Dashboard creation | RStudio Shiny Dashboard (GitHub repository) |
One goal of the MERL Center on GitHub is to help MERL practitioners identify open source tools for each stage of the MERL lifecycle, MERL technology workflow, functionality and output to increase adoption. By identifying examples of open source tools for each, we may encourage greater adoption of open source in MERL work.
Footnote 1
A core tenant of the Principles for Digital Development is using Open Standards, Open Licensing, Open Source and Open Innovation whenever possible. This reduces the number of “sector-specific solutions that are locked away behind licensing fees.” As described by the many contributors to the report Open Source Software in the Social Sector, the decision on if and how to use an open source product is complex and will be addressed in further guides and case studies of the MERL Center.
Further reading
- UNICEF et al, 2020. “Digital Public Goods, Explore”. Accessed May 28, 2020.
- Wasserman et al, 2017. “OSSpal: Finding and Evaluating Open Source Software”. Accessed May 26, 2020.
- Davies, 2011. “M&E Software: A List”. Accessed May 26, 2020.
- DIAL, 2020. Product Registry. Accessed May 27, 2020.
- DIAL, 2020. Workflows. Accessed March 7, 2022.