Open comparisons are a collection of indicators that are collected at the municipal level throughout Sweden. The indicators measure quality in social services and healthcare. In 2021, the National Audit Office submitted a report to the Swedish parliament: ’Open comparisons in social services - limited contribution to good quality’. The National Audit
Office recommended that a critical review of existing indicators be carried out. The basic premise was that only indicators of clear benefit to the municipalities, the state’s management of knowledge or national follow-ups should be included. The National Audit Office’s report was the start of extensive work with improvement of open comparisons. As part of this work, Point ran a project to examine the usefulness of open comparisons for the municipalities. The purpose was to develop a framework for the evaluation of indicators and to give suggestions for indicators that can be discontinued.
The project used Points strategic design framework as a basis and through several workshops with stakeholders, we explored issues and delimitations early on, to avoid a too narrow definition of the problem or that effective methods were overlooked.
In order to get the most complete picture possible of the issue, data from a number of different sources were used: survey data from the National Board of Health and Welfare, several years’ results from Open comparisons (both from the National Board of Health and Welfare and the database Kolada), desk research mainly in the form of academic research and reports from other consultants, data collected by Point through web scraping and data from the work group at the National Board of Health and Welfare. All this data was iteratively co-analyzed in a structured way to gradually build the insights that formed the basis for developing a new framework.
Point delivered a framework for critical evaluation of indicators within open comparisons with a focus on benefits for the municipalities. In addition, Point also set up interactive dashboards that made it possible for the National Board of Health and Welfare to visualize how different demarcations affect existing indicators. Point’s work also high-lighted the possibilities of new data collection methods such as web scraping for this type of evaluations.
This type of data can supplement other data and be co-analyzed to provide a more complete picture and a better understanding of indicator outcomes.