Trust and Automated Decision-Making: an interim report on the Digital Council’s 2020 research project
Digital and data-driven technologies have the potential to increase equity, inclusion and well-being for everyone. Our job as the Digital Council for Aotearoa New Zealand is to help the government make this vision of an inclusive, technology-driven future a reality.
In order to give informed, practical advice, we need a deep understanding of the barriers to achieving this vision, and the opportunities that are there to grasp. A quick scan of previous research showed us that trust and trustworthiness were key factors in unlocking the potential of digital technologies for social and economic well-being. This led us to choosing trust as the focus of our research and advice this year.
Our 2020 research project focuses on trust as it relates to a set of technologies and processes called automated decision-making (ADM). People not familiar with this topic might instead know about “algorithms”, “artificial intelligence”, “data science” or “machine learning”, which all overlap with ADM. When we unlock a smartphone with a fingerprint or turn to a streaming service for recommendations, we are interacting with ADM. When we apply for a job or loan, or need surgery, ADM might help to determine if our application is accepted or how far up the waiting list we are placed.
ADM has significant impacts on individual lives and how society functions. There are many beneficial uses of ADM, from helping us navigate a new city with an app to increasing the speed and accuracy of diagnosis of certain medical conditions. However, ADM is not without risks. For example, ADM can be used in ways that cause harm and reinforce historical bias and injustice. If not used responsibly, there is potential for ADM to have significant negative impacts for New Zealand, and especially for people who are already disadvantaged or marginalised.
There has already been lots of global research on ADM, including on its relationship to trust. However, we do not know very much about how people whose lives are impacted by ADM define trustworthiness. We do not know what their current levels of trust are, or what they need in order to trust these technologies. Our research will help to answer these questions in an Aotearoa New Zealand context, and advise the government on how to ensure ADM processes are both trusted and trustworthy. We are excited to be working with partners Brainbox and Toi Āria on this research.
This document is an overview of the research project and our progress so far. In December 2020, we will release a final report to the Minister for Government Digital Services and Minister of Statistics with our in-depth findings and insights. This report will include concrete steps the government, and other organisations, can take to ensure that ADM is trusted and that, as a nation, we make the most of these powerful technologies.
We will continue to work in the open and provide regular updates of our work on our Medium page over the coming months. We are excited about the next stages of this work and welcome you to join us on this journey.
Chair, Digital Council for Aotearoa New Zealand
This year’s research topic: trust and automated decision-making
The Digital Council for Aotearoa New Zealand’s major research project for 2020 is exploring trust and trustworthiness in digital technologies. Our research question is “What is needed to ensure New Zealand has the right levels of trust required to harness the full societal benefits of digital and data-driven technologies?”
This a broad topic which could be approached in a number of ways. To further focus our research, we decided to look at automated decision-making (ADM) as a case study in trust. ADM affects people’s everyday lives, and has deep societal impacts. It can be used in many beneficial ways, but there are also risks which can lead to negative or unforeseen consequences, particularly for disadvantaged or marginalised groups.
More information about how we chose ADM as the focus of our research, and the other topics we considered, is outlined in the Digital Council’s Weeknotes #4.
About the research project
This research project has 3 parts:
Part 1: Building foundations
A literature review to help us better understand the topic of trust and ADM was carried out between May and June 2020 by research partners Brainbox. This aimed to build further understanding and insights to inform the next stages of the research project, and to build a strong evidence base for this subject in an Aotearoa New Zealand context. The literature review drew on academic literature, other research, and discussions with a range of people from business, academic institutions and government agencies.
Some of the findings are outlined in the key insights section of this document. The literature review will be used to inform the Council’s final report in December.
Part 2: Finding out what people across New Zealand think about ADM
Workshops with a wide range of people will be held in August and September 2020 to find out what non-experts think about ADM, including any worries or concerns they have about how it is used. These will be interactive sessions where facilitators will take participants through a number of real-world scenarios where ADM components are employed. This part of the research is being led by Toi Āria public engagement centre, which is based out of Massey University.
The focus of this stage of the research is on hearing from the people and communities in Aotearoa New Zealand whose lives are most impacted by ADM.
Part 3: Bringing it all together
The insights from the first 2 parts of the research will be brought together to inform a final report to Ministers. This will include an explanation of key issues around ADM and trust, and recommendations for concrete actions and next steps government agencies and other organisations can take to promote trusted and trustworthy applications of ADM.
The process of bringing insights together will include engaging with a range of stakeholders to ensure that people have the opportunity to be an ongoing part of the conversation about ADM.
Defining automated decision-making
Broadly speaking, ADM refers to decision-making processes where some aspects are carried out by computer programs. ADM often has a data analysis component. When people talk about using algorithms, machine learning, personal data collection and use, data science, or predictive modelling, they are referring to processes that overlap with ADM to some extent.
ADM has been around for decades and is part of many of our day-to-day activities. You probably encountered ADM today without even realising it. As more data is collected and stored about people and the wider world, and computing power and storage increases, so do the potential applications and impacts of ADM. There are some applications of ADM that have been controversial due to potential bias or negative effects on people. An example is facial recognition technologies, which can exhibit systemic racial and gender bias, and has been banned in some American cities.
Where and how automated decision-making is used
There are a broad range of applications for ADM. We might interact with some of these every day, while others are fairly new and are still in the development phase. Examples include:
- music and video streaming services that give recommendations about what to watch or listen to next,
- online pricing systems that determine how much you are charged for a flight or hotel room,
- smart home speakers and devices which carry out voice-based instructions,
- medical diagnosis systems that identify potential medical conditions from scans or X-rays,
- systems that process applications for government services and entitlements like passports, benefits and visas,
- predictive maintenance systems that help companies like airlines identify when their planes need maintenance or repair, reducing the amount of time they are out of use,
- environmental monitoring to understand the numbers of a certain bird or animal in an area, and
- autonomous robots like self-driving cars.
Key insights from our research so far
Here are a few of the things we have found really interesting from our research so far. We will be testing some of these insights further in the next part of our research, and will outline our findings in more detail in the final December report.
Automated decision-making is not one thing or idea
ADM is an umbrella term that refers to a wide range of systems and processes. Each application of ADM will have differing levels of complexity, types of data inputs, and impacts on people and society. This diversity means there is not one set of risks and benefits that cover all ADM applications. Risks, benefits and trust levels are context specific and will likely vary depending on the specific application of ADM, the potential impacts, types of data and technology uses, and organisations who implement the systems.
Humans play an important part in ADM systems
The idea of “automation” brings to mind computers doing their own thing with little human intervention, and perhaps even a level of agency. However, there is always some level of human interaction throughout an ADM process. For example, automated systems are designed and built by humans, and processes surrounding ADM are developed, implemented and communicated by humans. To ensure trust in ADM systems, we propose that there must also be trust in the human elements of a system.
Trust and ADM is complex
Part of the aim of our literature review was to unpick some of the ideas around trust, especially in relation to ADM. As expected, we found that it is a very complex area.
Trust is hard to define, and like many terms, people use it to mean different things. We are using the working definition that trust is about whether people are comfortable in a situation where they are vulnerable to the consequences of someone else's actions. We will refine and test this definition further in the coming months.
It is important that our research recommendations do not move us towards a system that is trusted, without being trustworthy. Trust can be built in a number of ways that may not require a system itself to be trustworthy, for example through engaging branding and marketing. Aspects that could make a system trustworthy include whether it is reliable and accurate, mitigates against negative bias, is transparent or explainable, and is safe and effective. We will explore this area further, and we expect that desired levels of trustworthiness might vary depending on the situation in which ADM is being used.
We have identified 2 key areas where trust is relevant in an ADM system:
- Trust from people subject to, or actively interacting with, ADM systems and the organisations that implement them. Our current hypothesis is that people are unlikely to trust an ADM system if they do not trust the wider organisation that is deploying it. This reflects that trust in ADM is about both technology and the wider systems in which the technology is embedded.
- Trust from people who are thinking about implementing ADM systems in their business or organisation. This is about having trust in ADM technical systems. If there is not trust by decision-makers and implementers, there could be an opportunity cost from under-use or over-reliance on ADM.
Here are some other insights about trust and ADM that we want to test and investigate further.
Building trust in ADM systems can happen at a number of levels. For example, individuals might not need to trust an organisation implementing an ADM system if appropriate accountability mechanisms are in place so any harmful consequences can be identified and addressed. This spreads the vulnerability to both parties. In other words, if the ADM system does something wrong, the people responsible for it will be held to account.
There are some situations where a person may not trust an ADM system, but choose to participate because the benefits outweigh the risks. For example, people may use video and music streaming sites knowing all their clicks and preferences are tracked, as it provides them with a valuable service. Other times, people may not be given a viable alternative, so may participate even though they may have low levels of trust. This could include applying for a bank loan, where algorithms help to determine whether you are approved.
Topics to consider as we complete our research
During the literature review and initial stakeholder interviews, a few key topics arose that will be useful to keep in mind when carrying out the next stages of research:
- ADM involves data, and frequently personal and environmental data. Because of this, we should not think about ADM and trust in an Aotearoa New Zealand context without thinking about Te Tiriti o Waitangi, the specific qualities of Māori data, and concepts like Māori data sovereignty. There is an opportunity to work with experts in these areas, and engage further with Māori, to ensure that any recommendations and next steps from this research reflect the needs and aspirations of Māori.
- There are already many high-level principles around aspects of ADM, both globally and in an Aotearoa New Zealand context. There is an opportunity to go beyond principles, and consider practical steps and guidance for implementing ADM systems.
- Much of our focus to date has been around ensuring trust from the people who are subject to ADM systems. There is also an opportunity to further investigate the levels of trust needed by people implementing ADM systems. For example, what needs to be in place for senior decision-makers to be comfortable with implementing ADM systems to solve problems or grasp opportunities in their organisation?
During the second half of 2020, Toi Āria will be leading its workshop research and we will bring all findings and insights from this work and the literature review together. We will produce a report to Ministers with a set of recommendations. This will be completed in December and will also be available to the public. We will also explore other information or resources that we could produce to support the report and make the biggest possible positive impact for people using and adopting digital and data-driven technologies in Aotearoa New Zealand.
You can keep up to date with our progress on our Medium page, where the Council publishes weekly updates.