[{"command":"add_css","data":[{"rel":"stylesheet","media":"all","href":"\/sites\/default\/files\/css\/css_-kTcRwkqPhQLsqEvdSSyU2XZ3CV5VEleXoE3_xZi2cg.css?delta=0\u0026language=cs\u0026theme=hm_theme\u0026include=eJxLzi9K1U8pKi1IzNErSa0oSSxKTdQtSi3OrEoFAJQ3Cqw"}]},{"command":"openDialog","selector":"#drupal-modal","settings":null,"data":"\u003Cdiv id=\u0022republish_modal_form\u0022\u003E\u003Cform class=\u0022modal-form-example-modal-form ecl-form\u0022 data-drupal-selector=\u0022modal-form-example-modal-form\u0022 action=\u0022\/cs\/article\/modal\/7324\u0022 method=\u0022post\u0022 id=\u0022modal-form-example-modal-form\u0022 accept-charset=\u0022UTF-8\u0022\u003E\u003Cp\u003EHorizon articles can be republished for free under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence.\u003C\/p\u003E\n      \u003Cp\u003EYou must give appropriate credit. We ask you to do this by:\u003Cbr \/\u003E\n      1) Using the original journalist\u0027s byline\u003Cbr \/\u003E\n      2) Linking back to our original story\u003Cbr \/\u003E\n      3) Using the following text in the footer: This article was originally published in \u003Ca href=\u0027#\u0027\u003EHorizon, the EU Research and Innovation magazine\u003C\/a\u003E\u003C\/p\u003E\n      \u003Cp\u003ESee our full republication guidelines \u003Ca href=\u0027\/horizon-magazine\/republish-our-stories\u0027\u003Ehere\u003C\/a\u003E\u003C\/p\u003E\n      \u003Cp\u003EHTML for this article, including the attribution and page view counter, is below:\u003C\/p\u003E\u003Cdiv class=\u0022js-form-item form-item js-form-type-textarea form-item-body-content js-form-item-body-content ecl-form-group ecl-form-group--text-area form-no-label ecl-u-mv-m\u0022\u003E\n        \n\u003Cdiv\u003E\n  \u003Ctextarea data-drupal-selector=\u0022edit-body-content\u0022 aria-describedby=\u0022edit-body-content--description\u0022 id=\u0022edit-body-content\u0022 name=\u0022body_content\u0022 rows=\u00225\u0022 cols=\u002260\u0022 class=\u0022form-textarea resize-vertical ecl-text-area\u0022\u003E\u003Ch2\u003EOpening the \u2018black box\u2019 of artificial intelligence\u003C\/h2\u003E\u003Cp\u003EIn February of 2013, Eric Loomis was driving around in the small town of La Crosse in Wisconsin, US, when he was stopped by the police. The car he was driving turned out to have been involved in a shooting, and he was arrested. Eventually a court \u003Ca href=\u0022https:\/\/www.nytimes.com\/2017\/05\/01\/us\/politics\/sent-to-prison-by-a-software-programs-secret-algorithms.html\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003Esentenced\u003C\/a\u003E him to six years in prison.\u003C\/p\u003E\u003Cp\u003EThis might have been an uneventful case, had it not been for a piece of technology that had aided the judge in making the decision. They used COMPAS, an algorithm that determines the risk of a defendant becoming a recidivist. The court inputs a range of data, like the defendant\u2019s demographic information, into the system, which yields a score of how likely they are to again commit a crime.\u003C\/p\u003E\u003Cp\u003EHow the algorithm predicts this, however, remains non-transparent. The system, in other words, is a black box \u2013 a practice against which Loomis made a 2017 complaint in the US Supreme Court. He claimed COMPAS used gender and racial data to make its decisions, and ranked Afro-Americans as higher recidivism risks. The court eventually \u003Ca href=\u0022https:\/\/www.theatlantic.com\/technology\/archive\/2018\/01\/equivant-compas-algorithm\/550646\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003Erejected\u003C\/a\u003E his case, claiming the sentence would have been the same even without the algorithm. Yet there have also been a number of \u003Ca href=\u0022https:\/\/advances.sciencemag.org\/content\/4\/1\/eaao5580\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003Erevelations\u003C\/a\u003E which suggest COMPAS doesn\u2019t accurately predict recidivism.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAdoption\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EWhile algorithmic sentencing systems are already in use in the US, in Europe their adoption has generally been limited. A \u003Ca href=\u0022https:\/\/www.groene.nl\/artikel\/robotrechter-e-court-ligt-plat\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EDutch AI sentencing system\u003C\/a\u003E, that judged on private cases like late payments to companies, was for example shut down in 2018 after critical media coverage. Yet AI has entered into other fields across Europe. It is being rolled out to help European doctors diagnose \u003Ca href=\u0022https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/using-ai-fast-and-effectively-diagnose-covid-19-hospitals\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003ECovid-19\u003C\/a\u003E. And start-ups like the British \u003Ca href=\u0022https:\/\/sifted.eu\/articles\/ai-mortgage-decisions\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EM:QUBE\u003C\/a\u003E, which uses AI to analyse mortgage applications, are popping up fast.\u003C\/p\u003E\u003Cp\u003EThese systems run historical data through an algorithm, which then comes up with a prediction or course of action. Yet often we don\u2019t know how such a system reaches its conclusion. It might work correctly, or it might have a technical error inside of it. It might even reproduce some form of bias, like racism, without the designers even realising it.\u003C\/p\u003E\u003Cp\u003EThis is why researchers want to open this black box, and make AI systems transparent, or \u2018explainable\u2019, a movement that is now picking up steam. The \u003Ca href=\u0022https:\/\/ec.europa.eu\/info\/sites\/info\/files\/commission-white-paper-artificial-intelligence-feb2020_en.pdf\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EEU White Paper on Artificial Intelligence\u003C\/a\u003E released earlier this year called for explainable AI, major companies like \u003Ca href=\u0022https:\/\/www.researchworld.com\/can-googles-new-explainable-ai-make-it-easier-to-understand-artificial-intelligence\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EGoogle\u003C\/a\u003E and \u003Ca href=\u0022https:\/\/www.zdnet.com\/article\/ibm-offers-explainable-ai-toolkit-but-its-open-to-interpretation\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EIBM\u003C\/a\u003E are funding research into it and GDPR even includes a right to explainability for consumers.\u003C\/p\u003E\u003Cp\u003E\u2018We are now able to produce AI models that are very efficient in making decisions,\u2019 said Fosca Giannotti, senior researcher at the Information Science and Technology Institute of the National Research Council in Pisa, Italy. \u2018But often these models are impossible to understand for the end-user, which is why explainable AI is becoming so popular.\u2019\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EDiagnosis\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EGiannotti leads a research project on explainable AI, called \u003Ca href=\u0022https:\/\/cordis.europa.eu\/project\/id\/834756\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EXAI\u003C\/a\u003E, which wants to make AI systems reveal their internal logic. The project works on automated decision support systems like technology that helps a doctor make a diagnosis or algorithms that recommend to banks whether or not to give someone a loan. They hope to develop the technical methods or even new algorithms that can help make AI explainable.\u003C\/p\u003E\u003Cp\u003E\u2018Humans still make the final decisions in these systems,\u0027 said Giannotti. \u2018But every human that uses these systems should have a clear understanding of the logic behind the suggestion. \u2019\u003C\/p\u003E\u003Cp\u003EToday, hospitals and doctors increasingly experiment with \u003Ca href=\u0022https:\/\/www.wired.com\/story\/ai-doctor-will-see-you-now\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003EAI systems\u003C\/a\u003E to support their decisions, but are often unaware of how the decision was made. AI in this case analyses large amounts of medical data, and yields a percentage of likelihood a patient has a certain disease.\u003C\/p\u003E\u003Cp\u003EFor example, a system might be trained on large amounts of photos of human skin, which in some cases represent symptoms of skin cancer. Based on that data, it predicts whether someone is likely to have skin cancer from new pictures of a skin anomaly. These systems are not general practice yet, but hospitals are increasingly testing them, and integrating them in their daily work.\u003C\/p\u003E\u003Cp\u003EThese systems often use a popular AI method called deep learning, that takes large amounts of small sub-decisions. These are grouped into a network with layers that can range from a few dozen up to hundreds deep, making it particularly hard to see why the system suggested someone has skin cancer, for example, or to identify faulty reasoning.\u003C\/p\u003E\u003Cp\u003E\u2018Sometimes even the computer scientist who designed the network cannot really understand the logic,\u0027 said Giannotti.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ENatural language\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EFor Sen\u00e9n Barro, professor of computer science and artificial intelligence at the University of Santiago de Compostela in Spain, AI should not only be able to justify its decisions but do so using human language.\u003C\/p\u003E\u003Cp\u003E\u2018Explainable AI should be able to communicate the outcome naturally to humans, but also the reasoning process that justifies the result,\u2019 said Prof. Barro.\u003C\/p\u003E\u003Cp\u003EHe is scientific coordinator of a project called \u003Ca href=\u0022https:\/\/cordis.europa.eu\/project\/id\/860621\u0022 target=\u0022_blank\u0022 rel=\u0022noopener noreferrer\u0022\u003ENL4XAI\u003C\/a\u003E which is training researchers on how to make AI systems explainable, by exploring different sub-areas such as specific techniques to accomplish explainability.\u003C\/p\u003E\u003Cp\u003EHe says that the end result could look similar to a chatbot. \u2018Natural language technology can build conversational agents that convey these interactive explanations to humans,\u2019 he said.\u003C\/p\u003E\u003Cp\u003E\u003Cblockquote class=\u0022text-center text-blue font-bold text-2xl w-full lg:w-1\/2 border-2 border-blue p-12 my-8 lg:m-12 lg:-ml-16 float-left\u0022\u003E\n  \u003Cspan class=\u0022text-5xl rotate-180\u0022\u003E\u201c\u003C\/span\u003E\n  \u003Cp class=\u0022font-serif italic\u0022\u003E\u2018Explainable AI should be able to communicate the outcome naturally to humans, but also the reasoning process that justifies the result.\u2019\u003C\/p\u003E\n  \u003Cfooter\u003E\n    \u003Ccite class=\u0022not-italic font-normal text-sm text-black\u0022\u003EProf. Sen\u00e9n Barro, University of Santiago de Compostela Spain\u003C\/cite\u003E\n  \u003C\/footer\u003E\n\u003C\/blockquote\u003E\n\u003C\/p\u003E\u003Cp\u003EAnother method to give explanations is for the system to provide a counterfactual. \u2018It might mean that the system gives an example of what someone would need to change to alter the solution,\u2019 said Giannotti. In the case of a loan-judging algorithm, a counterfactual might show to someone whose loan was denied what the nearest case would be where they would be approved. It might say that someone\u2019s salary is too low, but if they earned \u20ac1,000 more on a yearly basis, they would be eligible.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EWhite box\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EGiannotti says there are two main approaches to explainability. One is to start from black box algorithms, which are not capable of explaining their results themselves, and find ways to uncover their inner logic. Researchers can attach another algorithm to this black box system \u2013 an \u2018explanator\u2019 \u2013 which asks a range of questions of the black box and compares the results with the input it offered. From this process the explanator can reconstruct how the black box system works.\u003C\/p\u003E\u003Cp\u003E\u2018But another way is just to throw away the black box, and use white box algorithms, \u2019 said Giannotti. These are machine learning systems that are explainable by design, yet often are less powerful than their black box counterparts.\u003C\/p\u003E\u003Cp\u003E\u2018We cannot yet say which approach is better,\u2019 cautioned Giannotti. \u2018The choice depends on the data we are working on.\u2019 When analysing very big amounts of data, like a database filled with high-resolution images, a black box system is often needed because they are more powerful. But for lighter tasks, a white box algorithm might work better.\u003C\/p\u003E\u003Cp\u003EFinding the right approach to achieving explainability is still a big problem though. Researchers need to find technical measures to see whether an explanation actually explains a black-box system well. \u2018The biggest challenge is on defining new evaluation protocols to validate the goodness and effectiveness of the generated explanation,\u2019 said Prof. Barro of NL4XAI.\u003C\/p\u003E\u003Cp\u003EOn top of that, the exact definition of explainability is somewhat unclear, and depends on the situation in which it is applied. An AI researcher who writes an algorithm will need a different kind of explanation compared to a doctor who uses a system to make medical diagnoses.\u003C\/p\u003E\u003Cp\u003E\u2018Human evaluation (of the system\u2019s output) is inherently subjective since it depends on the background of the person who interacts with the intelligent machine,\u2019 said Dr Jose Mar\u00eda Alonso, deputy coordinator of NL4XAI and also a researcher at the University of Santiago de Compostela.\u003C\/p\u003E\u003Cp\u003EYet the drive for explainable AI is moving along step by step, which would improve cooperation between humans and machines. \u2018Humans won\u0027t be replaced by AI,\u0027 said Giannotti. \u2018They will be amplified by computers. But explanation is an important precondition for this cooperation.\u2019\u003C\/p\u003E\u003Cp\u003E\u003Cem\u003EThe research in this article was funded by the EU. If you liked this article, please consider sharing it on social media.\u003C\/em\u003E\u003C\/p\u003E\u003C\/textarea\u003E\n\u003C\/div\u003E\n\n            \u003Cdiv id=\u0022edit-body-content--description\u0022 class=\u0022ecl-help-block description\u0022\u003E\n      Please copy the above code and embed it onto your website to republish.\n    \u003C\/div\u003E\n  \u003C\/div\u003E\n\u003Cinput autocomplete=\u0022off\u0022 data-drupal-selector=\u0022form-lbxnmvgzbmwjhhjdpm6cgfxbkt94ynx-01lpzp3nzci\u0022 type=\u0022hidden\u0022 name=\u0022form_build_id\u0022 value=\u0022form-lBXnmVgzbMWJhhJdpm6CGFxbKT94ynX-01lPZP3NZCI\u0022 \/\u003E\n\u003Cinput data-drupal-selector=\u0022edit-modal-form-example-modal-form\u0022 type=\u0022hidden\u0022 name=\u0022form_id\u0022 value=\u0022modal_form_example_modal_form\u0022 \/\u003E\n\u003C\/form\u003E\n\u003C\/div\u003E","dialogOptions":{"width":"800","modal":true,"title":"Republish this content"}}]