An algorithm that a major medical center used to identify patients for extra care has been shown to be racially biased, according to a paper published in Science. The IEEE Spectrum article has a nuanced discussion.

paper https://t.co/8ldhID0SZf

article https://t.co/g8nB5YzAEF pic.twitter.com/QHYq2T4A4v

— hardmaru ? (@hardmaru) October 26, 2019

Gosh, it's almost like if you train algorithms with racist data, they turn out to make racist decisions! https://t.co/27msjHxjIc

— Andrea Phillips (@andrhia) October 24, 2019

Wow: A health algorithm used in hospitals dramatically underestimated the medical needs of black patients, reducing the care they could get. "Correcting the bias would more than double the number of black patients flagged as at risk" https://t.co/K2C4Rb8ONB by @Carolynyjohnson pic.twitter.com/0npmiUgeD8

— Drew Harwell (@drewharwell) October 24, 2019

“...the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.”

Dissecting racial bias in an algorithm used to manage the health of populations. https://t.co/sgJFL26PdD | https://t.co/cVWAcHVeeI

— Brian Rahmer (@brianrahmer) October 26, 2019

“When the company replicated the analysis on a national data set...they found that black patients who were ranked by the algorithm as equally as in need of extra care as white patients were much sicker”https://t.co/D4hHCckXE0

— Legal Defense Fund (@NAACP_LDF) October 24, 2019

A series of studies argue that by focusing on costs as a proxy for health, risk algorithms are ignoring the racial inequalities in healthcare access.https://t.co/UfraQ3YgPf

— Motherboard (@motherboard) October 25, 2019

Would you get into a self-driving car trained only in Mountain View? Probably not. Then we better not tolerate medical #AI algorithms trained w/ small populations or only certain race/ethnicities! https://t.co/aBCREamRxc

— Atul Butte (@atulbutte) October 24, 2019

I also wonder whether women are going to be (potentially fatally) misdirected when triaging/symptom-checking chatbots become more pervasive. For instance, women can have different symptoms when experiencing heart attacks. https://t.co/jwmj89crsQ

— Christina Farr (@chrissyfarr) October 24, 2019

Seems like every generation of scientists has to re-learn this lesson the hard way! This mouse research mentality of small and homogeneous datasets is inconsistent with both #AI and #epidemiology. https://t.co/mygo4YaBvF

— Nima Aghaeepour (@nnimaa) October 25, 2019

What a surprise; For-profit industry coded, engineered and manufactured software and algorithms are biased to increase profits and earnings for the industry coding the software at the expense of patients and their doctors. Who woulda thought? https://t.co/RPLwfgGvIs

— Howard Green, MD (@DermHAG) October 25, 2019

Black patients were less likely than white ones to get extra medical help, despite being sicker, when an algorithm used by a large hospital chose who got the attention, acc to a new study underscoring the risks as tech gains a foothold in medicine https://t.co/ck7h1B5pN0

— Rayna ¯\_(ツ)_/¯ (@MaliciaRogue) October 27, 2019

Addressing #populationhealthmanagement requires diversity in scientists & ideas to develop innovative equitable tools as evidenced by this article. “Dissecting racial bias in an algorithm used to manage the health of populations” interesting read! https://t.co/77YwBSO2hc

— Joshua J. Joseph (@joshuajosephmd) October 27, 2019

Frightening: "Black patients who were ranked by the algorithm as equally as in need of extra care as white patients were much sicker." https://t.co/lckheHOjeR

— Blake Montgomery ? (@blakersdozen) October 24, 2019

Suchi Saria, a machine learning and health-care expert at @JohnsHopkins, said the study was fascinating because it showed how, once a bias is detected, it can be corrected. https://t.co/VL1F9xBToB

— Hopkins Nursing (@JHUNursing) October 26, 2019

Here's the study in Science. Researchers note that the algorithm predicts costs, not illness....and "unequal access to care means that we spend less money caring for Black patients than for White patients." https://t.co/Pv4IvZBYf7 https://t.co/VjJ8ga01lB

— Mohana Ravindranath (@ravindranize) October 24, 2019

❝Dissecting racial bias in an algorithm used to manage the health of populations❞ If the algorithm learns to predict health needs from examples of health costs + less money spent on black patients, then algorithm underpredicts needs of black patients https://t.co/TVN0wyHxZi pic.twitter.com/IM8SIp8n1S

— Jean-François Bonnefon (@JFBonnefon) October 25, 2019

"Cost is a reasonable proxy for health, but it’s a biased one & that choice is actually what introduces bias into the algorithm" --our affiliate @oziadias on consequences of algorithmic bias (+ potential fixes) in health system https://t.co/KQTjEKmHWd via @verge

— ideas42 (@ideas42) October 25, 2019

'Algorithm makes black patients substantially less likely than their white counterparts to receive important medical treatment. The major flaw affects millions of patients, and was just revealed in research published this week in the journal Science.'https://t.co/b1oiURgTGI

— Kobi Leins (@Kobotic) October 25, 2019

Racial bis in health care algorithms - one of the many reasons for interdisciplinary research to place data analytics in context and to understand pre-existing inequities https://t.co/MqCnyQdkxz

— Jaclyn Piatak (@JaclynPiatak) October 25, 2019

A health care algorithm affecting millions is biased against black patients https://t.co/JPfxqZCg7y

— Matteo G.P. Flora (@lastknight) October 27, 2019

#AI Bias: To predict who benefits from extra healthcare assistance, an algorithm uses costs to treat patients. Bc of unequal access, black patients have less $ spent than white patients. Fixing this increases extra care to black patients from 17.7% to 46.5%https://t.co/YAedwyNdbx

— Amy Diehl, PhD (@amydiehl) October 24, 2019

People who deny or minimize the role poverty/class plays in racial inequality are literally killing us.https://t.co/m8vIPJ81Bw pic.twitter.com/NO8HJKQ63W

— Briahna Joy Gray (@briebriejoy) October 24, 2019

Algorithms are not just "mathematics," but rather include decisions on how to prioritize or weight certain measurements, and use proxy measures to gain insight about what they are trying to... https://t.co/q2fMSpWD78

— AssocForWomenInMath (@AWMmath) October 25, 2019

Your daily reminder that algorithms aren't magical, they're mostly blackboxed human biases https://t.co/7ZZzsYn83J

— Philippe M. Frowd (@PhilippeMFrowd) October 25, 2019

#ADOS #ADOSPolitics
We need #Reparations for this. #Tangibles2020
A health care algorithm affecting millions is biased against black patients https://t.co/oSstuSe65w

— Bill Dixon (@BillDix15525176) October 25, 2019

Several, understandably, outraged friends have sent this to me. I've said it before & I'll say it again, if we don't make a conscious effort to train our algorithms against the current gaps in care, AI is only going to make them worse!!! https://t.co/n9JGAMZWKn via

— Lea Martin (@DocLeaMaria) October 25, 2019

A health care algorithm affecting millions is biased against black patients https://t.co/JY1M8sodst pic.twitter.com/YxaFtqSQ94

— Dr. David L. Katz (@DrDavidKatz) October 25, 2019

A health care algorithm affecting millions is biased against black patients https://t.co/L7jlPy732G

— Ehsan Zaffar (@Ezaffar) October 24, 2019

A medical algorithm used to determine level of care for high-risk patients was biased against black patients. This was because it relied on level-of-spending to determine care needs, which is unfairly skewed due to level of access to care https://t.co/BiwkmryABK

— ? ???? ????? ? (@royaltheartist) October 25, 2019

A health care algorithm affecting millions is biased against black patients https://t.co/WWKafp0k7X >>> https://t.co/HTA6IsoHwv #digitalhealth #socialmedia #digitalmarketing #IoT #industry40 #AI #healthtech #mhealth pic.twitter.com/1oVOOCInbt

— Dr Timos Papagatsias (@_timos_) October 24, 2019

Devastating: "Currently, 17.7 percent of black patients receive the additional attention, the researchers found. If the disparity was remedied, that number would skyrocket to 46.5 percent of patients." https://t.co/zU96ZYUQc6

— nilay patel (@reckless) October 24, 2019

'A health care algorithm affecting millions is biased against black patients' https://t.co/DsMuapWjPH

— Dr Wonhyuk Cho (@WonhyukCho) October 25, 2019

There's so much wrong with this.

"To make that prediction, the algorithm relies on data about how much it costs a care provider to treat a patient. In theory, this could act as a substitute for how sick a patient is."https://t.co/C4SoL1fV0A

— Binal Patel (@binalkp91) October 24, 2019

A health care algorithm affecting millions is biased against black patients - The Verge #AI ⁦@PachlUpa⁩ ⁦@jemalinina⁩ https://t.co/j8lqnGkcoy

— David Martin Ruiz (@damartinruiz) October 25, 2019

This is terrible, and worst of all I can't help but imagine it's increasingly commonplace, too. https://t.co/2G5NzbZtec

— Charlotte Jee (@charlottejee) October 25, 2019

A biased medical algorithm favored white people for health-care programs https://t.co/RozHZohEWr pic.twitter.com/oT3cf34X8t

— Rich Tehrani (@rtehrani) October 25, 2019

New York’s Financial Services Department is investigating whether an algorithm sold by UnitedHealth violates state antidiscrimination law, ⁦@_melaevans⁩ and ⁦@annawmathews⁩ report https://t.co/VoBD32EVmV

— Jonathan Rockoff (@jonathanrockoff) October 26, 2019

#ICYMI: Thank you @_melaevans for reporting on this important issue. https://t.co/tEMvc2M8eG via @WSJ
CC: @HealthNYGov

— Linda Lacewell (@LindaLacewell) October 26, 2019

"New York will not allow racial bias, especially where it results in discriminatory effects that could mean the difference between life and death for an individual patient and the overall health of an already-underserved community."https://t.co/9mJQo7LSZr

— Gareth Rhodes (@GarethRhodes) October 26, 2019

Oh? Not only is racism embedded in the bodies of our patients (and the systems we all live in), medicine itself as a field continually causes racial harm. In so many ways.

Exhibit 876234 https://t.co/WPAGPbKUOR https://t.co/0RZQLKh1FH

— (Student) doctor non-trad (@doctornontrad) October 24, 2019

Millions of Black people affected by racial bias in health-care algorithms https://t.co/Gkganoyxg4

— Mark Anthony Neal (@NewBlackMan) October 26, 2019

The downside of artificial intelligence in health care: Millions of black people may not receive appropriate personalized care due to racial bias in machine learning algorithms. @heidiledford writes in @NatureNews https://t.co/KhLHsXMTP4 pic.twitter.com/rFrTHa1xur

— Muin J. Khoury (@MuinJKhoury) October 24, 2019

"An algorithm widely used in US hospitals to allocate health care to patients has been systematically discriminating against black people, a sweeping analysis has found." Good summary by @NatureNews of an important new study. https://t.co/XBhfCc58Es

— David Gratzer (@DavidGratzer) October 27, 2019

Racism in, racism out in #ai algorithms perpetuating #healthdisparities @DrDesmondPatton https://t.co/ypXwMIdVY2

— Adrian Aguilera (@draguilera) October 25, 2019

"The study, published in Science on 24 October, concluded that the algorithm was less likely to refer black people than white people who were equally sick to programmes that aim to improve care for patients with complex medical needs"https://t.co/54RLeyuDuN

— Black in AI (@black_in_ai) October 26, 2019

Millions of black people affected by racial bias in health-care algorithms #NewJimCode https://t.co/DquKdkL7jp

— Du Boisian Biologist (@Hood_Biologist) October 27, 2019

Matter of grave concern - requires collaboration between policy makers, programmers, scientists and clinicians.

Millions of black people affected by racial bias in health-care algorithms.#ArtificialIntelligence https://t.co/HfEfS4vLc2

— Dr Sonu Bhaskar (@DrSonuBhaskar) October 27, 2019

A health care algorithm affecting millions is biased against black patients https://t.co/rG8T5vwLeD #ai

— Shocklee ⚡️ (@Shocklee) October 27, 2019

Either we start writing these algorithms or we will be the victims of them https://t.co/U3llWgi8vt

— Shawn ??‍? (@sdotstx) October 27, 2019

“An algorithm that many US health providers use to predict which patients will need extra medical care privileged white patients over black patients. Effectively, it bumped whites up the queue for special treatments for kidney problems or diabetes.” https://t.co/uWs2oPqtM6

— katharine jarmul (@kjam) October 28, 2019

Bias is a real concern in AI. Need for a framework to ensure that any AI solution rolled out for public use is tested for 7 parameters

1. Diversity
2. Equity & Fairness
3. Ethics & human values
4. Privacy & Data protection
5. Misu…https://t.co/ejgV2CIFP5 https://t.co/Dobrilo46i

— Santosh Misra (@misra_ias) October 28, 2019

#HealthIT #HealthTech #MedTech #mHealth

? #EthnicGroup ( #race ) #Bias in a #Healthcare #MachineLearning #Algorithm and #Model Disproportionately Hurts African American #Patients and puts their Lives in Danger.

?️https://t.co/m00iQyjmrV@HubBucket @HubBucketArrow @HubBaseDB pic.twitter.com/sEQgGsA3pe

— HubBucket | Science and Technology R&D (@HubBucket) October 28, 2019

Millions of black people affected by racial bias in health-care algorithms https://t.co/RM6eEPdBbj

— Madhu Pai (@paimadhu) October 28, 2019

Algorithm widely used in the USA to screen patients is less likely to refer black people than white people who are equally sick for treatment: https://t.co/Diq1JgCh0V https://t.co/jXZuzX0eKQ

— Kenan Malik (@kenanmalik) October 28, 2019

"An algorithm widely used in US hospitals to allocate health care to patients has been systematically discriminating against Black people. The algorithm was less likely to refer Black people than white people equally sick to programmes that improve care." https://t.co/aWvyHCmgdP

— ? Spectre Outrider ? (@ClaireShrugged) October 25, 2019

The algorithm, used by hospitals and insurers in care decision affecting more than 200 million americans, was less likely to refer black people than white people who were equally sick to care programs.
https://t.co/xlybhEC9hq

— Rita Vassena (@RitaVassena) October 25, 2019