Interpretation of the Outputs of Deep Learning Model Trained with Skin Cancer Dataset
2018.06.02 16:48
Our manuscript, "Interpretation of the Outputs of Deep Learning Model Trained with Skin Cancer Dataset" was published as a letter article in the Journal of Investigative Dermatology today (https://www.jidonline.org/article/S0022-202X(18)31992-4/fulltext).
When we train a CNN model, we somtimes get a disappointing Top-1 accuracy. I also suffered this problem and I did not understand exactly what was wrong at that time. When my early version of the 12DX paper was reviewed in JAMA dermatology 2 years ago, the biggest reason for rejection was the low Top-1 accuracy.
However, unlike general object recognition studies, it is very difficult to determine medical research results with Top-1 accuracy, and it is important that the AUC can be high even with a low Top-1 accuracy. If you look carefully, most of medical AI researches have used AUC rather than Top-(n) accuracy.
Because of small and imbalanced training data in medical researches, the analysis of each class as Top-(n) accuracy is inadequate (but the mean Top-(n) of all classes is meaningful). Top-(n) accuracy of each classes vary whenever we repeat the training of CNN with imbalanced dataset. Therefore, we should see the corrected value while using thresholds of each classes, that is ROC curve.
With the AUC results, we published "Classification of the Clinical Images for Benign and Malignant Cutaneous Tumors Using a Deep Learning Algorithm" (https://www.jidonline.org/article/S0022-202X(18)30111-8/fulltext)
There was a debate that my 12DX algorithm is not sensitive (low top-1 accuracy) with the ISIC dataset (Automated Dermatological Diagnosis: Hype or Reality?; https://www.jidonline.org/article/S0022-202X(18)31991-2/fulltext).
There was an additional problem as well as the Top accuracy problem.
When we analyze a clinical image, "the problem of judging whether it is melanoma or not" is easier than "the problem of matching the type of cancer".
Analyzing the output of the AI (CNN) model is equivalent to "the problem of matching the type of cancer", and analyzing the ratio of output is proper if we want to analyze the problem of judging "whether cancer or not".
We interpreted the ratio of melanoma output and nevus output rather than using melanoma output alone.
RATIO (Melanoma Index) = melanoma output / (melanoma output + nevus output).
The clinical image of skin cancer consists of a nodular lesion and a background. If you want to concentrate on only the lesion, we need to analyze it with RATIO as above to get more accurate results.
In the attached photograph, (b) is "matching what cancer is" and (a) is judging "whether it is cancer or not".
We made web-DEMO (http://dx.medicalphoto.org), and we have made it possible to show what conclusions are coming up depending on the Top-5 output and how it is interpreted.
번호 | 제목 | 글쓴이 | 날짜 | 조회 수 |
---|---|---|---|---|
1671 | 음란전화.... | 윤재원 | 2000.07.10 | 3884 |
1670 | 재원님이 퍼온 심리테스트...하하하 읽어봐라 | 재원님~ | 2000.07.12 | 3624 |
1669 | 의욕상실 | 채영광 | 2000.07.13 | 3949 |
1668 | Re: 의욕상실 | 한승석 | 2000.07.13 | 3407 |
1667 | Re: 사진 더 추가해줘잉 | 한승석 | 2000.07.13 | 3215 |
1666 | 비천무는 잼있다! | 채영광 | 2000.07.16 | 4091 |
1665 | 음악 니가 다 정리한 거야? | 채영광 | 2000.07.18 | 4489 |
1664 | 아주아주 멋진 시...한편..꼭 읽어봐랑~ | 재원님~ | 2000.07.18 | 3653 |
1663 | 승석아 오늘 오스키는 잘 보았니? | 채영광 | 2000.07.23 | 3605 |
1662 | Re: 비천무는 잼없다! | 헬로우멤 | 2000.07.24 | 3460 |
1661 | 비천무...말이지~~~ | 윤재원 | 2000.07.26 | 3619 |
1660 | 승석아 다다음주 말이야. | 채영광 | 2000.07.31 | 3468 |
1659 | 똑똑똑~~ | 조각가 | 2000.08.01 | 3641 |
1658 | 승석이오빠아...ㅜ.ㅜ 설가면 맛난거 사주우.ㅠ.ㅠ | 김승욱 | 2000.08.06 | 4337 |
1657 | 승석 X-ray는 많이 찍었니? | 정석원 | 2000.08.07 | 3837 |
https://i.imgur.com/jnZUavi.png