實體辨識是讓機器了解自然語言的關鍵任務，目標是擷取文本中具有特定意義的文字區塊，例如組織名、人名、地名、時間等等。現今，深度學習模型透過捕捉文本的上下文特徵，可以自動找出文本中的實體，在自然語言的領域中，獲得很大的進步，然而，深度學習複雜的模型結構，導致專家難以理解模型決策背後的原因，形成黑盒子的問題，限制研究人員研究及改進模型的能力。

相較於英文，中文文本 (或其他語言，如韓語和日語) 詞與詞之間沒有有明確的邊界，執行中文自然語言的任務前，需要經過基本的語言任務，例如斷詞和詞性標註，以利分辨文本中的詞義，才能進行其他的自然語言任務的預測。因此，中文實體辨識除了存在深度學習模型黑盒子的問題，斷詞與詞性標註的正確性也影響預測結果，導致分析中文實體辨識的決策，變得更加複雜。

我們提出視覺化的分析工具，讓使用者可以透過互動的方式檢查斷詞、詞性與實體辨識預測之間的關係，並探索深度學習模型的內部決策過程。此外，我們的系統允許使用者載入大量的測試資料，並協助使用者從大量的數據中探索，找出關鍵實例進行追蹤及分析，且透過實例證明我們系統可用性及有效性。

Named entity recognition (NER) is a crucial initial task that identifies both spans and types of named entities to extract the specific information, such as organization, person, location, time, etc. Nowadays, the NER task achieves state-of-the-art performance by deep learning approaches for capturing contextual features. However, the complex structures of deep learning make a black-box problem and limit researchers' ability to improve it. Unlike the Latin alphabet, Chinese (or other languages such as Korean and Japanese) do not have an explicit word boundary. Therefore, some preliminary works, such as word segmentation (WS) and part-of-speech tagging (POS), are needed before the Chinese NER task. The correctness of preliminary works importantly influences the final NER prediction. Thus, investigating the model behavior of the Chinese NER task becomes more complicated and challenging. In this paper, we present CNERVis, a visual analysis tool that allows users to interactively inspect the WS-POS-NER pipeline and understand how and why a NER prediction is made. Also, CNERVis allows users to load the numerous testing data and explores the critical instances to facilitate the analysis from large datasets. Our tool's usability and effectiveness are demonstrated through case studies.

[1] MartinAbadi,PaulBarham,JianminChen,ZhifengChen,AndyDavis,Jeffrey Dean,MatthieuDevin,SanjayGhemawat,GeoreyIrving,MichaelIsard, et al.Tensor ow:Asystemforlarge-scalemachinelearning.In 12th USENIX SymposiumonOperatingSystemsDesignandImplementation, pages265{283, 2016.
[2] JasonPCChiuandEricNichols.Namedentityrecognitionwithbidirectional lstm-cnns. TransactionsoftheAssociationforComputationalLinguistics, 4:357{ 370, 2016.
[3] KyunghyunCho,BartVanMerrienboer,CaglarGulcehre,DzmitryBahdanau, FethiBougares,HolgerSchwenk,andYoshuaBengio.Learningphraserepresentationsusingrnnencoder-decoderforstatisticalmachinetranslation. arXiv preprintarXiv:1406.1078, 2014.
[4] JianjingCui,JunLong,ErxueMin,andYugangMao.Wedl-nids:improving networkintrusiondetectionusingwordembedding-baseddeeplearningmethod. In International ConferenceonModelingDecisionsforArticialIntelligence, pages 283{295.Springer,2018.
[5] JacobDevlin,Ming-WeiChang,KentonLee,andKristinaToutanova.Bert:Pretraining ofdeepbidirectionaltransformersforlanguageunderstanding. arXiv preprintarXiv:1810.04805, 2018.
[6] AlexEndert,WilliamRibarsky,CagatayTurkay,BLWilliamWong,IanNabney, I DazBlanco,andFabriceRossi.Thestateoftheartinintegratingmachine learning intovisualanalytics.In Computer GraphicsForum, volume36,pages 458{486. WileyOnlineLibrary,2017.
[7] KawinEthayarajh.Howcontextualarecontextualizedwordrepresentations? comparing thegeometryofbert,elmo,andgpt-2embeddings. arXiv preprint arXiv:1909.00512, 2019. 39
[8] FrancescoGargiulo,StefanoSilvestri,MarioCiampi,andGiuseppeDePietro. Deep neuralnetworkforhierarchicalextrememulti-labeltextclassication. AppliedSoftComputing, 79:125{138,2019.
[9] FelixAGers,JurgenSchmidhuber,andFredCummins.Learningtoforget: Continualpredictionwithlstm. Neuralcomputation, 12(10):2451{2471,2000.
[10] DanGillick,NevenaLazic,KuzmanGanchev,JesseKirchner,andDavid Huynh. Context-dependentne-grainedentitytypetagging. arXiv preprint arXiv:1412.1820, 2014.
[11] MiguelGrinberg. Flask webdevelopment:developingwebapplicationswith python. "O'ReillyMedia,Inc.",2018.
[12] EduardHovy,MitchMarcus,MarthaPalmer,LanceRamshaw,andRalph Weischedel.Ontonotes:the90%solution.In Proceedingsofthehumanlanguage technologyconferenceoftheNAACL,CompanionVolume:ShortPapers, pages 57{60,2006.
[13] YAJoarder,KhMustazurRahman,andFabihaFaizMahi.Upliftedtissue characterizationandclassicationoffattyliverdiseasefromultrasoundimages. AdvancementinImageProcessingandPatternRecognition, 3(3),2020.
[14] AndrejKarpathy,JustinJohnson,andLiFei-Fei.Visualizingandunderstanding recurrentnetworks. arXiv preprintarXiv:1506.02078, 2015.
[15] ZeynepHKilimciandSelimAkyokus.Deeplearning-andwordembedding-based heterogeneous classierensemblesfortextclassication. Complexity, 2018,2018.
[16] GuanLi,JunpengWang,Han-WeiShen,KaixinChen,GuihuaShan,and ZhonghuaLu.Cnnpruner:Pruningconvolutionalneuralnetworkswithvisual analytics. IEEE TransactionsonVisualizationandComputerGraphics, 2020.
[17] JiweiLi,XinleiChen,EduardHovy,andDanJurafsky.Visualizingandunderstanding neuralmodelsinnlp. arXiv preprintarXiv:1506.01066, 2015.
[18] Peng-HsuanLi,Tsu-JuiFu,andWei-YunMa.Whyattention?analyzebilstm deciencyanditsremediesinthecaseofner.In ProceedingsoftheAAAI ConferenceonArticialIntelligence, volume34,pages8236{8244,2020.
[19] ShixiaLiu,XitingWang,MengchenLiu,andJunZhu.Towardsbetteranalysis of machinelearningmodels:Avisualanalyticsperspective. Visual Informatics, 1(1):48{56, 2017. 40
[20] LelandMcInnes,JohnHealy,andJamesMelville.Umap:Uniformmanifold approximationandprojectionfordimensionreduction. arXiv preprint arXiv:1802.03426, 2018.
[21] YaoMing,ShaozuCao,RuixiangZhang,ZhenLi,YuanzheChen,YangqiuSong, and HuaminQu.Understandinghiddenmemoriesofrecurrentneuralnetworks. In 2017 IEEEConferenceonVisualAnalyticsScienceandTechnology(VAST), pages 13{24.IEEE,2017.
[22] Thai-HoangPhamandPhuongLe-Hong.End-to-endrecurrentneuralnetwork modelsforvietnamesenamedentityrecognition:Word-levelvs.character-level. In International ConferenceofthePacicAssociationforComputationalLinguistics, pages219{232.Springer,2017.
[23] FrederickReiss,HongXu,BryanCutler,KarthikMuthuraman,andZachary Eichenberger.Identifyingincorrectlabelsintheconll-2003corpus.In Proceedings of the24thConferenceonComputationalNaturalLanguageLearning, pages215{ 226, 2020.
[24] ClaudeEShannon.Amathematicaltheoryofcommunication. The Bellsystem technicaljournal, 27(3):379{423,1948.
[25] BenShneiderman.Theeyeshaveit:Ataskbydatatypetaxonomyforinformation visualizations.In The craftofinformationvisualization, pages364{371. Elsevier, 2003.
[26] HendrikStrobelt,SebastianGehrmann,MichaelBehrisch,AdamPerer, HanspeterPster,andAlexanderMRush.Seq2seq-vis:Avisualdebugging toolforsequence-to-sequencemodels. IEEE transactionsonvisualization and computergraphics, 25(1):353{363,2018.
[27] HendrikStrobelt,SebastianGehrmann,HanspeterPster,andAlexanderM Rush. Lstmvis:Atoolforvisualanalysisofhiddenstatedynamicsinrecurrent neural networks. IEEE transactionsonvisualizationandcomputergraphics, 24(1):667{676, 2017.
[28] ZihanWang,JingboShang,LiyuanLiu,LihaoLu,JiachengLiu,andJiawei Han. Crossweigh:Trainingnamedentitytaggerfromimperfectannotations. arXiv preprintarXiv:1909.01441, 2019.
[29] XueXia,ThaddeusRoppel,JohnYHung,JianZhang,SenthilkumarCGPeriaswamy,andJustinPatton.Environmentalcomplexitymeasurementusing shannon entropy.In 2020 SoutheastCon, pages1{6.IEEE,2020. 41
[30] VikasYadavandStevenBethard.Asurveyonrecentadvancesinnamedentity recognition fromdeeplearningmodels. arXiv preprintarXiv:1910.11470, 2019.
[31] MatthewDZeilerandRobFergus.Visualizingandunderstandingconvolutional networks.In Europeanconferenceoncomputervision, pages818{833.Springer, 2014.
[32] YueZhangandJieYang.Chinesenerusinglatticelstm. arXiv preprint arXiv:1805.02023, 2018.
[33] YingZhao,FengLuo,MinghuiChen,YingchaoWang,JiazhiXia,Fangfang Zhou, YunhaiWang,YiChen,andWeiChen.Evaluatingmulti-dimensionalvisualizations forunderstandingfuzzyclusters. IEEE transactionsonvisualization and computergraphics, 25(1):12{21,2018.
[34] YuyingZhu,GuoxinWang,andBorjeFKarlsson.Can-ner:Convolutional attentionnetworkforchinesenamedentityrecognition. arXiv preprint arXiv:1904.02141, 2019.