9115/24, 10:27AM —_ GitHub -PadelePadle/ERNIE: Official implementations for various prersining models of ERNIE-faly covering tops of Langu Bi PaddlePaddle / ERNIE Pui: Coicial implementations for various pre-traning models of ERNIE-family, covering topics of Language Understanding & Generation, Multimodal Understanding & Generation, and beyond fr GSksuis Y 1Skfons PF Branches © Tays-Ar Acoty sur © Code © tssves Th Pullrequests © Discussions © Actions PF emieckiopen-vid ~ YG 8rancher OB ta PO @ Vocorine Updte README situ Be mete Research 1 spokeations ert D sitgnore L. pre-commit-conigyant D ReAoMen README ENV 1D eaoMe sockemd README: htps:igthub.com/PadélePadale/ERNIE Update issue tempts (#753) peste README ma pate dta.20997 Dyaraph e436) update to pasele 20 Update README md Upaate README ENV.na sc ing 2 notations © Security LX Insights Goto fie entes-beryer ® 2 years 390 years 390 4 yeas 290 3 years 290 yeas 290 yeas 290 yeas 290 wn9115/24, 10:27AM GitHub -PadelePadle/ERNIE: Official implementations for various prersining models of ERNIE-Iamly covering tops of Langu © ERNIE Milestones a) O22. O a=. ee | ee hae Spe Uaeretntee! Won the CAAT Award LOAN ESRI, TART NPLATE ARM, 201983, FRET RAE ARR O ERNIE10, RTE SSAA RAMSAR, HRVGNTIRSTIOT ROUEN, ARMA SULA, 2M: RNAS CTE Eepro DL, MUMIA TROT AMSA MUTRERNERATTR, FP, twH LcasyDL. BML sam, [7 MES) FFifRoadmap + 2022818: © BISLEBSRSHAVIERERE, cevrevit 2.0 (are) JESCHEIE + 2022520: © SBSFFFIRERNIE 3 OF FURAN? 1 TOMASHRRREERNIE 3.0 Base = 200M HS ESEFRIAIERNIE 3.0 XBase = TAMPERRSLIEFRIREZERNIE 3.0 Mecium ‘Hie ESR MEOMMIERNIE SAT It SPSBERNIE-Gen (4252) RVI, SEAR TRCEARE: PRMOREAREE, TRNCEOR, NE, (oH EMEA OERNICTTAREYE: AF CRADORUNR, SISTLOERNICRRIRT SARI, CORASELEMRUIUG TENA, (SB — NRT A, AUSTRIAN, © SeCRAILE, RRUNGMEE, RISRHNER, HARIIPSS, HeRURSENLeTTARRABIMTE, © SRIYNLPMFESS: SCARAB, SCARCE, ARUN, (BIRR, SER, RTE, © SAORI, RUBIA, ial SUI, JUNE RURILA, # 2021-123: © SiEVISIR! concen TESLA + 2021520: © ERNIE SR FFIADUA SU IE © SPERAIOREE core-oron ETE = HERSCARRARERIERL ewwre-ooc TESTI = AS AMRIT ASRS ARID eance-van TESST = BSS AOORUIERE ence-uno JECT + 2020824 © eMe-viL SORAES! GGEAD htps:igthub.com/PadélePadale/ERNIE ar9115/24, 10:27AM —_ GitHub -PadelePadle/ERNIE: Official implementations for various prersining models of ERNIE-Iaily covering tops of Langu + HORROR TIME, EXER RS TRUINNS) orem, + ARGRECOM, ORT OK RENE, Mee LNT, + DRS WEARER, UNMIS, + 2020520 © ERVTE-cEN BRESCIA = BEARERS, THEA vear-2070 BOR, = ARE ERNIE HUGO REAGAN, STR FRE, = COMTEROET PRIEIRAONTATIAN (ELA baseslarge/anne=4206) , 1 PVRTEIRUIESI ARMA span-by-span SESRES5, LAULSHRRR SEN —NELRBH, 1 LUCE AL EIR RRL SANTOR EROS, 1 RISHY Mult-Flow Attention SVEN, + 2020420 ARAHCRNIESo9e, —APRTHRRATSRIEHENY, SRARERNIEHEagoreagtor. HPCLSA, + 2020327 EESemtvalP0205 NFA be, + 2019.12.26 cLUEW— ee, + 2019.116 #AFERNEE Te + 2019.77 MAsEANIE 20, + 2019.3.16 ARABLRNIE 10, RRR 1 eR: Tae 2. SeREMieRHE S ait clone htps://gtthub.con/Padderaddle/ERM. pit PRRLE: GASTOERNIEAR RTI + SRARERNIES OfFPNSRUSGAREY, ERS LYFE: o Regine fe © RRRLUIEBjson © unenae © ReREEaIjsons © abe + BUNA REIS ND), SRL FERNIEA BRAN S Fea + HeRERNIES OTMUIMRIRMR ASCARI HEESS 1 ERNMES ORAS PSE ernie 3.0 BEF e 8 Be Aroseis noes applications nodes rub 2 ere eR + OS MESA RTOS AINE, ET, TERETE. + SORE HONE htps:igthub.com/PadélePadale/ERNIE an9115/24, 10:27AM —_ GitHub -PadelePadle/ERNIE: Official implementations for various prersining models of ERNIE-Iaily covering tops of Langu DBASE RH 6 «/applseations/ tases PRELASREEARRER 1 fase o + TRS, ERIN, REBURison se + FAPAMIsonSRH47E /oxamples/ERRER, {RFEIERNIES OFLU RUE FT IMBEOMR ESL! ISH) /examples/clserie f.chjson, TERJsonSCHAEPRS 28, OD, TSRRS TER, RT CRIES. oI. UMA AAT RESON e at Jexanples/els_ernte_fe_eh-3son aap + ORRWSRETERE, FHRRMRITKls_erie fe chjson, SADAFILLNRRMUUURASIES, + ich, BLRIRS I pytron run trainer.py , SITRAER, SEARS Femieieh ek ARNE LIA aHoH a, # emie seascape @ 1 EF /sonsCRAMMLGU, RUAITIERLL. /exanples/cis_ernie_fc_ch-3son python run_srainer.py ~paran_path «/evanples/cls_ernie_fe_ch- 301 + Sein ‘Alectoun —-poussnsy run_trainer. py. exanples/els_eonde_fe_ch. on o + UIMRBEFRO RES RNR /logitestlogt SCH, + IME a RRS SOMES RRNRAPTE /outputlEINTR, Hisave_inference.model/ SCHR it AF AMIR, save_checkpoint/ XHREARE ET INSEE, RBRMjsonxcee + RAB sonSRHHZE /eramples/ERRER, {RFRERNIE2 ORTUESAAMY IS COHRNWIY 5 RUMOR INE /examples/clsernie_fchnferjson + EREHVAL /examples/ls_emie fe ch inferjson FHORUREIOMA RE, HUAI ARE. FARUGIROOARIRNS, ISR PRAN ‘ o er" igh: ("data_path":"-/6ata/precict sata"), Snfererce” ("inference rodel path”: /outpit/els_ernie_fe_ch/save_snference roel /Snfarence_ step 751", output_path"? * Jou “y > aa run inferpy , RRENAASAMRRM HRT]. SRA + BRISA ARE /outpul/predict_resultcxHep, FURR * SHRUG RURE MRNA htps:igthub.com/PadélePadale/ERNIE ar9115/24, 10:27AM —_ GitHub -PadelePadle/ERNIE: Official implementations for various prersining models of ERNIE-Iaily covering tops of Langu + SRA F&: 2 /applicaions/models hubEIRTR, FER, BAINBMETRER e 4 /applications/nodels, nub PRES. base + BSF, MResearcn ot od CLUES ule ont sia NERA BERRA + SHOOTERURAACLUERLEEA, CLUE FRU: i a CLUEWSC2020IFLYTEK TNEWS §AFQMC =CMNU CSL) OCNLI SSG 24.1024 ROBERTo-exlarge 9079 e202 $9.33 760063888367 7081 71636, 20L1024H ERNIE 30-xBase gnae 6222 6037595 84988427 SOT TTA 12LT6BH | ROBERTa-wowm-ext-base 855 e122 seus 74751617728 12U762H ERNIE 30-Base 818 sore 58737553365 8330 DSI 75.63 6L768H — RAT6, Chinese 7500 5968 $662 73S 7926004 73S 7099 ©L766H —-ERNIE30-Medium 7993, cork 5716 7456 60878123. 7702 72.9, 1 CLEARS ESSISF Grid Search 75: SSN, SIAEITIGATIIN 100 steps FHEMEMAUR, MUIR RAMRLURYE PoaethE 2. SYKCESF Grd Search REEPEEE batch siz’ 16, 32,64 learning rates: Te-5, 26-5, 30-5, Se-5; AY9 CLUEWSC2020 RARAEAR/), AUBLRTE RAL LAORHERT batch size ARBUAR, FELURY CLUEWSC2020 SEJURTERSMEEMIT batch size = 8 AORRESHER; PRISCLUEWSC2020 A TFLYTEK BIGRSERY dropout MARIEARUGRUNL, FFLLRY CLUEWSC2020 X] IFLYTEX BRIGSEFAUindropout prob = 0.0 ANARSHDR. TiS hheesre ‘SOROS: TASK. |AFOMC NEWS IFLYTEK CMINLI OCNLI CLUEWSC2020 CSL epoch 3 3 3 2 5 so 5 maxseatensth 128 28 82818 1B 256 vwarmup_proportion 0:1 04 or ont 01 htps:igthub.com/PadélePadale/ERNIE 379115/24, 10:27AM —_ GitHub -PadelePadle/ERNIE: Official implementations for various prersining models of ERNIE-Iaily covering tops of Langu ERNIES@@Grid Search MHS Model FMC ‘TNEWS ryTEK cM cn cLuEWsc2020 cst ERNE Aa tee Giese. | BRIE! 30. oe Ce BROS osjosz 64 Ih2e Medium os Lire sz 32 Ir bapinzeos Be sz. 8.20 05idrop_out oy ERNIE ber 64 Ide bez 16 hse 52 16.in2e 30-Base 05 0s os betehzeos 9 bez 16 In 0s NIE 30 XBase RGR Sep (LAD) SRR (ATE) bez 16Ie2e- be 16 h3e- 5 Bae 32 Ine e e betehteos 9 bse. 6 20-05 eexoene (oe) saat (CURT) TA (Lae) SCBKS FA ERNIE 1.0 areicte{sunzeisernte, o ‘itles(éenie: Emhanced representation through knowledge integration), SJournal=(ankiv preprint apxiv3904.09225), year=(2039) ) ERNIE 2.0, inproceedings (sunzezderaie, e titie=(crnle 2.0: A continual pre-tratning ¢ranewore for language understanding}, suthor=(Son, Yu and kong, Shuohuan ane (4, Yin and Feng, Shikon and Tian, #90 ané hu, Hus and Kang, nfene), booktitles(Poceecings of the ANAT Conference on Artificial Intelligence), ERNIE-GEN farcicie(xtaozeneernie, o ‘Uities{Ennle-gens An emrarced multi-taw pre-tratning and fine-tuntng framework for natural language generation}, htps:igthub.com/PadélePadale/ERNIE er9118/24, 10:27AM —_ GitHub -PadelePadle/ERNIE: Official implementations for various prersning models of ERNIE-Ialy, covering tops of Langu author=(Xia0, Oongling and Zhang, Han and Li, YuKun and Sun, Yu ané Tian, Hao and wy Ha and Wangs Haifene), Sournale(anciv preprint sexsys290s.1314), vyear=(202e) > ERNIE-ViL eseticie(yureavernse, e ‘ities(tenie-vil: knowledge enhanced viston-2anguage representations through scene 8r3ph), futhor=(Vu, Fel and Tangy 24ft and Vin, Wetchong and Sin, Yu and Tian, #20 and Way Hun and Mang, Halong), ERNIE-Gram areiere(xtsoz022ernie, e ‘titlee{RNIE-Gran: Pre-Training with GxLicithy N-cran Masked Language Modeling for Natural Language Understanding, suthor=(Xiao, Dangling and Li, Westin and Zang, Non and Sun, Yu and Than, Mao and Wu, Ha and Wang, HoLferg), $Journale(arxiv preprint apkivi2ei9, 124M yeara{202e) Releases © ERNIE aygrpah Packages Contributors 34 SOS eC ONELOR TOs 220 contrtor Languages htps:igthub.com/PadélePadale/ERNIE 7