Llama-2 Mistral MPT=？融合多個異構大模型顯奇效

機器之心專欄

機器之心編輯部

融合多個異構大語言模型，中山大學、騰訊 AI Lab 推出 FuseLLM

隨著 LLaMA、Mistral 等大語言模型的成功，各傢大廠和初創公司都紛紛創建自己的大語言模型。但從頭訓練新的大語言模型所需要的成本十分高昂，且新舊模型之間可能存在能力的冗餘。

近日，中山大學和騰訊 AI Lab 的研究人員提出了 FuseLLM，用於「融合多個異構大模型」。

不同於以往的模型集成和權重合並，前者需要在推理時同時部署多個大語言模型，後者需要合並模型具備相同的結果，FuseLLM 能夠從多個異構大語言模型中外化知識，將各自的知識和能力通過輕量的持續訓練轉移到一個融合大語言模型中。

該論文剛剛在 arXiv 上發佈就引起了網友的大量關註和轉發。

有人認為，「當想要在另一種語言上訓練模型時，使用這種方法是非常有趣的」，「我一直在思考這件事」。

目前該論文已被 ICLR 2024 接受。

論文標題：Knowledge Fusion of Large Language Models
論文地址：https://arxiv.org/abs/2401.10491
論文倉庫：https://github.com/fanqiwan/FuseLLM

方法介紹

FuseLLM 的關鍵在於從概率分佈表征的角度來探討大語言模型的融合，對於同樣的輸入文本，作者認為由不同大語言模型生成的表征可以反映出它們在理解這些文本時的內在知識。因此，FuseLLM 首先利用多個源大語言模型生成表征，將它們的集體知識和各自優勢外化，然後將生成的多個表征取長補短進行融合，最後經過輕量級的持續訓練遷移到目標大語言模型。下圖展示了 FuseLLM 方法的概述。

考慮到多個異構大語言模型的 tokenizer 以及詞表存在差異，在融合多個表征時，如何對齊分詞結果是一大關鍵: FuseLLM 在 token 級別的完全匹配之上，額外設計了基於最小編輯距離的詞表級別對齊，最大程度地保留了表征中的可用信息。

為了在結合多個大語言模型的集體知識的同時保持其各自的優勢，需要精心設計用於融合模型生成表征的策略。具體而言，FuseLLM 通過計算生成表征和標簽文本之間交叉熵來評估不同大語言模型對這條文本的理解程度，然後引入了兩種基於交叉熵的融合函數：

MinCE: 輸入多個大模型為當前文本生成的表征，輸出交叉熵最小的表征；
AvgCE: 輸入多個大模型為當前文本生成的表征，輸出基於交叉熵獲得的權重加權平均的表征；

在持續訓練階段，FuseLLM 使用融合後的表征作為目標計算融合損失，同時也保留了語言模型損失。最終的損失函數為融合損失和語言模型損失之和。

實驗結果

在實驗部分，作者考慮了一個通用但具有挑戰性的大語言模型融合場景，其中源模型在結構或能力上具備較小的共性。具體來說，其在 7B 規模上進行了實驗，並選擇了三個具有代表性的開源模型：Llama-2、OpenLLaMA，和 MPT 作為待融合的大模型。

作者在通用推理、常識推理、代碼生成、文本生成、指令跟隨等場景評估了 FuseLLM，發現其相較於所有源模型和繼續訓練基線模型取得了顯著的性能提升。

通用推理 & 常識推理

在測試通用推理能力的 Big-Bench Hard Benchmark 上，經過持續訓練後的 Llama-2 CLM 相較於 Llama-2 在 27 個任務上取得了平均 1.86% 的提升，而 FuseLLM 則相較於 Llama-2 取得了 5.16% 的提升，顯著優於 Llama-2 CLM，說明 FuseLLM 能結合多個大語言模型的優勢取得性能提升。

在測試常識推理能力的 Common Sense Benchmark 上，FuseLLM 超過了所有的源模型和基線模型，在所有任務上都取得了最佳的性能。

代碼生成 & 文本生成

在測試代碼生成能力的 MultiPL-E Benchmark 上，FuseLLM 在 10 個任務中，有 9 個超過了 Llama-2，取得了平均 6.36% 的性能提升。而 FuseLLM 沒有超過 MPT 和 OpenLLaMA 的原因可能是由於使用 Llama-2 作為目標大語言模型，其代碼生成能力較弱，且持續訓練語料中的代碼數據比例較低，僅占約 7.59%。

在多個測量知識問答（TrivialQA）、閱讀理解（DROP）、內容分析（LAMBADA）、機器翻譯（IWSLT2017）和定理應用（SciBench）的文本生成 Benchmark 上，FuseLLM 也在所有任務中超過了所有源模型，並在 80% 的任務中超過了 Llama-2 CLM。

指令跟隨

由於 FuseLLM 僅需提取多個源模型的表征進行融合，然後對目標模型持續訓練，因此其也能適用於指令微調大語言模型的融合。在評估指令跟隨能力的 Vicuna Benchmark 上，FuseLLM 同樣取得了出色表現，超過了所有源模型和 CLM。

FuseLLM vs. 知識蒸餾 & 模型集成 & 權重合並

考慮到知識蒸餾也是一種利用表征提升大語言模型性能的方法，作者將 FuseLLM 和用 Llama-2 13B 蒸餾的 Llama-2 KD 進行了比較。結果表明，FuseLLM 通過融合三個具有不同架構的 7B 模型，超過了從單個 13B 模型蒸餾的效果。

為了將 FuseLLM 與現有融合方法進行比較（例如模型集成和權重合並），作者模擬了多個源模型來自相同結構的底座模型，但在不同的語料庫上持續訓練的場景，並測試了各種方法在不同測試基準上的困惑度。可以看到雖然所有的融合技術都可以結合多個源模型的優勢，但 FuseLLM 能達到最低的平均困惑度，表明 FuseLLM 具備能比模型集成和權重合並方法更有效地結合源模型集體知識的潛力。

最後，盡管社區目前已經關註大模型的融合，但目前的做法大多基於權重合並，無法擴展到不同結構、不同規模的模型融合場景。雖然 FuseLLM 隻是一項初步的異構模型融合研究，但考慮到目前技術社區存在大量不同的結構和規模的語言、視覺、音頻和多模態大模型，未來這些異構模型的融合會迸發出怎樣驚人地表現呢？讓我們拭目以待！

.data_color_scheme_dark{--weui-BTN-ACTIVE-MASK: rgba(255, 255, 255, .1)}.data_color_scheme_dark{--weui-BTN-DEFAULT-ACTIVE-BG: rgba(255, 255, 255, .126)}.data_color_scheme_dark{--weui-DIALOG-LINE-COLOR: rgba(255, 255, 255, .1)}.data_color_scheme_dark{--weui-BG-COLOR-ACTIVE: #373737}.data_color_scheme_dark{--weui-BG-6: rgba(255, 255, 255, .1);--weui-ACTIVE-MASK: rgba(255, 255, 255, .1)}.data_color_scheme_dark{--weui-REDORANGE: #ff6146;--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}.data_color_scheme_dark{--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}.data_color_scheme_dark{--weui-elpsColor: rgba(255, 255, 255, .8)}.data_color_scheme_dark{--weui-mask-elpsColor: rgba(255, 255, 255, .8);--weui-mask-gradient: linear-gradient(to right, rgba(25, 25, 25, 0), #191919 40%)}.data_color_scheme_dark{--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}.rich_media_content{color:#000000e5;font-size:17px;font-size:var(--articleFontsize);overflow:hidden;text-align:justify}.rich_media_content{color:#ffffffa6;color:var(--weui-FG-HALF)}.rich_media_content{position:relative;z-index:0}.wxw-img{vertical-align:bottom}.rich_media_content:not(.old_list_style) .list-paddingleft-1,.rich_media_content:not(.old_list_style) .list-paddingleft-2,.rich_media_content:not(.old_list_style) .list-paddingleft-3{padding-left:2.2em}.rich_media_content:not(.old_list_style) .list-paddingleft-1 .list-paddingleft-2,.rich_media_content:not(.old_list_style) .list-paddingleft-2 .list-paddingleft-2,.rich_media_content:not(.old_list_style) .list-paddingleft-3 .list-paddingleft-2{padding-left:30px}.rich_media_content:not(.old_list_style) .list-paddingleft-1{padding-left:1.2em}.rich_media_content:not(.old_list_style).fix_apple_default_style .list-paddingleft-1{padding-left:1.5em}.rich_media_content.old_list_style .list-paddingleft-1,.rich_media_content.old_list_style .list-paddingleft-2,.rich_media_content.old_list_style .list-paddingleft-3{padding-left:0}.rich_media_content.old_list_style .list-paddingleft-1 .list-paddingleft-1,.rich_media_content.old_list_style .list-paddingleft-2 .list-paddingleft-2,.rich_media_content.old_list_style .list-paddingleft-3 .list-paddingleft-3{padding-left:1.2em}@media screen and (min-width:1024px){body:not(.pages_skin_pc) :root{--appmsgPageGap: 20px;--appmsgPageBottomGap: 40px}}:root{--articleFontsize: 17px}:root{--sab: env(safe-area-inset-bottom)}:root{--wxBorderAvatarRatio: 3}:root{--discussPageGap: 20px}:root{--appmsgPageGap: 20px;--appmsgPageBottomGap: 40px}*{margin:0;padding:0}.rich_media_content *{max-width:100%!important;box-sizing:border-box!important;-webkit-box-sizing:border-box!important;word-wrap:break-word!important}body,.wx-root,page{--weui-BTN-HEIGHT: 48;--weui-BTN-HEIGHT-MEDIUM: 40;--weui-BTN-HEIGHT-SMALL: 32}body{--weui-elpsLine: 2;--weui-elpsFontSize: 1rem;--weui-elpsColor: rgba(0, 0, 0, .9)}body{--weui-mask-elpsLine: 2;--weui-mask-elpsLineHeight: 1.4;--weui-mask-elpsFontSize: 1rem;--weui-mask-elpsColor: rgba(0, 0, 0, .9);--weui-mask-gradient: linear-gradient(to right, rgba(255, 255, 255, 0), #ffffff 40%)}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]){--weui-elpsColor: rgba(255, 255, 255, .8)}}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]){--weui-mask-elpsColor: rgba(255, 255, 255, .8);--weui-mask-gradient: linear-gradient(to right, rgba(25, 25, 25, 0), #191919 40%)}}@media screen and (min-width:1024px){body:not(.pages_skin_pc) body,body:not(.pages_skin_pc) .wx-root{--weui-BG-0: #ededed;--weui-BG-1: #f7f7f7;--weui-BG-2: #fff;--weui-BG-3: #f7f7f7;--weui-BG-4: #4c4c4c;--weui-BG-5: #fff;--weui-FG-0: rgba(0, 0, 0, .9);--weui-FG-HALF: rgba(0, 0, 0, .9);--weui-FG-1: rgba(0, 0, 0, .55);--weui-FG-2: rgba(0, 0, 0, .3);--weui-FG-3: rgba(0, 0, 0, .1);--weui-FG-4: rgba(0, 0, 0, .15);--weui-FG-5: rgba(0, 0, 0, .05);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #fa9d3b;--weui-YELLOW: #ffc300;--weui-GREEN: #91d300;--weui-LIGHTGREEN: #95ec69;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1485ee;--weui-PURPLE: #6467f0;--weui-WHITE: #fff;--weui-LINK: #576b95;--weui-TEXTGREEN: #06ae56;--weui-FG: #000;--weui-BG: #fff;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: #fa9d3b;--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: #06ae56;--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: #10aeff;--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(0, 0, 0, .5);--weui-TAG-BACKGROUND-BLACK: rgba(0, 0, 0, .05)}}@media screen and (min-width:1024px){body:not(.pages_skin_pc) .wx-root[data-weui-mode=care],body:not(.pages_skin_pc) body[data-weui-mode=care]{--weui-BG-0: #ededed;--weui-BG-1: #f7f7f7;--weui-BG-2: #fff;--weui-BG-3: #f7f7f7;--weui-BG-4: #4c4c4c;--weui-BG-5: #fff;--weui-FG-0: #000;--weui-FG-HALF: #000;--weui-FG-1: rgba(0, 0, 0, .6);--weui-FG-2: rgba(0, 0, 0, .42);--weui-FG-3: rgba(0, 0, 0, .1);--weui-FG-4: rgba(0, 0, 0, .15);--weui-FG-5: rgba(0, 0, 0, .05);--weui-RED: #dc3636;--weui-ORANGERED: #d14730;--weui-ORANGE: #e17719;--weui-YELLOW: #bb8e00;--weui-GREEN: #4f8400;--weui-LIGHTGREEN: #2e8800;--weui-BRAND: #018942;--weui-BLUE: #007dbb;--weui-INDIGO: #0075e2;--weui-PURPLE: #6265f1;--weui-WHITE: #fff;--weui-LINK: #576b95;--weui-TEXTGREEN: #06ae56;--weui-FG: #000;--weui-BG: #fff;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: #e17719;--weui-TAG-BACKGROUND-ORANGE: rgba(225, 119, 25, .1);--weui-TAG-TEXT-GREEN: #06ae56;--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: #007dbb;--weui-TAG-BACKGROUND-BLUE: rgba(0, 125, 187, .1);--weui-TAG-TEXT-BLACK: rgba(0, 0, 0, .5);--weui-TAG-BACKGROUND-BLACK: rgba(0, 0, 0, .05)}}@media screen and (min-width:1024px){body:not(.pages_skin_pc) .wx-root[data-weui-mode=care][data-weui-theme=dark],body:not(.pages_skin_pc) body[data-weui-mode=care][data-weui-theme=dark]{--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .85);--weui-FG-HALF: rgba(255, 255, 255, .65);--weui-FG-1: rgba(255, 255, 255, .55);--weui-FG-2: rgba(255, 255, 255, .35);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}}@media screen and (min-width:1024px){body:not(.pages_skin_pc) .wx-root,body:not(.pages_skin_pc) body{--appmsgExtra-BG: #F7F7F7}}@media screen and (min-width:1024px){body:not(.pages_skin_pc) .wx-root[data-weui-theme=dark],body:not(.pages_skin_pc) body[data-weui-theme=dark]{--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}}@media screen and (min-width:1024px){body:not(.pages_skin_pc) .wx-root[data-weui-theme=dark],body:not(.pages_skin_pc) body[data-weui-theme=dark]{--appmsgExtra-BG: #121212}}@media screen and (min-width:1024px){body:not(.pages_skin_pc)body:not(.pages_skin_pc){background:#191919;background:var(--weui-BG-2)}}@media screen and (min-width:1024px) and (prefers-color-scheme:nodark){body:not(.pages_skin_pc) .wx-root[data-weui-mode=care]:not([data-weui-theme=light]),body:not(.pages_skin_pc) body[data-weui-mode=care]:not([data-weui-theme=light]){--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .85);--weui-FG-HALF: rgba(255, 255, 255, .65);--weui-FG-1: rgba(255, 255, 255, .55);--weui-FG-2: rgba(255, 255, 255, .35);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}}@media screen and (min-width:1024px) and (prefers-color-scheme:nodark){body:not(.pages_skin_pc) .wx-root:not([data-weui-theme=light]),body:not(.pages_skin_pc) body:not([data-weui-theme=light]){--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}}@media screen and (min-width:1024px) and (prefers-color-scheme:nodark){body:not(.pages_skin_pc) .wx-root:not([data-weui-theme=light]),body:not(.pages_skin_pc) body:not([data-weui-theme=light]){--appmsgExtra-BG: #121212}}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]).my_comment_empty_data{background-color:#111}}body,.wx-root{--weui-BG-6: rgba(0, 0, 0, .05);--weui-ACTIVE-MASK: rgba(0, 0, 0, .05)}body,.wx-root{--weui-FG-1: rgba(0, 0, 0, .5);--weui-REDORANGE: #ff6146;--weui-BG-0: #ededed;--weui-BG-1: #f7f7f7;--weui-BG-2: #fff;--weui-BG-3: #f7f7f7;--weui-BG-4: #4c4c4c;--weui-BG-5: #fff;--weui-FG-0: rgba(0, 0, 0, .9);--weui-FG-HALF: rgba(0, 0, 0, .9);--weui-FG-1: rgba(0, 0, 0, .55);--weui-FG-2: rgba(0, 0, 0, .3);--weui-FG-3: rgba(0, 0, 0, .1);--weui-FG-4: rgba(0, 0, 0, .15);--weui-FG-5: rgba(0, 0, 0, .05);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #fa9d3b;--weui-YELLOW: #ffc300;--weui-GREEN: #91d300;--weui-LIGHTGREEN: #95ec69;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1485ee;--weui-PURPLE: #6467f0;--weui-WHITE: #fff;--weui-LINK: #576b95;--weui-TEXTGREEN: #06ae56;--weui-FG: #000;--weui-BG: #fff;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: #fa9d3b;--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: #06ae56;--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: #10aeff;--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(0, 0, 0, .5);--weui-TAG-BACKGROUND-BLACK: rgba(0, 0, 0, .05)}body,.wx-root{--weui-BG-0: #ededed;--weui-BG-1: #f7f7f7;--weui-BG-2: #fff;--weui-BG-3: #f7f7f7;--weui-BG-4: #4c4c4c;--weui-BG-5: #fff;--weui-FG-0: rgba(0, 0, 0, .9);--weui-FG-HALF: rgba(0, 0, 0, .9);--weui-FG-1: rgba(0, 0, 0, .55);--weui-FG-2: rgba(0, 0, 0, .3);--weui-FG-3: rgba(0, 0, 0, .1);--weui-FG-4: rgba(0, 0, 0, .15);--weui-FG-5: rgba(0, 0, 0, .05);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #fa9d3b;--weui-YELLOW: #ffc300;--weui-GREEN: #91d300;--weui-LIGHTGREEN: #95ec69;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1485ee;--weui-PURPLE: #6467f0;--weui-WHITE: #fff;--weui-LINK: #576b95;--weui-TEXTGREEN: #06ae56;--weui-FG: #000;--weui-BG: #fff;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: #fa9d3b;--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: #06ae56;--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: #10aeff;--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(0, 0, 0, .5);--weui-TAG-BACKGROUND-BLACK: rgba(0, 0, 0, .05)}.wx-root,body{--weui-BTN-ACTIVE-MASK: rgba(0, 0, 0, .1)}.wx-root,body{--weui-BTN-DEFAULT-ACTIVE-BG: #e6e6e6}.wx-root,body{--weui-DIALOG-LINE-COLOR: rgba(0, 0, 0, .1)}.wx-root,body{--weui-BG-COLOR-ACTIVE: #ececec}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-BTN-ACTIVE-MASK: rgba(255, 255, 255, .1)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-BTN-DEFAULT-ACTIVE-BG: rgba(255, 255, 255, .126)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-DIALOG-LINE-COLOR: rgba(255, 255, 255, .1)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--APPMSGCARD-BG: #1E1E1E}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--APPMSGCARD-LINE-BG: rgba(255, 255, 255, .07)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-BG-COLOR-ACTIVE: #373737}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-BG-6: rgba(255, 255, 255, .1);--weui-ACTIVE-MASK: rgba(255, 255, 255, .1)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-REDORANGE: #ff6146;--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--discussInput-BG: rgba(255, 255, 255, .03)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--nickName-FG: #959595}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--weui-BG-0: #111;--weui-BG-1: #1e1e1e;--weui-BG-2: #191919;--weui-BG-3: #202020;--weui-BG-4: #404040;--weui-BG-5: #2c2c2c;--weui-FG-0: rgba(255, 255, 255, .8);--weui-FG-HALF: rgba(255, 255, 255, .6);--weui-FG-1: rgba(255, 255, 255, .5);--weui-FG-2: rgba(255, 255, 255, .3);--weui-FG-3: rgba(255, 255, 255, .1);--weui-FG-4: rgba(255, 255, 255, .15);--weui-FG-5: rgba(255, 255, 255, .1);--weui-RED: #fa5151;--weui-ORANGERED: #ff6146;--weui-ORANGE: #c87d2f;--weui-YELLOW: #cc9c00;--weui-GREEN: #74a800;--weui-LIGHTGREEN: #3eb575;--weui-BRAND: #07c160;--weui-BLUE: #10aeff;--weui-INDIGO: #1196ff;--weui-PURPLE: #8183ff;--weui-WHITE: rgba(255, 255, 255, .8);--weui-LINK: #7d90a9;--weui-TEXTGREEN: #259c5c;--weui-FG: #fff;--weui-BG: #000;--weui-TAG-TEXT-RED: rgba(250, 81, 81, .6);--weui-TAG-BACKGROUND-RED: rgba(250, 81, 81, .1);--weui-TAG-TEXT-ORANGE: rgba(250, 157, 59, .6);--weui-TAG-BACKGROUND-ORANGE: rgba(250, 157, 59, .1);--weui-TAG-TEXT-GREEN: rgba(6, 174, 86, .6);--weui-TAG-BACKGROUND-GREEN: rgba(6, 174, 86, .1);--weui-TAG-TEXT-BLUE: rgba(16, 174, 255, .6);--weui-TAG-BACKGROUND-BLUE: rgba(16, 174, 255, .1);--weui-TAG-TEXT-BLACK: rgba(255, 255, 255, .5);--weui-TAG-BACKGROUND-BLACK: rgba(255, 255, 255, .05)}}@media(prefers-color-scheme:nodark){.wx-root:not([data-weui-theme=light]),body:not([data-weui-theme=light]){--appmsgExtra-BG: #121212}}blockquote{padding-left:10px;border-left:3px solid #dbdbdb;color:#00000080;font-size:15px;padding-top:4px;margin:1em 0}blockquote{color:#ffffff8c;color:var(--weui-FG-1)}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]) .rich_media_meta_text,body:not([data-weui-theme=light]) .icon_appmsg_tag.default,body:not([data-weui-theme=light]) .simple_pagination,body:not([data-weui-theme=light]) .sp_page_current,body:not([data-weui-theme=light]) blockquote,body:not([data-weui-theme=light]) .blockquote_info{color:#ffffff4d}}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]) blockquote{border-color:#404040}}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]) blockquote{border-color:#404040}}.data_color_scheme_dark .rich_media_meta_text,.data_color_scheme_dark .icon_appmsg_tag.default,.data_color_scheme_dark .reward_area_win .reward_user_tips .weui-loadmore__tips a,.data_color_scheme_dark .simple_pagination,.data_color_scheme_dark .sp_page_current,.data_color_scheme_dark blockquote,.data_color_scheme_dark .blockquote_info{color:#ffffff4d}.data_color_scheme_dark blockquote{border-color:#404040}@media(prefers-color-scheme:nodark){body:not([data-weui-theme=light]) .rich_media_content img:not(.wx_img_placeholder){filter:brightness(.8)}}.rich_media_content p{clear:both;min-height:1em}td p{margin:0;padding:0}

Llama-2 Mistral MPT=？ 融合多個異構大模型顯奇效

Llama-2 Mistral MPT=？融合多個異構大模型顯奇效