Huawei's recently released PanGu Pro MoE large model has achieved excellent performance through its innovative design of dynamically activating expert networks. However, a recent study published on GitHub has attracted industry attention. The authors of the study believe that Huawei's PanGu large model (PanGu Pro MoE) and Alibaba's Tongyi Qianwen Qwen-2.5 14B model have an "astonishing consistency" in terms of parameter structure.
Regarding this study, on the afternoon of July 5, the PanGu team at Huawei issued a statement, stating that the PanGu Pro MoE open-source model was developed and trained based on the Ascend hardware platform, not incrementally trained from other manufacturers' models. It has made key innovations in architecture design and technical characteristics, making it the world's first same-specification mixture-of-experts model designed for the Ascend hardware platform. It innovatively proposed the Group Mixture-of-Experts (MoGE) architecture, effectively solving the load balancing problem in large-scale distributed training and improving training efficiency.
The statement pointed out that the PanGu Pro MoE open-source model's basic components partially reference industry open-source practices, involving parts of the open-source code from other open-source large models. "We strictly follow the requirements of open-source licenses, clearly marking the copyright notices of open-source code in the open-source code files. This is not only a common practice in the open-source community but also aligns with the open-source collaboration spirit advocated by the industry. We always adhere to open innovation, respect third-party intellectual property rights, and advocate inclusive, fair, open, united, and sustainable open-source concepts."
Below is the original statement from the Huawei PanGu team:
We have noted the recent discussions about the PanGu large model's open-source code in the open-source community and online platforms.
The PanGu Pro MoE open-source model is a fundamental large model developed and trained based on the Ascend hardware platform, not incrementally trained from other manufacturers' models. It has made key innovations in architecture design and technical characteristics, making it the world's first same-specification mixture-of-experts model designed for the Ascend hardware platform. It innovatively proposed the Group Mixture-of-Experts (MoGE) architecture, effectively solving the load balancing problem in large-scale distributed training and improving training efficiency. For other technical innovation features, please refer to the content disclosed in the Ascend ecosystem competitiveness series technical reports.
The PanGu Pro MoE open-source model's basic components partially reference industry open-source practices, involving parts of the open-source code from other open-source large models. We strictly follow the requirements of open-source licenses, clearly marking the copyright notices of open-source code in the open-source code files. This is not only a common practice in the open-source community but also aligns with the open-source collaboration spirit advocated by the industry. We always adhere to open innovation, respect third-party intellectual property rights, and advocate inclusive, fair, open, united, and sustainable open-source concepts.
We appreciate the attention and support from global developers and partners for the PanGu large model. We highly value the constructive opinions from the open-source community. We hope to explore and continuously optimize model capabilities with like-minded partners through the open-sourcing of the PanGu large model, accelerating technological breakthroughs and industrial applications.
We welcome and look forward to in-depth and professional exchanges on technical details in the open-source community Ascend Tribe.
This article is an exclusive work of Observer, and without authorization, it shall not be reprinted.
Original: https://www.toutiao.com/article/7523529532817834537/
Statement: This article represents the personal views of the author. Please express your opinion by clicking on the 【up/down】 buttons below.