publications

2026

1. TASLP

   CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance

   Junchuan Zhao, Wei Zeng, Tianle Lyu, and Ye Wang

   IEEE Transactions on Audio, Speech and Language Processing, 2026

   Abs PDF Website

   Singing Voice Synthesis (SVS) aims to generate expressive vocal performances from structured musical inputs such as lyrics and pitch sequences. While recent progress in discrete codec-based speech synthesis has enabled zero-shot generation via in-context learning, directly extending these techniques to SVS remains non-trivial due to the requirement for precise melody control. In particular, prompt-based generation often introduces prosody leakage, where pitch information is inadvertently entangled within the timbre prompt, compromising controllability. We present CoMelSinger, a zero-shot SVS framework that enables structured and disentangled melody control within a discrete codec modeling paradigm. Built on the non-autoregressive MaskGCT architecture, CoMelSinger replaces conventional text inputs with lyric and pitch tokens, preserving in-context generalization while enhancing melody conditioning. To suppress prosody leakage, we propose a coarse-to-fine contrastive learning strategy that explicitly regularizes pitch redundancy between the acoustic prompt and melody input. Furthermore, we incorporate a lightweight encoder-only Singing Voice Transcription (SVT) module to align acoustic tokens with pitch and duration, offering fine-grained frame-level supervision. Experimental results demonstrate that CoMelSinger achieves notable improvements in pitch accuracy, timbre consistency, and zero-shot transferability over competitive baselines. Audio samples are available at https://danny-nus.github.io/CoMelSinger/.

2. ICLR 2026

   Bridging Piano Transcription and Rendering via Disentangled Score Content and Style

   Wei Zeng, Junchuan Zhao, and Ye Wang

   In The Fourteenth International Conference on Learning Representations, Rio de Janeiro, Brazil, Apr 2026

   Abs PDF Website

   Expressive performance rendering (EPR) and automatic piano transcription (APT) are fundamental yet inverse tasks in music information retrieval: EPR generates expressive performances from symbolic scores, while APT recovers scores from performances. Despite their dual nature, prior work has addressed them independently. In this paper, we propose a unified framework that jointly models EPR and APT by disentangling note-level score content and global performance style representations from both paired and unpaired data. Our framework is built on a transformer-based sequence-to-sequence (Seq2Seq) architecture and is trained using only sequence-aligned data, without requiring fine-grained note-level alignment. To automate the rendering process while ensuring stylistic compatibility with the score, we introduce an independent diffusion-based performance style recommendation (PSR) module that generates style embeddings directly from score content. This modular component supports both style transfer and flexible rendering across a range of expressive styles. Experimental results from both objective and subjective evaluations demonstrate that our framework achieves competitive performance on EPR and APT tasks, while enabling effective content–style disentanglement, reliable style transfer, and stylistically appropriate rendering. Demos are available at https://wei-zeng98.github.io/joint-apt-epr/.

2025

1. APSIPA ASC 2025

   Peransformer: Improving Low-informed Expressive Performance Rendering with Score-aware Discriminator

   Xian He, Wei Zeng, and Ye Wang

   In 2025 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, Singapore, Apr 2025

   Abs PDF

   Highly-informed Expressive Performance Rendering (EPR) systems transform music scores with rich musical annotations into human-like expressive performance MIDI files. While these systems have achieved promising Results, the availability of detailed music scores is limited compared to MIDI files and are less flexible to work with using a digital audio workstation (DAW). Recent advancements in low-informed EPR systems offer a more accessible alternative by directly utilizing scorederived MIDI as input, but these systems often exhibit suboptimal performance. Meanwhile, existing works are evaluated with diverse automatic metrics and data formats, hindering direct objective comparisons between EPR systems. In this study, we introduce Peransformer, a transformer-based low-informed EPR system designed to bridge the gap between low-informed and highly-informed EPR systems. Our approach incorporates a score-aware discriminator that leverages the underlying scorederived MIDI files and is trained on a score-to-performance paired, note-to-note aligned MIDI dataset. Experimental Results demonstrate that Peransformer achieves state-of-the-art performance among low-informed systems, as validated by subjective evaluations. Furthermore, we extend existing automatic evaluation metrics for EPR systems and introduce generalized EPR metrics (GEM), enabling more direct, accurate, and reliable comparisons across EPR systems. The repository is available at https://github.com/Bigstool/peransformer.

2024

1. IJCAI 2024

   End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding

   Wei Zeng, Xian He, and Ye Wang

   In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, Jeju Island, South Korea, Aug 2024

   Abs PDF Code Website

   Piano audio-to-score transcription (A2S) is an important yet underexplored task with extensive applications for music composition, practice, and analysis. However, existing end-to-end piano A2S systems faced difficulties in retrieving bar-level information such as key and time signatures, and have been trained and evaluated with only synthetic data. To address these limitations, we propose a sequence-to-sequence (Seq2Seq) model with a hierarchical decoder that aligns with the hierarchical structure of musical scores, enabling the transcription of score information at both the bar and note levels by multi-task learning. To bridge the gap between synthetic data and recordings of human performance, we propose a two-stage training scheme, which involves pre-training the model using an expressive performance rendering (EPR) system on synthetic audio, followed by fine-tuning the model using recordings of human performance. To preserve the voicing structure for score reconstruction, we propose a pre-processing method for **Kern scores in scenarios with an unconstrained number of voices. Experimental results support the effectiveness of our proposed approaches, in terms of both transcription performance on synthetic audio data in comparison to the current state-of-the-art, and the first experiment on human recordings.

2. TOMM

   Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing

   Xiangming Gu, Longshen Ou, Wei Zeng, Jianan Zhang, and 2 more authors

   ACM Transactions on Multimedia Computing, Communications, and Applications, May 2024

   Abs PDF Code

   Automatic lyric transcription (ALT) refers to transcribing singing voices into lyrics, while automatic music transcription (AMT) refers to transcribing singing voices into note events, i.e., musical MIDI notes. Despite these two tasks having significant potential for practical application, they are still nascent. This is because the transcription of lyrics and note events solely from singing audio is notoriously difficult due to the presence of noise contamination, e.g., musical accompaniment, resulting in a degradation of both the intelligibility of sung lyrics and the recognizability of sung notes. To address this challenge, we propose a general framework for implementing multimodal ALT and AMT systems. Additionally, we curate the first multimodal singing dataset, comprising N20EMv1 and N20EMv2, which encompasses audio recordings and videos of lip movements, together with ground truth for lyrics and note events. For model construction, we propose adapting self-supervised learning models from the speech domain as acoustic encoders and visual encoders to alleviate the scarcity of labeled data. We also introduce a residual cross-attention mechanism to effectively integrate features from the audio and video modalities. Through extensive experiments, we demonstrate that our single-modal systems exhibit state-of-the-art performance on both ALT and AMT tasks. Subsequently, through single-modal experiments, we also explore the individual contributions of each modality to the multimodal system. Finally, we combine these and demonstrate the effectiveness of our proposed multimodal systems, particularly in terms of their noise robustness.

2023

1. ACM MM 2023

   Elucidate Gender Fairness in Singing Voice Transcription

   Xiangming Gu, Wei Zeng, and Ye Wang

   In Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, Canada, May 2023

   Abs PDF

   It is widely known that males and females typically possess different sound characteristics when singing, such as timbre and pitch, but it has never been explored whether these gender-based characteristics lead to a performance disparity in singing voice transcription (SVT), whose target includes pitch. Such a disparity could cause fairness issues and severely affect the user experience of downstream SVT applications. Motivated by this, we first demonstrate the female superiority of SVT systems, which is observed across different models and datasets. We find that different pitch distributions, rather than gender data imbalance, contribute to this disparity. To address this issue, we propose using an attribute predictor to predict gender labels and adversarially training the SVT system to enforce the gender-invariance of acoustic representations. Leveraging the prior knowledge that pitch distributions may contribute to the gender bias, we propose conditionally aligning acoustic representations between demographic groups by feeding note events to the attribute predictor. Empirical experiments on multiple benchmark SVT datasets show that our method significantly reduces gender bias (up to more than 50%) with negligible degradation of overall SVT performance, on both in-domain and out-of-domain singing data, thus offering a better fairness-utility trade-off.

2. Appl. Energy

   Deep reinforcement learning towards real-world dynamic thermal management of data centers

   Qingang Zhang, Wei Zeng, Qinjie Lin, Chin-Boon Chng, and 2 more authors

   Applied Energy, May 2023

   Abs

   Deep Reinforcement Learning has been increasingly researched for Dynamic Thermal Management in Data Centers. However, existing works typically evaluate the performance of algorithms on a specific task, utilizing models or data trajectories without discussing in detail their implementation feasibility and their ability to deal with diverse work scenarios. The lack of these works limits the real-world deployment of Deep Reinforcement Learning. To this end, this paper comprehensively evaluates the strengths and limitations of state-of-the-art algorithms by conducting analytical and numerical studies. The analysis is conducted in four dimensions: algorithms, tasks, system dynamics, and knowledge transfer. As an inherent property, the sensitivity to algorithms settings is first evaluated in a simulated data center model. The ability to deal with various tasks and the sensitivity to reward functions are subsequently studied. The trade-off between constraints and power savings is identified by conducting ablation experiments. Next, the performance under different work scenarios is investigated, including various equipment, workload schedules, locations, and power densities. Finally, the transferability of algorithms across tasks and scenarios is also evaluated. The results show that actor-critic, off-policy, and model-based algorithms outperform others in optimality, robustness, and transferability. They can reduce violations and achieve around 8.84% power savings in some scenarios compared to the default controller. However, deploying these algorithms in real-world systems is challenging since they are sensitive to specific hyperparameters, reward functions, and work scenarios. Constraint violations and sample efficiency are some aspects that are still unsatisfactory. This paper presents our well-structured investigations, new findings, and challenges when deploying deep reinforcement learning in Data Centers.