Audio ControlNet for Fine-Grained Audio Generation and Editing

This work studies fine-grained text-to-audio (T2A) generation with explicit control over key audio attributes, including loudness, pitch, and sound events. Instead of retraining models for individual control types, a ControlNet-based framework is built on top of pre-trained T2A backbones to enable flexible and extensible controllable generation. Two designs, T2A-ControlNet and the more lightweight T2A-Adapter, are introduced. With only 38M additional parameters, T2A-Adapter achieves state-of-the-art performance on AudioSet-Strong while maintaining strong control ability. The framework is further extended to audio editing, enabling the insertion and removal of audio events at specified time locations.

The core idea is to introduce an auxiliary conditional network that injects conditions into the pretrained FluxAudio, enabling controllable audio generation without retraining the backbone. We introduce two variants of Audio ControlNet: T2A-ControlNet and T2A-Adapter. T2A-ControlNet replicates FluxAudio, including both MMDiT and DiT layers, and directly adds the processed latent to each layer in FluxAudio. T2A-Adapter employs a lightweight encoder to extract features from time-varying conditions. It extracts condition features and injects them into the latent of the backbone.

Our experiments demonstrate that both T2A-ControlNet and T2A-Adapter achieve consistent improvements in controlling loudness, pitch, and sound events. Notably, T2A-Adapter achieves state-of-the-art performance on the AudioSet-Strong in both event-level and segment-level F1 scores while introducing only 38M additional parameters. Subjective evaluations further indicate that our methods outperform all baseline models.