How Audio DSPs Compute Real-Time Sound in Modern Wearables

Inside the Millisecond Window of Modern Audio Silicon

Active noise control, voice enhancement, and audio transparency all operate within a narrow physical timing window. In a hearable, the system must capture sound at a MEMS microphone, move the samples through conversion and filtering stages, compute a corrective signal, and reproduce that signal through a speaker before the acoustic event has materially changed. For practical active noise control, the round-trip group latency generally needs to remain below approximately 5 milliseconds, while the most demanding paths target sub-millisecond processing. Excess delay can produce comb filtering, audible coloration, or unstable control behavior rather than clean cancellation.

This timing requirement is why a specialized audio DSP remains valuable even when a product already contains a powerful application processor. A general-purpose microcontroller can execute audio algorithms, but its interrupt behavior, cache activity, operating-system overhead, and comparatively inefficient multiply-accumulate execution make deterministic latency difficult under a milliwatt-scale thermal envelope. A modern wearable instead divides the work across an analog front end, digital microphone interface, decimation filters, DMA engines, tightly coupled memory, vector or SIMD DSP units, and a low-power audio output path. Products such as chip-con semiconductor solutions illustrate the broader component-selection challenge: the processing device must fit the complete power, interface, memory, and lifecycle requirements rather than simply advertise a high clock frequency.

Hardware Pipeline and Signal Ingestion Architecture

The signal path begins at one or more MEMS microphones. Many digital MEMS devices output pulse-density modulation, or PDM, in which the instantaneous density of one-bit pulses represents the acoustic waveform. A hardware decimation filter converts that high-rate bitstream into lower-rate PCM samples, commonly at audio rates such as 16, 24, 32, or 48 kHz. Decimation is not merely a format conversion. It includes low-pass filtering to suppress quantization noise outside the retained audio band, and its group delay becomes part of the system latency budget.

After conversion to PCM, samples may be routed through I2S or a multichannel time-division multiplexed interface. TDM is particularly useful when a wearable has several microphones because multiple channels share a synchronous serial link while retaining deterministic frame positions. The audio hardware must preserve channel alignment accurately. A one-sample timing error between microphones can degrade beamforming, while clock drift between capture and playback domains can cause buffer growth or periodic sample slips.

DMA engines move completed sample blocks without requiring the application CPU to service every sample. A common design uses ping-pong buffers: while DMA fills one buffer, the DSP processes the other. Tightly coupled SRAM or scratchpad memory then supplies predictable, low-latency access to filter states, coefficient tables, and intermediate vectors. Cache-based systems can work, but cache misses introduce timing variation that is undesirable in a control loop. A practical ingestion architecture typically follows these priorities:

• Use hardware PDM decimation when its passband, stopband, and delay characteristics meet the product requirement.
• Use DMA with short, bounded blocks to reduce interrupt frequency without allowing buffering latency to grow.
• Reserve tightly coupled memory for active filter state and hot coefficient data.
• Expose programmable DSP instructions where microphone calibration, beamforming, or adaptive filtering will evolve during product development.

Fixed-function blocks usually provide the best energy efficiency for repetitive operations such as decimation, sample-rate conversion, and simple equalization. Their limitation is inflexibility. A programmable arithmetic unit consumes more energy per operation, but it can implement a new beamformer, a different acoustic model, or a field-updatable noise-reduction algorithm. The correct partition depends on algorithm stability. Hardware should absorb mature, high-volume operations, while programmable DSP resources should remain available for features whose coefficients and topology are likely to change.

Arithmetic Trade-Offs in Fixed-Point versus Floating-Point DSPs

Fixed-point DSPs remain attractive because a 24-bit or 32-bit multiply-accumulate unit can deliver deterministic throughput with modest silicon area and low switching energy. Audio samples and coefficients are represented as scaled integers, often using formats such as Q1.23 or Q1.31. Floating-point DSPs, by contrast, use IEEE 754 single-precision values with a sign bit, exponent, and fraction. They simplify algorithm development because scaling is largely automatic, particularly in cascaded filters and adaptive algorithms whose internal values vary widely.

The engineering cost of fixed point is headroom management. A filter accumulator may require guard bits because the sum of many products can exceed the nominal sample range even when the final output does not. Designers must select coefficient and signal scaling, monitor saturation, and test worst-case correlated inputs. Quantization also changes the behavior of narrowband filters, adaptive updates, and feedback paths. Floating point reduces these risks, but it does not eliminate numerical design work. Denormals, rounding behavior, memory bandwidth, and conversion overhead still matter in a low-power implementation.

There is no universal milliwatt-per-MAC value. Energy depends on process technology, clock frequency, voltage, memory traffic, SIMD width, and whether the operation is performed in a dedicated accelerator. In practical profiling, a 16-bit or 24-bit fixed-point MAC often consumes less energy than a single-precision floating-point MAC because the multiplier, accumulator, register file, and data movement are narrower. However, an inefficient fixed-point implementation can lose that advantage through repeated shifts, saturation checks, and format conversions. The following comparison captures the design trade-off rather than prescribing one format.

Characteristic	Fixed-point DSP	Single-precision floating point
Arithmetic efficiency	Typically high, especially with SIMD MAC instructions	Higher per-operation cost, but simpler scaling
Dynamic range	Defined by word length and scaling strategy	Broad dynamic range with exponent support
Primary risk	Overflow, saturation, and coefficient quantization	Memory traffic, conversion cost, and higher power
Best fit	Stable production pipelines with known signal bounds	Rapid algorithm development and highly variable adaptive states

The embedded SONIC system provides a useful example of the algorithmic side of this decision. Its ARM Cortex-M7 implementation favors LMS-based adaptive filtering because the algorithm has O(N) work per sample, modest memory requirements, and low latency. That makes careful fixed-point optimization realistic, provided the reference signal, error signal, and coefficient updates are scaled consistently. The published system description is available in the SONIC research paper. Before selecting arithmetic, profile complete frames, including loads, stores, state updates, and saturation handling. The MAC alone is not the energy cost of the audio algorithm.

Acoustic Beamforming and Spatial Sound Isolation

Beamforming combines signals from spatially separated microphones so that sound from a desired direction adds constructively while interference is attenuated. In its simplest delay-and-sum form, each microphone signal is delayed by an estimate of its propagation time and multiplied by a weight. The output can be expressed as y(n) = sum of wmxm(n – dm), where xm is the signal from microphone m, dm is the delay, and wm is its gain or phase weight. In the frequency domain, fractional delays become phase rotations, which makes frequency-domain beamforming practical for longer filters.

Minimum variance distortionless response, or MVDR, adds an adaptive spatial filter. It chooses a weight vector that minimizes output power while preserving a specified response toward the desired direction. In simplified form, the weight vector depends on the inverse spatial covariance matrix and the steering vector. This provides sharper interference rejection than a conventional delay-and-sum beamformer, but covariance estimation, matrix inversion, regularization, and update scheduling consume substantially more compute. A 2024 study using a tetrahedral microphone array compared conventional phase-shift processing with adaptive MVDR and documented the resulting precision and computational trade-offs in the Portland State University thesis.

Compact arrays in smart glasses and hearables impose additional constraints. The microphones may be separated by only a few centimeters, so the available time difference is small and becomes frequency dependent. Mechanical tolerances, microphone sensitivity mismatch, wind turbulence, frame reflections, and user head movement all affect the steering model. A practical directional enhancement calculation can be organized as follows:

1. Capture synchronized PCM frames from each microphone and apply per-channel gain and phase calibration.
2. Transform the frame into frequency bins or apply time-domain fractional-delay filters.
3. Estimate the steering vector for the target direction and form the microphone signal vector.
4. Compute the weighted sum for a conventional beamformer, or update the covariance matrix and solve the regularized MVDR weights.
5. Apply post-filtering, inverse transformation, and output limiting before sending the enhanced signal to the playback path.

The architectural implication is clear. A two-microphone voice path may fit comfortably in a low-power DSP, while an eight-channel spatial system requires efficient vector operations, carefully bounded matrix updates, and memory bandwidth that does not erase the arithmetic savings. Microphone arrays are valuable because semiconductor-manufactured MEMS devices can be closely matched, but calibration must still be treated as a production and field-reliability requirement rather than an optional software detail.

Adaptive Filtering and Sub-Band Active Noise Cancellation

Adaptive ANC uses a reference signal to estimate unwanted sound and generates an anti-noise signal with opposite phase at the listener position. The least mean squares algorithm updates a filter coefficient vector in the direction that reduces the measured error. Normalized LMS divides the update by an estimate of reference-signal power, improving stability when the acoustic environment changes significantly. Filtered-X LMS goes further by passing the reference through an estimate of the secondary path, which includes the DSP, amplifier, speaker, acoustic propagation, and error-sensing path. Without this compensation, the adaptive update may move in the wrong direction because the control signal is not experienced at the error microphone exactly as calculated.

Sub-band processing reduces the cost of long adaptive filters. The audio spectrum is divided into bands, and each band receives a filter order suited to its acoustic behavior. Low-frequency bands often require longer filters because their wavelengths and reverberation characteristics produce slower, extended responses. Higher bands can use shorter filters or be handled by conventional noise reduction, especially when the speaker and acoustic path cannot reproduce them efficiently. Polyphase filter banks, FFT-based partitioned convolution, and decimated low-frequency control paths all reduce the number of active MAC operations, but each adds buffering and analysis delay.

Open-ear architectures are harder because there is no sealed ear canal and often no convenient in-ear error microphone. Leakage, head movement, and changing speaker-to-ear geometry make the secondary path strongly user dependent. Recent open-ear work addresses this by using distributed microphones and a learned estimate of the sound arriving at the ear, while reserving a dedicated low-latency DSP for the actual anti-noise generation. A prototype using eight frame microphones and open-ear speakers reported mean reductions of 9.6 dB without calibration and 11.2 dB with brief user-specific calibration across eight environments and 100 to 1000 Hz, as described in the open-ear ANC research. These figures are experimental results, not universal product specifications, but they demonstrate the value of separating slow model estimation from the fast control loop.

• Keep coefficient adaptation slower than sample generation so the control path remains deterministic.
• Use leakage-aware regularization and conservative step sizes when the acoustic path changes rapidly.
• Allocate more filter order to bands where the speaker and enclosure provide useful control authority.
• Monitor reference coherence, error power, and actuator headroom to detect conditions in which cancellation should be reduced.
• Validate walking, wind, speech, and frame-contact conditions, not only stationary laboratory noise.

In practice, battery life depends as much on memory movement and microphone count as on the selected LMS variant. A low-complexity algorithm running continuously across eight channels can exceed the energy of a more sophisticated algorithm operating on two decimated reference bands. The system should therefore adapt duty cycle, update rates, and channel participation to the acoustic context while protecting the sub-millisecond path from unpredictable workloads.

The Path Toward Ultra-Low-Power Acoustic Intelligence

Sub-millisecond wearable audio is achieved through hardware-software co-design, not through processor speed alone. The microphone interface, decimation filter, DMA schedule, memory placement, arithmetic format, vector width, adaptive update policy, and playback clock must be designed as one timing system. Fixed-point execution is often the most energy-efficient choice for a stable production path, while floating point can reduce development risk for algorithms with large dynamic range or rapidly changing internal states. The decision should come from measured end-to-end energy and latency, including data movement and worst-case scheduling.

The next generation of acoustic intelligence will likely combine conventional DSP control loops with specialized acceleration. Neuromorphic or analog front ends may perform low-power feature extraction, while dedicated DSP or tensor cores handle beamforming, learned acoustic-path estimation, and neural denoising. The design priorities remain practical: preserve deterministic latency, maintain sufficient dynamic range, control thermal load, and verify performance across users and environments. For systems engineers, the strongest architecture is not the one with the largest compute specification. It is the one that delivers transparent acoustics, stable adaptation, and reliable battery operation within the complete wearable power budget.