How to Use AVAudioUnitSampler in a Custom Audio Unit

In CoreAudio, a sampler can be instantiated using the Audio Unit V2 API (referred to as AUSampler, for iOS 8 or earlier) or with the V3 API (AUAudioUnit, for iOS 9 or later). Usually they are attached to an AUGraph or connected to an AVAudioEngine.

For a customer I wrote an iOS app where I needed a sampler Audio Unit with a number of special MIDI functions. Also there should be an Audio Unit App Extension. Instead of writing a new sampler from scratch I had a look at AVAudioUnitSampler which already has useful features. It can load SoundFonts, and it supports most MIDI Controller messages.

So I decided to build an Audio Unit (V3 API) that internally uses an AVAudioUnitSampler. In this post I will describe the Audio Unit in detail.

The AU is written in Objective-C++ for easy invocation of the internal C++ kernel. I guess with some effort the AU could be written in Swift. However, with the render callback in mind which is invoked on a high-priority thread, and with audio I/O buffering, the audio rendering system in CoreAudio can be considered a realtime subsystem where the render callback must return within the latency time. We should avoid ARC locks and other side effects in the render callback.

Audio Unit Interface

The Objective-C interface looks like this:

The pan property is just there to show a simple use of the internal sampler (see implementation below).

Audio Unit Implementation

Before we write the implementation we define this extension for private instance variables:

There is a _kernel instance variable. The C++ class SamplerKernel is described later.

We keep a private AVAudioUnitSampler instance — this is the internal sampler that generates audio for us.

The variable presetParameter is just for fun, to have at least one parameter in the Audio Unit.

In the SamplerAudioUnit initializer method we allocate the output bus:

Then we create the internal sampler and allocate the so-called kernel which is a C++ class instance that handles custom MIDI processing. The kernel takes the AudioUnit of the internal sampler as a parameter to its constructor so later it can send MIDI events to it:

In the interface definition you have seen a pan property. Its purpose is to set the panorama based on values between 0 and 127 (like MIDI CC 10). Here is its setter which converts and forwards the value to the internal sampler:

The method loadCustomSoundFont loads a given SoundFont preset into the internal sampler.

Here we maintain the boolean flag presetLoaded in the kernel. When the flag is false the kernel avoids sending MIDI messages to the internal sampler so we don’t hear any default sine wave notes. This happens if no preset is loaded.

We override a few mandatory methods of the AUAudioUnit superclass. In their implementation we also invoke the same methods of the internal sampler. Normally this should not be necessary but we do it anyway, just in case it has some internal states that need to be consistent.

The reset method is not mandatory to override but we do it anyway to invoke the same method on the internal sampler.

Now we override the internalRenderBlock property getter which is where the magic with the internal sampler happens:

The returned block captures two local pointer variables. kernel is a pointer copy of the instance variable _kernel, to avoid capturing self. The same is true for proxySelf. This way we ensure that self is not captured in any way.

The block invokes the handleRealtimeMIDI function of the kernel which processes all MIDI events. The kernel will send some MIDI events to the internal sampler. In the simple implementation of the kernel (see below) all MIDI events are passed through. The kernel maintains an internal flag presetLoaded which, if true, means that the sampler has successfully loaded a preset so the kernel may send MIDI events to it. The flag is atomic to ensure its value is always intact, even if it is modified at the same time by another thread.

Then the outputData buffers are prepared. This is basically a plain copy from Apple’s AUv3Host example code.

In the last line of the block the function AudioUnitRender is invoked. Here we pass the AudioUnit of the internal sampler so it fills the audio buffers for us. All other parameters are passed through.

Audio Unit Kernel

The so-called kernel is responsible for the processing of MIDI events. Audio is not generated here because it is generated by the internal sampler in the render block. The kernel just sends MIDI events to the internal sampler.

The handleRealtimeMIDI method manages all incoming MIDI events with their timestamps:

This method is basically a dispatcher. It invokes the handleMIDIEvent method which finally does something interesting with MIDI events:

In this method we simply forward MIDI events to the internal sampler by calling the C function MusicDeviceMIDIEvent with the AudioUnit of the internal sampler. This causes the internal sampler to render some audio which is captured in internalRenderBlock (see above in Audio Unit Implementation).

The MusicDeviceMIDIEvent function takes a sample offset as its last parameter. It is important to use the sample offset of the MIDI event so the internal sampler can render its audio with sample-frame precision.

You are free to implement any custom MIDI behavior here. For example, you could:

  • Filter or pass events based on some condition;
  • Transform events (e.g. transpose, map velocity ramps, or map pitchbend range);
  • Implement an arpeggiator;
  • Implement chord detection;
  • Provide functions to certain MIDI controllers or channel pressure which are currently unused in AUSampler;
  • Implement a custom function based on MIDI channels (note that AUSampler is omniphonic, ie. it ignores the MIDI channel), or
  • Implement tempo-based functions synced to MIDI clock or MTC.

Conclusion

For simple sampler jobs it is easy to use this approach. You can implement your own MIDI functions. Alternatively you could also write a MIDI effect, then cascade it with an AVAudioUnitSampler instrument in your AU graph. But the presented approach is useful if you intend to create an Audio Unit App Extension where both the sampler and some MIDI functions are in one Audio Unit.

Should you experience issues with your SoundFont, then I recommend reading my other post about supported SoundFont parameters in Apple’s AUSampler.

The complete source code is available.