There currently is no way to do this kind of audio mixing. Only one audio source is used at a time.
But why GPUImageMovieWriter only support one audio track writing?!!! This really limits its capability and it shouldn’t be like that.
#GPUImageMovieWriter: One Audio Source Only?
The reason that cause it only could handle one audio source during writing lies in the structure.
First, let’s recall the code to do audio writing setup:
Yes, it is this code who does the audio writing setup:
self.movieWriter.shouldPassthroughAudio = YES;
self.gpuMovieA.audioEncodingTarget = self.movieWriter;
//add this line otherwise it will cause "Had to drop an audio frame" which cozes the saved video lose some sounds
Here you tell writer saying “hey, I want you to writer audio from gpuMovieA to final output video”.
Once you dive into the code enableSynchronizedEncodingUsingMovieWriter in GPUImageMovie, it has called important method readNextAudioSampleFromOutput,which is kinda like following:
Now you understand why it could only handle one audio writing. It is because audio’s sample buffer is not like video’s sample buffer which you could upload to GPU, render it using shader, then pull it down and write to disk. AKA: there is no way you could merge two audio sample buffer to create a “combining” effect like video.
First, let’s see how we modified the GPUImageMovie by simply adding one method (void)loadAsset:(dispatch_group_t)readyGroup; and change other little bit:
So basically we moved out the asset initialization code to loadAsset and now in startProcessing it only need to kick off processAsset(asset reading).
You may wonder what’s the parameter in loadAsset is doing. Yeah, it is quite simple as GPUImageMovie need load tracks async for asset, we need a way to notify others that my asset is fully loaded and ready and I’m really good to kick asset reading part.
Let’s move to GPUImageMovieWriter class.
This is gonna be more change comparing to GPUImageMovie, but it is quite simple. We start by adding two variables:
Followed by adding setupAudioReaderWithTracks method to setup audio reader that mixing multiple audio tracks from movie:
Nothing fancy except AVAssetReaderAudioMixOutput this part, which is magic :)
Then - (void)startAudioRecording to start reading, so that our audio reader is ready to go!
Next is the audio writing part, - (void)startAudioWritingWithComplectionBlock:(void (^)())completionBlock:
Again, nothing quite special here. One thing to note is that we dont’ call asset write finishing writing when audio wrting is done because that we need check video writing part, which means that we also need to add new method for handling video wrting finish.
The last thing which is also very important is that we need to modify - (void)startRecording to uncomment last line.
[assetWriter startSessionAtSourceTime:kCMTimeZero]; this line of code has to be fired!
That’s it for GPUImageMovieWriter. And That’s all the changes for GPUImage part, but how the demo running code is gonna be looked like ?
We’re all set, let’s take a look the demo code:
Let’s start with this part:
We have used a dispatch group to make sure all GPUImageMovies has finished their initialization process, which at that time all assets are ready. Then we got all the audio tracks from movies for later use.
This part is really easy, we check if any track is available, then tell movie writer to setup audio reader and writer.
Here we use another dispatch group who helps us to cooridinate audio and video writing part as those two are both asynchrously. Then we fire startRecording,which just starts asset writer.
This is the video writing part, be careful with finishVideoRecordingWithCompletionHandler as it is totally wrong if you call original method finishRecordingWithCompletionHandler or finishRecording because we control how movie writer when and how it should be finished.
Audio handling is pretty much same like video.(ps: I really should rename those methods name to be more consistent. For example, startAudioRecording acutally should be renamed to startAudioReading, and startAudioWritingWithComplectionBlock should be finishAudioWritingWithComplectionBlock)
Last part is the final completion code that video and audio are both finished their writing:
That’s it, give a shoot in XCode, you should be able to what you expected! :)
Man, working with low level threading and avfoundation is a challenage, working on a very complex existing codebase is damn super ultra challenge.
I help someone could give some suggestions to make this library even better :)