使用音频单元记录回调[iOS] [Swift]处理数据 [英] Processing data with Audio Unit recording callback [iOS][Swift]
问题描述
我正在创建一个跨平台的VOIP应用程序，该应用程序使用UDP发送和接收数据.我正在使用音频单元进行实时记录和播放.在处理原始数据时，通信是快速而流畅的，但是当我涉及到像 OPUS 这样的编解码器时，从iPhone编码并发送到Android的数据在其间会发出咔嗒声.我一直在努力解决这个问题.
从Android到iPhone的编码数据可以完美播放，并且没有任何问题.我正在使用 TPCircularBuffer 处理记录和播放时的数据.
这是我到目前为止在录音回调中所拥有的:
var samplesForEncoder: UInt32 = 640
var targetBuffer = [opus_int16](repeating: 0, count: 1500)
_ = TPCircularBufferProduceBytes(&circularBuffer, mData, inNumberFrames * 2)
self.samplesSinceLastCall += inNumberFrames
encodingQueue.async {
if self.samplesSinceLastCall > self.samplesForEncoder {
let samplesToCopy = min(self.bytesToCopy, Int(self.availableBytes))
self.bufferTailPointer = TPCircularBufferTail(&self.circularBuffer, &self.availableBytes)
memcpy(&self.targetBuffer, self.bufferTailPointer, samplesToCopy)
self.semaphore.signal()
self.semaphore.wait()
self.opusHelper?.encodeStream(of: self.targetBuffer)
self.semaphore.signal()
self.semaphore.wait()
TPCircularBufferConsume(&self.circularBuffer, UInt32(samplesToCopy))
self.samplesSinceLastCall = 0
self.semaphore.signal()
self.semaphore.wait()
}
}
这是编码功能:
var encodedData = [UInt8](repeating: 0, count: 1500)
self.encodedLength = opus_encode(self.encoder!, samples, OpusSettings.FRAME_SIZE, &self.encodedData, 1500)
let opusSlice = Array(self.encodedData.prefix(Int(self.encodedLength!)))
self.seqNumber += 1
self.protoModel.sequenceNumber = self.seqNumber
self.protoModel.timeStamp = Date().currentTimeInMillis()
self.protoModel.payload = opusSlice.data
do {
_ = try self.udpClient?.send(data: self.protoModel)
} catch {
print(error.localizedDescription)
}
我尝试使用 DispatchGroups ， DispatchSourceTimers ， DispatchSemaphores ， DispatchQueues 处理另一个线程中的繁重处理. >但我无法获得所需的结果.有人可以帮忙吗?
任何人都可以指导我如何使编码独立于实时音频线程时，我试图创建一个轮询线程，但即使这样也没有用.我需要在具有不同数据大小要求的2个线程之间传输数据的帮助.我正在从麦克风接收341-342字节，但我需要将640字节发送给编码器，因此我将合并2个样本，并将剩余的字节重用，以备后用.
@ hotpaw2建议使用此 https://stackoverflow.com/a/58947295/12020007 ，但我只需要一点更多指导.
根据@ hotpaw2的答案更新代码:
录音回调:
_ = TPCircularBufferProduceBytes(&circularBuffer, mData, inNumberFrames * 2)
self.samplesSinceLastCall += inNumberFrames
if !shouldStartSending {
startLooping()
}
更新的轮询线程:
func startLooping() {
loopingQueue.async {
repeat {
if self.samplesSinceLastCall > self.samplesForEncoder {
let samplesToCopy = min(self.bytesToCopy, Int(self.availableBytes))
self.bufferTailPointer = TPCircularBufferTail(&self.circularBuffer, &self.availableBytes)
memcpy(&self.targetBuffer, self.bufferTailPointer, samplesToCopy)
self.semaphore.signal()
self.semaphore.wait()
self.opusEncodedStream = self.opusHelper?.encodeStream(of: self.targetBuffer)
self.semaphore.signal()
self.semaphore.wait()
self.send(stream: self.opusEncodedStream!)
self.semaphore.signal()
self.semaphore.wait()
TPCircularBufferConsume(&self.circularBuffer, UInt32(samplesToCopy))
self.samplesSinceLastCall = 0
}
self.shouldStartSending = true
} while true
}
}
Apple建议不要在任何实时音频单元回调中使用信号量或调用Swift方法(例如编码器).只需将数据复制到音频单元回调内部的预分配循环缓冲区中即可.时期.在回调之外执行其他所有操作.包括信号灯和信号.
因此，您需要创建一个轮询线程.
执行轮询循环，计时器回调或网络就绪回调中的所有操作.只要FIFO中有足够的数据，就可以进行工作.经常进行呼叫(轮询)(足够高的轮询频率或计时器回调速率)，以确保您不会丢失数据.在轮询循环的每次迭代中处理所有数据(一次可能有多个缓冲区，如果可能的话).
在开始发送之前，您可能需要预先填充循环缓冲区(可能是640 UDP帧大小的几倍)，以解决网络和计时器抖动.
I am creating a cross platform VOIP application which uses UDP to send and receive data. I am using audio units for the real time recording and playback. The communication is fast and smooth when working with raw data but when I involve a codec like OPUS, the data which is being encoded and sent from iPhone to Android has clicking and popping sounds in between. I have been pulling my hair out trying to solve this issue.
The encoded data which is coming from Android to iPhone plays perfectly and there are no issues with that. I am using TPCircularBuffer to handle the data when recording and playback.
This is what I have so far in the recording callback:
var samplesForEncoder: UInt32 = 640
var targetBuffer = [opus_int16](repeating: 0, count: 1500)
_ = TPCircularBufferProduceBytes(&circularBuffer, mData, inNumberFrames * 2)
self.samplesSinceLastCall += inNumberFrames
encodingQueue.async {
if self.samplesSinceLastCall > self.samplesForEncoder {
let samplesToCopy = min(self.bytesToCopy, Int(self.availableBytes))
self.bufferTailPointer = TPCircularBufferTail(&self.circularBuffer, &self.availableBytes)
memcpy(&self.targetBuffer, self.bufferTailPointer, samplesToCopy)
self.semaphore.signal()
self.semaphore.wait()
self.opusHelper?.encodeStream(of: self.targetBuffer)
self.semaphore.signal()
self.semaphore.wait()
TPCircularBufferConsume(&self.circularBuffer, UInt32(samplesToCopy))
self.samplesSinceLastCall = 0
self.semaphore.signal()
self.semaphore.wait()
}
}
This is the encoding function:
var encodedData = [UInt8](repeating: 0, count: 1500)
self.encodedLength = opus_encode(self.encoder!, samples, OpusSettings.FRAME_SIZE, &self.encodedData, 1500)
let opusSlice = Array(self.encodedData.prefix(Int(self.encodedLength!)))
self.seqNumber += 1
self.protoModel.sequenceNumber = self.seqNumber
self.protoModel.timeStamp = Date().currentTimeInMillis()
self.protoModel.payload = opusSlice.data
do {
_ = try self.udpClient?.send(data: self.protoModel)
} catch {
print(error.localizedDescription)
}
I have tried to handle the heavy processing inside another thread by using DispatchGroups, DispatchSourceTimers, DispatchSemaphores, DispatchQueues but I just cannot get the result that I need. Can anyone help?
Can anyone guide me how to make the encoding independent of the real time audio thread, I tried to create a polling thread but even that did not work. I need assistance on transferring data between 2 threads with different data size requirements. I am receiving 341-342 bytes from the mic but I need to send 640 bytes to the encoder hence I am combining 2 samples and reusing the left over bytes for later.
@hotpaw2 recommends this https://stackoverflow.com/a/58947295/12020007 but I just need a little more guidance.
Updated code as per @hotpaw2's answer:
Recording callback:
_ = TPCircularBufferProduceBytes(&circularBuffer, mData, inNumberFrames * 2)
self.samplesSinceLastCall += inNumberFrames
if !shouldStartSending {
startLooping()
}
Updated polling thread:
func startLooping() {
loopingQueue.async {
repeat {
if self.samplesSinceLastCall > self.samplesForEncoder {
let samplesToCopy = min(self.bytesToCopy, Int(self.availableBytes))
self.bufferTailPointer = TPCircularBufferTail(&self.circularBuffer, &self.availableBytes)
memcpy(&self.targetBuffer, self.bufferTailPointer, samplesToCopy)
self.semaphore.signal()
self.semaphore.wait()
self.opusEncodedStream = self.opusHelper?.encodeStream(of: self.targetBuffer)
self.semaphore.signal()
self.semaphore.wait()
self.send(stream: self.opusEncodedStream!)
self.semaphore.signal()
self.semaphore.wait()
TPCircularBufferConsume(&self.circularBuffer, UInt32(samplesToCopy))
self.samplesSinceLastCall = 0
}
self.shouldStartSending = true
} while true
}
}
Apple recommends against using semaphores or calling Swift methods (such as encoders) inside any real-time Audio Unit callback. Just copy the data into a pre-allocated circular buffer inside the audio unit callback. Period. Do everything else outside the callback. Semaphores and signals included.
So, you need to create a polling thread.
Do everything inside a polling loop, timer callback, or network ready callback. Do your work anytime there is enough data in the FIFO. Call (poll) often enough (high enough polling frequency or timer callback rate) that you do not lose data. Handle all the data you can (perhaps multiple buffers at a time, if available) in each iteration of the polling loop.
You may need to pre-fill the circular buffer a bit (perhaps a few multiples of your 640 UDP frame size) before starting to send, to account for network and timer jitter.
这篇关于使用音频单元记录回调[iOS] [Swift]处理数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
