通过Flutter从C ++插件访问资产 [英] Accessing assets from C++ plugin through Flutter

问题描述

由于延迟时间短，我正在尝试将Google Oboe用于3D音频处理应用程序.该应用程序将具有一个C ++后端来执行处理，而前端则是通过Flutter完成的.我正在运行一些测试，以查看它是否可以正常工作，但是在将资产从Flutter加载到Oboe时遇到了问题.我在Java上检查了Oboe的 repo 中的RhythmGame示例，但无法安静地找到一种方法直接从Dart到C ++.前端和后端之间的连接是通过 dart :: ffi

I'm trying to use Google Oboe for a 3D audio processing app due to it's low latency. The app will have a C++ backend, which does the processing, and the frontend is done with Flutter. I'm running a couple of tests to see if it'll work but I'm having issues loading assets from Flutter to Oboe. I checked the example RhythmGame in Oboe's repo, done with Java, but couldn't quiet find a way of doing that straight from Dart to C++. The connection between front and backend is through dart::ffi

这是到目前为止我尝试过的.根据Richard Heap在此处发布的示例，我更改了 noise 变量，从正弦波到wav文件中歌曲的一小段:

Here's what I've tried so far. Based on the example published by Richard Heap here, I changed the noise variable from just a sine wave to a short fragment of a song in a wav file:

class _MyAppState extends State<MyApp> {
  final stream = OboeStream();

  var noise = Float32List(512);
  Timer t;

  @override
  void initState() {
    super.initState();
    // for (var i = 0; i < noise.length; i++) {
    //   noise[i] = sin(8 * pi * i / noise.length);
    // }

    _loadSound();
  }

  void _loadSound() async {
    final ByteData data = await rootBundle.load('assets/song_cut.wav');
    noise = data.buffer.asFloat32List();
  }

(...)

然后Dart中的此函数调用本机库的Dart包装器:

Then this function in Dart calls the Dart wrapper of the native library:

void start() {
    stream.start();
    var interval = (512000 / stream.getSampleRate()).floor() + 1;
    t = Timer.periodic(Duration(milliseconds: interval), (_) {
      stream.write(noise);
     });
  }

Dart中的包装器是:

The wrapper in Dart is:

void write(Float32List original) {
    var length = original.length;
    var copy = allocate<Float>(count: length)
        ..asTypedList(length).setAll(0, original);

    FfiGoogleOboe()._streamWrite(_nativeInstance, copy, length);
    free(copy);
  }

_streamWrite 是C ++中的本机函数:

_streamWrite is the native function in C++:

EXTERNC void stream_write(void* ptr, void* data, int32_t size) {
    auto stream = static_cast<OboeFfiStream*>(ptr);
    auto dataToWrite = static_cast<float*>(data);
    stream->write(dataToWrite, size);
}

void OboeFfiStream::write(float *data, int32_t size) {
    managedStream->write(data, size, 1000000);
}

现在我可以听到这首歌了，但它的失真太大了.尝试使用正弦时，我也可以听到它的声音，但是它也有些失真.我还没有在Oboe中使用回调模式，因为我想尝试一下是否先行.

Now I can hear the song but it comes out with too much distortion. When trying with the sine I could hear it too, but it also had some distortion. I'm not yet using the callback mode in Oboe, since I wanted to try if this worked first.

推荐答案

1-您的WAV文件采用哪种格式?是32位浮点数吗?不要忘记WAV文件具有标头，因此您应该丢弃前几十个字节(直到 data 段).请确保您开始在浮动边界上读取音频数据(如果不是标头，则它可能不是4的倍数).如有必要，只需使用十六进制编辑器来确定浮点数据的偏移量，然后从那里开始读取.或者，截短标题并将您的资产重命名为 song_cut.raw .Audacity应该能够产生无标题的原始音频文件.

1 - what format is your WAV file in? Is it 32 bit floats? Don't forget that WAV files have a header, so you should discard the first few tens of bytes (up to the data segment). Be sure that you start reading the audio data on a float boundary (which may not be a multiple of 4 if the header isn't). If necessary, just use a hex editor to ascertain the offset of the float data and start reading there. Or, truncate the header and rename your asset to song_cut.raw. Audacity should be able to produce a header-less raw audio file.

2-您的音频剪辑以什么采样率录制?这与设备的采样率匹配吗?(请注意，iOS设备通常为44.1k，而Android设备通常为48k.在macOS上使用Android仿真器时，谁知道报告的采样率是多少！如果您的采样率不匹配，则音调会失真-或使用重采样器.我认为Oboe有一个.或者，与演讲相关的样本回购包含一个您可以使用的回购.)

2 - What sample rate is your audio clip recorded at? Does that match the sample rate of the device? (Note that iOS devices are normally 44.1k, but Android devices are frequently 48k. When using an Android emulator on macOS, who knows what the reported sample rate will be! Expect pitch distortion if your rates don't match - or use a resampler. I think Oboe has one. Alternatively, the sample repo associated with the talk contains one you can use.)

3-请注意，将计时器间隔微调(出于演示目的)，以声卡速率传输512个样本所花费的大约时间.这对于演示可能是可以的，但对于现实生活却不可行.另外，您的wav文件中可能没有512个样本.要么将音频循环调整为512个样本，要么调整512000常数以匹配循环中的样本数.

3 - note that the timer interval is finely tuned (for demo purposes) to the approximate time taken to deliver 512 samples at the sound card rate. This might be ok for demos, but isn't for real life. Also, your wav file probably doesn't have exactly 512 samples in it. Either adjust your audio loop to 512 samples, or adjust the 512000 constant to match the number of samples in your loop.

4a-您尚未使用回调方法，但您可能应该尽快.我成功的一种方法是使用无锁循环缓冲区.Oboe回调尝试清空缓冲区，而Dart计时器例程尝试填充缓冲区.缓冲区越大，下溢的机会就越少，但是延迟就越差.

4a - You aren't using the callback method yet, but you probably should as soon as possible. One method I've had success with is to use a lock-free circular buffer. The Oboe callback tries to empty the buffer, while the Dart timer routine tries to fill it. The bigger the buffer the less chance there is of an underflow, but the worse the latency.

4b-理想的解决方案是将Oboe回调调用到Dart中，但是我还没有找到一种方法，因为C-> Dart调用必须在Dart主线程上，但是Oboe回调是肯定是在高优先级的IO线程上.

4b - The ideal solution would be to have the Oboe callback call up into Dart, but I haven't found a way to do that as C->Dart calls must be on the main Dart thread, but the Oboe callbacks are surely on a high-priority IO thread.

这篇关于通过Flutter从C ++插件访问资产的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！