Architecture A2: Test models are available

Progress is coming along on work to develop Architecture A2, NAM's update to the "standard WaveNet" and its lite/feather/nano variations (retroactively, "A1").

NeuralAmpModelerCore v0.4.0.rc2

First, I have published NeuralAmpModelerCore v0.4.0.rc2, a new release candidate that supersedes what I shared previously. Please update to this version of the library.

Stage 3: Test models are ready!

Second, I have completed the first part of Stage 3 of the project: I have created a set of test models whose performance I need builders to evaluate on their hardware. These models require NeuralAmpModelerCore v0.4.0, so you must implement the new core on your device to be able to participate.

You can download them here: Google Drive.

More on what to do with them below, so read on. Please aim to deliver feedback to me no later than Friday, February 13th. Get in contact if you are having trouble--the earlier I hear from you the better.

What's inside

You'll find a number of folders inside which there are .nam models that exercise the various potential A2 features. If you can run wavenet_a2_max.nam, then you should be able to run any of these models. If not, please file a bug report or let me know otherwise.

The A2 test model directories

There are 165 models in total. Note that these models haven't been trained, so they won't sound like anything--in fact, their outputs might be very odd! The focus right now is on computational performance. For that, I need you to profile these models.

I understand that profiling all of these models is a lot to ask for. If you can only do a few, please focus on the "activations" and the "channels" folders. These should provide me with the bare minimum information, and I'll do my best to fill in the gaps. However, the more you can do, the better I can accurately model your hardware's performance.

How to evaluate the models

I need you to help me understand how these models' needs compare to your hardware's capabilities. For example, I used NeuralAmpModelerCore's benchmodel tool to compute how long it took the models to process 2 seconds of audio at a sample rate of 48 kHz with a buffer size of 64 samples. I ran each model 20 times, then computed the median and tabulated the results into a CSV:

Example profiling results

I need you to do this (or something similar) and share your results with me.

Also, if I were you, I might also have some idea of what my threshold is--perhaps a CPU usage of 5% is my limit (so, 100 ms). Please also share this in a way that I can compare it with the metrics you report.

Some models should be very easy to run; others will probably be too heavy. This is on purpose--I want to see what crosses the line. Even if a model's real-time factor is below 1, please quantify it for me!

My tools are open-sourced

I wrote some tools to help me automate this work. They are open-sourced and you can get them here: https://github.com/Atkinson-Advanced-Modeling/ProjectA2Tools

Like I said, I'm just using benchmodel from NeuralAmpModelerCore run on my local computer; you'll want to adapt this to your testing platform.

My profiling code running all of the test models

Example visualization of results

Conclusion

This stage is at the heart of what will make A2 work. A1 was developed in isolation; the feedback I'm provided on these test models is how I will ensure that A2 will work for everyone. Again, please aim to deliver feedback to me no later than Friday, February 13th. Get in contact if you are having trouble--the earlier I hear from you the better.

In the meantime, I'm going to start developing the tools that I'll be using to optimize A2. I'll prototype with my own data, but I'll incorporate your data as I receive it.

This is a very cool project to me. As this work will be open-sourced, I'll also be blogging my findings along the way. My hope is that as I start figuring out what works (and what doesn't), you'll be able to prioritize the work on your side to support A2--we'll converge together on the answer. So please stay tuned!

Lastly, I just wanted to say thanks again. The interest I've received in A2 has been incredible so far; I'm very excited to provide A2 as a resource for the industry..

Onward!