NN | SuperCollider 3.13.0 Help

Source: NN.sc

Description

Load torchscripts on scsynth. Tested with RAVE (v1 and v2) and msprior. Models are loaded asynchronously on the server, and stored in a global dictionary so that they can then be accessed by key.

Loading models

Models are loaded with a key to identify them and a path to a torchscript file: NN.load(\modelName, "/path/to/torchscript.ts")

The sclang interface instructs the server to load the .ts file, receives and stores models' info from the server, and keeps track of which models are loaded. Once a model is loaded, and its info received, it becomes possible to create UGens for processing.

Real-time processing

You can get UGens for each models' method like this: NN(\modelName, \methodName).ar(inputs, blockSize, debug, attributes)

Each NN().ar UGen is specific to a loaded model and method. This is because different models and methods require different numbers of inputs and outputs. Each UGen loads an independent instance of the model, to make sure independent inferences on the same model don't interfere with each other. For this reason, setting attributes is supported only at the UGen level.

Attributes

Torchscript can support settable attributes: NN(\modelName, \methodName).ar(inputs, attributes: [ attributeName: LFNoise0.kr(1), anotherAttr: LFTri.kr(1).range(0, 10) ])

NNUGen accepts a list of pairs (attributeName, attributeValue), and will set each attribute when its value changes. It might be useful to limit the rate at which an attribute is set by using Latch: NN(\modelName, \methodName).ar(inputs, attributes: [ temperature: Latch.kr(LFTri.kr(1).range(0, 10)), reset: MouseButton.kr(lag:0), ], debug: 1)

With debug: 1, NNUGen will print the attribute value every time it's set. The printed value is read from the model for every print.

NRT processing

In order to load and play with models on an NRT server, models' informations have to be stored in a file. This method is intended for running NRT servers without even booting a real-time one:// Prerequisite: // ask a running server to dump all currently loaded models' info to a file NN.dumpInfo("models.yaml"); a = Score([ // NN.nrt reads that info file and makes an osc bundle [0.0] ++ NN.nrt("models.yaml") { // load models NN.load(\a, "path/to/model_a.ts"); NN.load(\b, "path/to/model_b.ts"); // synthdefs: use .doSend(s) to get them into the bundle SynthDef(\nnfb) { |out=0| var fb = LocalIn.ar(1); var sig = NN(\b, \forward).ar( NN(\a, \forward).ar(fb) ); LocalOut.ar(sig); Out.ar(out, sig); }.doSend(s); }, [0.0, Synth.basicNew(\nnfb).newMsg], [30.0] ]).recordNRT( outputFilePath: "gen.wav", headerFormat: "wav", sampleFormat: "float", action: { "done".postln } )

In the NN.nrt() { ... } block, the syntax to load methods and create SynthDefs is almost the same as in real-time, with the only difference being that SynthDefs need to be "sent" to server with .doSend(s) instead of other methods.

A second method is available if models are already loaded on a running server. The following code creates messages for the NRT server to load all models currently loaded on a RT server, with the same indices, so that SynthDefs built on the RT server work also on the NRT one. The obvious drawback is that this method is more expensive in terms of resources, since models are loaded on both the real-time and any NRT servers that are launched.Score([ [0.0] ++ NN.models.collect(_.loadMsg).postln, [0.0, ["/d_recv", SynthDescLib.global[\nnar].asBytes]], [0.0, Synth.basicNew(\nnar).newMsg], [30.0] ]).recordNRT( outputFilePath: "gen.wav", headerFormat: "wav", sampleFormat: "float", action: { "done".postln } )

First-execution warmup

If after a model methods are very slow for the first execution right after the model is loaded, and then become much faster, it might be due to torchscript performing optimization during the first pass. NN offers a way of performing this first pass silently: { NN(\model, \forward).ar(WhiteNoise.ar) }.play // long dropouts at first, then fine { NN(\model, \forward).ar(WhiteNoise.ar, warmup: 1) }.play // longer silence, but no initial stuttering

Class Methods

NN.load(key, path, id: -1, server, action)

Sends a message to the server to load a torchscript file, and gathers model informations as the server returns them. This method should be use to initialize a new NNModel object.

Arguments:

key	a Symbol to identify this model object, and to access it after it's loaded.
path	the file path of the torchscript file to load. The path is standardized with String: -standardizePath internally.
id	a number that identifies this model on the server. Pass `-1` (default) to let the server set this number automatically.
server	the server that should load this model. Defaults to Server: *default.
action	function called after the model and its info are loaded. The callback function is given the model as argument.

NN.new(key, methodName)

This class doesn't construct any instance, but provides this as a convenience method for retrieving loaded models or their methods.

Arguments:

key	a Symbol that identifies the loaded model (see /Classes/NN#*load).
methodName	a Symbol. Optional.

Returns:

A NNModel, if called without providing methodName, otherwise a NNModelMethod. If the requested model or method is not found, an error is thrown. NN(\mymodel); // -> NNModel(mymodel ...) NN(\mymodel, \forward); // -> NNModelMethod(forward ...) NN(\blah); // sERROR: NNModel: model 'blah' not found NN(\mymodel, \blah); // ERROR: NNModel(mymodel): method blah not found

NN.nrt(infoFile, makeBundleFn)

From extension in /home/bgola/.local/share/SuperCollider/Extensions/nn.ar/Classes/NN_nrt.sc

Facility to load model information from a YAML file and create an OSC bundle suitable for loading models and SynthDefs on an NRT server. See NN: NRT processing.

Arguments:

infoFile	path to a YAML file which contains model informations. Such a file can be obtained from a running RT server with NN: *dumpInfo
makeBundleFn	a Function to be used to create an OSC bundle. All OSC messages sent from this function will not be sent to server, but added instead to the returned bundle. See Server: -makeBundle.

Returns:

an OSC bundle, a.k.a. an Array of OSC messages.

NN.model(key)

Gets a loaded model by key. Equivalent to NN(key), but it doesn't throw an Error if the model is not found.

Arguments:

key

Returns:

an NNModel or nil if not found.

NN.models

Returns:

an Array of all loaded NNModel.

NN.describeAll

Prints all loaded model informations

NN.dumpInfo(outFile, server)

Queries the server to dump all currently loaded models informations to a YAML file or to the console.

Arguments:

outFile	path to the YAML file to be written. If `nil` it prints to console instead.
server

NN.keyForModel(model)

Returns the key with which a model is stored in the registry.

Arguments:

model

an NNModel

Returns:

a Symbol, or nil if model is not found in registry.

OSC Messages

NN.loadMsg(id, path, infoFile)

Returns the OSC message for the server to load a torchscript file.

Arguments:

id	a number that identifies this model on the server. Pass `-1` (default) to let the server set this number automatically.
path	the file path of the torchscript file to load. The path is standardized with String: -standardizePath internally.
infoFile	the path to a file where the server is going to write model info. Defaults to `nil` which disables writing to a file (useful for NRT servers since they can't write to files).

NN.dumpInfoMsg(modelIdx, outFile)

Returns the OSC message for the server to print models info or write them to a file

Arguments:

modelIdx	an Integer that identifies an already loaded model on the server. Defaults to `-1` which causes a dump of all loaded model informations in the same output.
outFile	the path to a file where the server is going to write model info. Defaults to `nil` which disables writing to a file (useful for NRT servers since they can't write to files) and prints to console instead.

Inherited class methods

Undocumented class methods

NN.prCacheInfo(info)

NN.prGetCachedInfo(path)

NN.prPut(key, model)

NN.prReadInfoFile(infoFile)

From extension in /home/bgola/.local/share/SuperCollider/Extensions/nn.ar/Classes/NN_nrt.sc

Instance Methods

Inherited instance methods

Examples

(
// loading
s.reboot.waitForBoot {
    NN.load(\rave, "~/rave/model.ts", action: _.describe)
    NN.load(\prior, "~/msprior/prior_model.ts": action: _.describe)
};
)

// playing

// resynth live input
{ NN(\rave, \forward).ar(SoundIn.ar) }.play

(
// blockSize: 0 disables aync processing
// processing is now synchronous, but it can generate more dropouts
{ NN(\rave, \forward).ar(SoundIn.ar, blockSize: 0) }
)

(
{
    // drive with a sine wave sound
    var in = SinOsc.ar(MouseX.kr.exprange(20,20000));
    // encode input sound to latent space
    var latent = NN(\rave, \encode).ar(in);
    // modulate latent space trajectories...
    var mod = latent.collect {|l| l + LFNoise1.ar(1).range(-1,1) };
    // resynth
    NN(\rave, \decode).ar(mod);
}.play
)

(
{
    var in = WhiteNoise.ar();
    // encode input sound to latent space
    var latent = NN(\rave, \encode).ar(in);
    // generate new latent codes with msprior
    var prior = NN(\prior, \forward).ar(latent);
    // resynth latent codes to sound out
    // dropping the last prior element because msprior returns perplexity as 9th output
    var resynth = NN(\rave, \decode).ar(prior.drop(-1));

resynth;
}.play
)

helpfile source: /home/bgola/.local/share/SuperCollider/Extensions/nn.ar/HelpSource/Classes/NN.schelp
link::Classes/NN::

NN : Object Extension

NN.load(key, path, id: -1, server, action)

Arguments:

NN.new(key, methodName)

Arguments:

Returns:

NN.nrt(infoFile, makeBundleFn)

Arguments:

Returns:

NN.model(key)

Arguments:

Returns:

NN.models

Returns:

NN.describeAll

NN.dumpInfo(outFile, server)

Arguments:

NN.keyForModel(model)

Arguments:

Returns:

NN.loadMsg(id, path, infoFile)

Arguments:

NN.dumpInfoMsg(modelIdx, outFile)

Arguments:

NN.isNRT

NN.nextModelID

NN.nrtModelStore

NN.nrtModelsInfo

NN.prCacheInfo(info)

NN.prGetCachedInfo(path)

NN.prPut(key, model)

NN.prReadInfoFile(infoFile)

NN : Object
Extension