About value inaccuracy

[tulamide]

value_inaccuracy.png (6.08 KiB) Viewed 21965 times

What do you see there? You see a value from an array. The first shows the value as calculated within Ruby. The second shows the same value after having been sent out to a green array and then fed back to Ruby.

I'm not really interested in what value is the correct one. I'm only interested in consistency. I have to rely on the float being the same wether in green or in Ruby. This is for same bit-depth floats, of course, as it is the case here.

[TheOm]

Ruby uses double precision floats, green uses single precision.

[tulamide]

Another reason to take care of consistency.

If Ruby can't exchange values with green, then we need a new preset management with direct Ruby value input, for example. No, there must be one way to represent a float in Flowstone, wether single or double, I don't care. It just needs to be consistent!

[TheOm]

I don't know why this is so important to you, but you can force rounding to single precision in ruby like this:

Code: Select all

def to_single_precision(x)
  [x].pack("f").unpack("f")[0]
end


[tulamide]

Thank you very much for the workaround.

But it's only a workaround. Flowstone should be consistent.

Why it is important? Well, a quick example: You have an instrument that works with samples and stores samplepoints in an array. This needs to be saved via preset. Of course, VST only allows the value range 0-1, so you divide by whatever you came up with. Now lets say, you work with 2 second long mono-samples. That's 88200 possible values, which makes the float's differences very small, and accuracy becomes very important.

In result, the values in this example would be calculated correctly in Ruby, but then you are forced to output them to green, in order to attach them to the preset management - at this point the values to be stored aren't correct anymore.

[KG_is_back]

The inconsistency has a very good reason. The VST format is build to use 32bit floats for both audio and parameters. In case of audio, the 32bit precision itself is actually an overkill - there is no device on this planet which could measure voltage/current at 44kHz sample-rate more precisely than 24bit integers (24bit int format has maximum SNR (signal to noise ration) of 150dB - most devices struggle to reach 120dB SNR).
Why VST format also uses 32bit floats also for parameters is a mystery, but probably has something to do with consistency. 32bit float in range 0-1 was probably meant to represent something like a continuous knob position.

32bit float can precisely store 24bit integer. It is possible to split more precise value into multiple 4bit ints and then store them separately.

[tulamide]

Hey KG,

good to see you around here again

I explicitly said that I don't care for single or double precision, but that both Ruby and green seemlessly work together: "No, there must be one way to represent a float in Flowstone, wether single or double, I don't care."

Also, it doesn't help that a float can represent ints to a certain degree. VST only uses the range 0.0 to 1.0

It's good that you see ways to manage it, but all I was saying is, that this should be done by Flowstone, so that you as a designer can rely on 0.00123456789 (or whatever value is representable) being exactly this value, wether you send it from Ruby to green or from green to Ruby.

[tulamide]

Unfortunately I can't tell what I'm actually doing, nor give any examples.

But I'm also aware that I will never see consistency in Flowstone (which is not good!).

So, to the real Ruby Gurus, TheOm, MyCo, Exo (should you read this), Trog (if you're secretly reading here, just pm me, I'll keep my mouth shut):

How do I constantly calculate in Ruby with single precision floats. Converting a double to single doesn't help. Without being able to give exact examples, it is hard to tell, so please try to imagine that I have some integer value, say,

100

this will be normalized and results in
0.7874015748031497
(here the divider is 127)

it is send to the preset manager as a parameter, which sends the value back to Ruby. But now the value is
0.787401556968689

calculating back an integer from it results in
99

(note: I can't round to nearest it must be floor)

Since there's information lost on the way from green back to Ruby, I can't recreate the original value, which of course ends in a bunch of consequential errors (for example when comapring for equality, or in displays where the user sees two values, first 100, then 99, etc.)

I can't recreate lost information, so it seems to me that calculating with single precisions all over the code would be the only way out. So, how do I tell Ruby, that the 99 is actually a 100. Don't tell me to add 1. On other int to float conversions via division the result is the same for Ruby and green, adding 1 there would falsify the results again.

Or alternatively, how to tell Ruby that 100 divided by 127 in this example results in a float that green can safely use, BUT also this float will result in 100 when multiplied with 127, when the float is fed back to Ruby? And that with rounding to floor as only option?

I worked with a divider that used way less bits and that worked like a charm. Unfortunately, the range is then 0.0078125 to 1.0, instead of 0.0 to 1.0, and that was not accepted, although the count of individual numbers didn't change. I have to provide a range where 0.0 and 1.0 mark the start and end.

It's really annoying. And I probably already said more than I was allowed to. Still, if anyone gets what I'm saying and knows a bulletproof solution, I will listen.

[martinvicanek]

Have you considered using integers?

[tulamide]

What does "preset manager" tell you?