handling float resolution in green

[tester]

floats have their resolution, thus accuracy in terms of decimal representations and calculations.

I recently encountered one issue like that, and I thought this little fix will help.

Apparently I still have cases, when it's not working.

Any ways to handle this correctly?

[Spogg]

Tulamide is good at this stuff!

But in the meantime my schematic demonstrates the rounding behaviour of floats. I don’t think we can correct for the limited precision of the float system. Something to do with 32 bit binary representation of decimal numbers or something.

[tulamide]

One thing you will never have, when dealing with floats, is exactness! As Spogg said in his schematic, the display of a float is just a rounded version of the real value.

So, if you want to get comparables, you need to first round yourself, and that means to decide to how many digits you want to round to.

I've taken two of your values into Ruby (which has an exact display of the value and works with double precision floats, and your real values are

65321.6484375
65321.6015625

That's a significant difference. Now, rounding is possible in three types: Ceiling, rounding, flooring. If you'd decide to floor to one digit, the result would be .6 for both, and ceil to one digit would make it .7 for both. rounding would floor anything that's below half and ceil everything else. In this case, you'd get away, they'd both end in 0.6. But if the first was 65321.6584375, you'd end up with .7 for the first and .6 for the second.

Flowstone's float displays use floor for the number display, but do not actually round the stored number. If you don't have access to prims (or don't want to use them), the easiest way to perform a floor rounding is to multiply the number with 10^(amount of digits). For example, for one digit, that's * 10. The you send it to an integer, which gets rid of any remaining digits, then back to float and divide by 10. For two digits, the process is the same, but you multiply by 100 (10^2) and divide by 100. Etc.

Note that this will NOT give you exact values (a float will always pick the nearest representation it can do, if it can't represent the real value), but it will make sure, both numbers have the SAME inexact value. I show this with your second number pair in the schematic.

[tester]

My basic question is:

if flowstone displays floats with 6 digit aaccuracy, then I'd like to trim/cut both values in the pair from the remaining invisible rest, and get coherent 6 digit value per slot. It doesn't have to be precise recalc, it just has to be 6-digit equality.

What kind of rounding should I use, to get the same end-result as I see in the float prim? Is there a ruby routine for that?

If I send these values through string instead of int, the opposite happens - bottom example gives expected result, and top example not.

[tulamide]

I told you how to do it. 10^(amount of digits)

That's all there is to say about it. Also, in your example schematic the values weren't equal, so why do you think you would make them equal by just using the same amount of digits? That's not how math works.

If I misunderstood you, please provide a written example with exactly the properties you have at the start, and the outcome you want.

[tester]

In case you didn't notice:

The first thing what I'm doing in my trimmer - I'm converting it to 6 digit value without comma, by multiplying it by 10^number_of_digits. But when the int-like value goes to int prim (to round to the floor) - it's either rounded to floor or not. So I don't get it. But if flowstone can display it, then it should be doable somehow.

So either I don't understand your english (which is not my native language) or I'm missing something from what you are saying.

Apart from that, this routine should work for any range of two similar numbers (within 6 digit range), either very small or very large.

[tulamide]

Your routine wasn't fail-safe, that's why I made a schematic with a simpler approach, that doesn't fail. I think, what you might have missed in my explanations, is that the display of the float prim is just a display. It is not a number. The string version can be shortened in totally different ways, because it is not bound to the float format (of course, since it's a string). But the stored number is still the number with lots of digits, you just don't get to see it. There is no way to make the float have a specific value, that it can't represent with its matrix.

Therefore you have to decide, what you want: either two clean looking strings, that both say "x.6", although they are different numbers, or two floats that are rounded to the nearest representation of x.6, which might be x.60012635 or something the like.

You for sure have to part the visual and the calculation, it won't go together. That's due to the mantissa/exp matrix of a float. It can represent a lot of numbers, but not all. And those exceptions can't be made x.6 (to stay at this example) magically. Imagine a float like being a long list of numbers. Whenever a float is given some value, it looks at this list. If the value is on the list, it will become the exact value, if not, it will choose the closest value in the list.

I hope I could make it more understandable this time. People often think of a float as a number. But that's wrong. It's a formula that's calculated each time the float is used in an algorithm. Hence the image of a long list of numbers.

[tester]

Nevermind for now, testing tomorrow.

Apparently, the format prim handles 0.5 endings differently than ruby.

[tulamide]

Nevermind for now, testing tomorrow.

Apparently, the format prim handles 0.5 endings differently than ruby.

Yes, any prim related to float will handle things differently. As I mentioned earlier, Ruby uses 64-bit double precision floats.

[Spogg]

All good stuff!

In my post I missed an important point which tulamide made, namely the Float prim with an input is actually just a rounded display. I tried to enter a high precision number into the Float prim directly and in that situation the rounding is applied to the entered value correctly and displayed accurately and I couldn’t produce a weird result in the subtraction.