Ruby and Unicode

[KG_is_back]

How can I get unicode characters into ruby/flowstone in general?

if I try to copy-paste them, they turn into "?" I have tried using force_encode method in ruby but with no luck... Namely I want these characters to display "˂˃˄˅"

[tulamide]

Unfortunately you can't. FS Ruby is a stripped down version of the full Ruby, and unicode is one of the victims. It was one of the first questions I made here, and it was legendary Trog that explained it to me.

[KG_is_back]

too bad... well, I'll have to use <>^v instead. It's only for "default" place-holder gui anyway... Final version will have an option to replace "solid background with <>^v char arrow"-button with bitmap one, so no big deal...

[psionprime]

Speaking strictly about FS 3.0.8 and it's ruby engine being 1.9.3 (labeled 1.9.1 for ruby internal compatibility):

FS ruby is the full version of ruby per FS user guide page 233 with a minor addition to integrate ruby and the FS framework. That said ruby 1.9.3, as a language can handle unicode with its default code-page being US_ASCII. The previous version was plain ASCII. The version >=2.0 default to UTF-8.

The FS text controls appear to use defaults so pasting from a different code-page source can produce strange results. A couple of suggestions (which I have no idea how they may help or not depending on what you are doing):

- check the forums and simply ask how non-english language people use FS
- you can change Windows country locations so the entire system defaults to a certain code page
- write translation code in ruby to take what is pasted, parse and translate then replace what was pasted with the translated text, eg, some UTF that looks like US-ASCII but has different base numbers and you know for sure that is what it is so you can use a table, translate and replace; it should be fairly invisible to the user
- use the win32 api and ruby to create your own control that will accept native code page characters - lots of coding but that is how legacy programs are developed.
- google: ruby 1.9.3 unicode support

A lot depends on what you need to accomplish vs the time/cost investment.

As an aside I am a little concerned about the version. ruby 1.9.3 is pretty solid...but its support lifetime was ended February this year. That sounds like a decent reason to move on to 2.x, better threading/performance etc. I've poked DSP a bit offering to help port to Linux and Mac /shrug I'd like to see it happen. ruby is supported on those platforms and makes a lot of sense to support them.

Good Luck !

[KG_is_back]

I personally never got problems with any characters that can be typed directly form keyboard, including any special characters used by European languages I'm aware of. Namely " áéíóúýäüöŕĺňřťďčšžôěđĐŁß" and their upper case versions. They sometimes cause problems when you accidentally type them into ruby component.

[tulamide]

FS ruby is the full version of ruby per FS user guide page 233 with a minor addition to integrate ruby and the FS framework.

You will notice that it refers to the kernel only. The full version of Ruby 1.9.3 comes with a set of libraries that are not part of the FS Ruby installation and can't easily be integrated.

That said ruby 1.9.3, as a language can handle unicode with its default code-page being US_ASCII. The previous version was plain ASCII. The version >=2.0 default to UTF-8.

That's what I meant. You don't have access to full unicode. Here's what a full Ruby 1.9.3 installation supports: https://en.wikibooks.org/wiki/Ruby_Programming/Encoding
In fact you don't even have full utf-8 support. Just 256-byte ascii.

Try

Code: Select all

s1 = "\u0044"
s2 = "\u25B3"
s3 = "\u00CA"

The first one will correctly display the letter 'D', but the second one refuses to display '△', and even the third one, 'Ê', doesn't work, although pure utf-8.

Trog once guessed the changes were made because FS itself only works with ascii and so no compatibility issues can occur. I agree to that assumption.

[tulamide]

They sometimes cause problems when you accidentally type them into ruby component.

That's because of CodeRay. It interprets everything you type in a RubyEdit as a string with ", which means you would need to escape that letters (which doesn't make sense, of course).

The german umlauts are interpreted as some file access, which is really annoying when you want to comment something in German.

[KG_is_back]

The german umlauts are interpreted as some file access, which is really annoying when you want to comment something in German.

Yeah, german is relatively fine though - you can always use ae/oe/ue instead of ä/ö/ü and ss instead of ß and keep the text sensible and grammatically correct. AFAIK, there is no such notation for special characters in Slovak (or any European language with phonetic orthography that I know of) Just try to put "Ľan","Laň" and "Lán" into google translate and guess which one I mean when I write "lan"
Digitalizing speech is always about trade-off, I guess... phonetic orthography simplifies writing of spoken word and reading with correct pronunciation to a level where basically a parrot could do it. On the other hand, exhaustingly big alphabets have proven to be very difficult to digitalize once digital transmission (starting with Morse code) became go-to approach. Perhaps if countries with phonetic orthographies have had the technical leadership in early telecommunications and computers, the problem could've been solved and standardized much earlier and we wouldn't be having these "oh God why programs stick with ASCII and not support Unicode"-type of conversations in 2015.

[tulamide]

The german umlauts are interpreted as some file access, which is really annoying when you want to comment something in German.

Yeah, german is relatively fine though - you can always use ae/oe/ue instead of ä/ö/ü and ss instead of ß and keep the text sensible and grammatically correct. AFAIK, there is no such notation for special characters in Slovak (or any European language with phonetic orthography that I know of) Just try to put "Ľan","Laň" and "Lán" into google translate and guess which one I mean when I write "lan"
Digitalizing speech is always about trade-off, I guess... phonetic orthography simplifies writing of spoken word and reading with correct pronunciation to a level where basically a parrot could do it. On the other hand, exhaustingly big alphabets have proven to be very difficult to digitalize once digital transmission (starting with Morse code) became go-to approach. Perhaps if countries with phonetic orthographies have had the technical leadership in early telecommunications and computers, the problem could've been solved and standardized much earlier and we wouldn't be having these "oh God why programs stick with ASCII and not support Unicode"-type of conversations in 2015.

Google Translate is overchallenged by this. It could translate the first 2 but translated the third from "Lán" to "lán"
I also tried dict.cc, which offered replacements because of the lack of "Lán": the offers were "Laň" and "lokálna sieť" which was translated to "local area network" and therefore "LAN". I guess I would need to give up - and I see the issue. We were lucky enough to have realized two letter replacements (crazy enough the word used to express the letter ß means "sz", but they chose "ss" for the replacement) for the problematic chars, but it were just a few compared to other languages. That's why I totally agree with your last sentence!

[KG_is_back]

Google Translate is overchallenged by this. It could translate the first 2 but translated the third from "Lán" to "lán"

"Lán" is a field (of grain)... strange that dictionaries can't figure it out... it's not exactly a common word though. Another example where Slovak language fails to keep up with modern age and thus is slowly going extinct. Even words that have perfectly valid slovak counterpart (which was in common use just 5-10 years ago) are being pushed out by internationalisms (for example "riaditeľ" -> "direktor"). Also Most slovak dialects are basically dead. Just 30years ago western slovak was commonly spoken by entire western part of Slovakia (excluding Presburg, the capital). Now it's basically endemic to 30km radius around Trnava and people just 5years younger than me no longer use it in common conversation. There are complete sentences they fail to understand me when I speak "echt trnafsky". Because of that Slovak folklore (especially the western one) is basically dead or mummified at best, preserved by few enthusiasts - traditions are discontinued and innovation is not happening.
It's very sad... damage that failed to happen during 1000years (when we were part of Hungarian empire and Slovak language was more-or-less banned) was done just in 10years of us being in EU (this time unintentionally and with no effort at all). The trend is obvious to common public and it's a main reason why Slovakia is rather xenophobic by EU standards (e.g. the recent refusal of immigrants).
And yes, poor support of Slovak special characters in digital world has its part of guilt in all of this. Just imagine the annoyance and awkwardness when writing an SMS and your phone is incapable of writing "ť" and "ž", which are like in every other Slovak word (note: this issue is fixed now, but was rather a standard in early 2000ths)... it undermines national identity in the most subtle way.