How to compile Ruby from source on Windows for FS3

[trogluddite]

Well argued rlr.

It would be s shame to close off the opportunity to follow any advances in Ruby. For example, between the 1.8 version and 1.9 there was a significant increase in processing speed. And being stuck with the same .dll, potentially prevents even DSPr themselves from making changes and improvements in the custom FS classes and methods. That's awkward enough if you are a developer, and have to twiddle with your source code - but the end user of a plugin usually doesn't have that luxury.

Trouble upgrading needn't be because of anything as major as a rewrite of the Ruby core, either - it only takes one instruction in one place in a schematic not to work to break it, so even what might seem like a small change in the behaviour of one method can still kill a plugin if it is used in a critical part of the schematic. I have been caught out by this already myself when moving between FS (Ruby 1.9.1) and Sketchup (Ruby 1.8.7) - the differences are very subtle, but enough to have had me really scratching my head over why I'm not getting what I expect.
This seems, from what I can tell, to be mostly why the Sketchup dev's have been so slow to adopt 1.9.1, despite the performance advantages it could bring. They realise that the free plugins add massive value to their software - some are considered almost essential to use it efficiently - and they are cautious about disrupting so many user's workflow and losing their good faith.

[DaveClark]

rlr wrote:
In case the 2nd plugin would need a different ruby: Game over.

I agree that we are very limited to what can be done with rewriting the Ruby DLL, but this is because a linking was done to a specific DLL already --- specific name, specific EXPORTS --- not because of the "one Ruby interpreter per process" problem. Therefore, don't mess with the interpreter because of the name and EXPORTS.

I am personally not bothered by this --- I have no intention to redesign the language. If Ruby is updated and FS begins to use a new one (such as happened with 1.8 to 1.9), it should re-linked to a DLL with the new name such as "192," "193," with possibly new EXPORTS, etc. Now it could be that there is something about the way in which the Ruby developers compile and link different versions that could be problematic, but if so and this is known, please somebody speak up.

For bug fixes of current DLL's (that is, same name is kept but code changes), then with a proper installation process (not overwriting stuff at runtime) and with the latest DLL installed in its lib folder, all VST's use the fixed version. This should be considered an advantage, not a disadvantage.

---------

This "one Ruby interpreter per process" problem is actually something else quite different:

One cannot load more than one EXACTLY the SAME interpreter (same DLL name, same EXPORTS), yet for multi-processor systems, this is what one would like to do to scale the processing appropriately for multiple processors, the architecture that many of us now have. This is why I say that it "depends on what Ruby is asked to do." If you want to keep out of trouble, you probably should not request it to do a whole lot within a single plugin instance.

Regards,
Dave Clark

[support]

@Support..
We should NOT be distributing VST and EXE files though right?

You can - just don't modify the ruby dll. If we make any changes to the way things work you'd just need to distribute a new update.

There is a big chance VST plugins break other plugins without knowing.

Indeed but, as you pointed out, only if you change the Ruby dll. If you stick with our dll it's fine.

As Trog has mentioned before you could also get problems if there are class name (or other global name) clashes due to the sharing of same Ruby interpreter for VSTs. At the moment you just have to do your best to avoid these. Let's not forget that there are so many things you can do with Ruby in FS that don't involve creating your own classes, namespaces or globals - it's much more of a help than a hinderance.

That then leaves the issue of us changing the dll in the future. For exported plugins ideally you want them to stick with the dll they were created with. If we can't find a way to do this (say by statically linking) then there aren't many options. Either we need to hope that any future changes to the dll are backward compatible or older plugins need to be updated when a new ruby DLL is issued.

We could add a check to see if the version of the Ruby dll is different from expected. However, we'd need to think carefully about what would happen if it is different. Should it still load the plugin? Should it show a message? Maybe we can cross this bridge once we determine whether the shared interpreter issue can be overcome.

Even if we can get round the shared interpreter issue for VSTs, I'm not sure that you can fully future proof against changes to Ruby. The language is evolving and if we keep up to date then you could in theory load an old schematic into a new version of FlowStone which uses the latest version of Ruby and that schematic may not work quite the same. This is regardless of the shared interpreter, wrapping or other dll issues - just a consequence of the language moving forward.

I think that the core aspects of the language are unlikely to change. Fundamental types and classes will remain the same as will our own methods and classes that we use for things like drawing. So for the vast majority of people the evolution of Ruby is unlikely to cause them any problems. It's only those that use more advanced parts that *may* have an issue going forward. You either accept these small uncertainties and enjoy all the benefits or you stay away but you can't have one without the other - c'est la vie.

[DaveClark]

Hi all,

Frankly, I don't think that Ruby improvements are a problem. Consider an example:

Exports from FS version 3.x are linked to dspr_msvcr90_Ruby191.dll

Exports from FS version 4.x are linked to dspr_msvcr90_Ruby205.dll

From the perspective of the loader, these are DLL's not Ruby interpreters --- the O/S doesn't know or care about Ruby.

Changes to Ruby in this example do require changes to FS, but that simply means that releases need to be coordinated. Yes, people who use plugins from more than one version of FS that use more than one version of Ruby need to keep both versions of Ruby installed. This is nothing new. A clear example is prevalent in Linux --- look at the versioning and how this is dealt with. Ruby is following this.

The "one interpreter" problem is the far more general "one DLL named as such" problem, which is a processor-scaling problem arising from changes in architecture. Although this is something to worry about, it's mostly a red herring in the discussion of what to do about the Ruby interpreter (*). (For some versions of Windows, WinSxS is an approach for alleviating this "one DLL named as such" problem, but it has its own problems, some of which appear to be pretty serious.)

Having incompatible versions of DLL's out there with the same names should be avoided in all cases, not just for the Ruby interpreter.

------------

Now it's possible that the Ruby community has done something strange in compile/link that I'm not aware of, which is why I asked for somebody to "speak up." Without further explanation, I conclude that some people here are misunderstanding some of the facts, possibly leading to poor solutions and practices. If I am misunderstanding something here, please explain what this is.

Thanks for your attention, and I sincerely hope that a solution that is compatible with known good practices can be implemented.

Regards,
Dave Clark

(*) Red herring: If the solution to the Ruby interpreter was to have side-by-side DLL's as Support indicated they had tried, and if people go to modifying the interpreter, then yes, this "red herring" becomes a real problem. But I state once again that using the same name for incompatible DLL's needs to be avoided. It comes about in the "side-by-side" discussions because FS exports are linked to only one name, forcing the same names to be used. So: Do not modify the Ruby interpreter! Anyone who does this is going to cause problems for everyone, unless static linking can be made to work. This is why I suggested that the Ruby interpreter DLL needs to be placed where it will not so easily be overwritten.

[VPDannyMan]

Hi all,

Frankly, I don't think that Ruby improvements are a problem. Consider an example:

Exports from FS version 3.x are linked to dspr_msvcr90_Ruby191.dll

Exports from FS version 4.x are linked to dspr_msvcr90_Ruby205.dll

From the perspective of the loader, these are DLL's not Ruby interpreters --- the O/S doesn't know or care about Ruby.

Changes to Ruby in this example do require changes to FS, but that simply means that releases need to be coordinated. Yes, people who use plugins from more than one version of FS that use more than one version of Ruby need to keep both versions of Ruby installed. This is nothing new. A clear example is prevalent in Linux --- look at the versioning and how this is dealt with. Ruby is following this.

The "one interpreter" problem is the far more general "one DLL named as such" problem, which is a processor-scaling problem arising from changes in architecture. Although this is something to worry about, it's mostly a red herring in the discussion of what to do about the Ruby interpreter (*). (For some versions of Windows, WinSxS is an approach for alleviating this "one DLL named as such" problem, but it has its own problems, some of which appear to be pretty serious.)

Having incompatible versions of DLL's out there with the same names should be avoided in all cases, not just for the Ruby interpreter.

------------

Now it's possible that the Ruby community has done something strange in compile/link that I'm not aware of, which is why I asked for somebody to "speak up." Without further explanation, I conclude that some people here are misunderstanding some of the facts, possibly leading to poor solutions and practices. If I am misunderstanding something here, please explain what this is.

Thanks for your attention, and I sincerely hope that a solution that is compatible with known good practices can be implemented.

Regards,
Dave Clark

(*) Red herring: If the solution to the Ruby interpreter was to have side-by-side DLL's as Support indicated they had tried, and if people go to modifying the interpreter, then yes, this "red herring" becomes a real problem. But I state once again that using the same name for incompatible DLL's needs to be avoided. It comes about in the "side-by-side" discussions because FS exports are linked to only one name, forcing the same names to be used. So: Do not modify the Ruby interpreter! Anyone who does this is going to cause problems for everyone, unless static linking can be made to work. This is why I suggested that the Ruby interpreter DLL needs to be placed where it will not so easily be overwritten.

Considering that one can load their own DLL file and make calls to it from Ruby, there is little reason that you would "NEED" to edit the Ruby DLL anyways. The problem is and has always been that you "CAN" edit it and that the mechanism by which it is copied to the App data folder is fundamentally flawed. So, I agree, as Support says, even if it were statically linked and we had no opportunity to edit the Ruby DLL at all, then we still have lots of functionality. My total preference right now is that its statically linked with WIN32OLE working 100%. Then we know that the plug will not stop working because of a Ruby change, all the calls will be encapsulated in the VST/EXE. WIN32OLE working 100% because this is to be used on the Windows Platform and it should be supported 100%..

[DaveClark]

VPDannyMan wrote:
My total preference right now is that its statically linked with WIN32OLE working 100%.

I would go even further and strongly advise DSP Robotics to support the *full official* Ruby installation, assuming that this isn't already being done by default. (It appears that more stuff from the Ruby libs works than I was expecting, but I haven't fully tested the limitations.)

For those who don't understand what this is, please take a look. There is a matrix solver, compression --- all sorts of great stuff. It would also be good to *fully* support RubyGems --- i.e. support shared objects, again assuming that this truly isn't the case already.

Doing this would add tremendous capability to FS and greatly simplify many programming tasks.

Full support, however, means supporting a LOT more than just the interpreter DLL. Although static linking of the interpreter could still be supported as far as I know, the argument for "single-file distribution" more or less goes away due to the fact that more and more people will probably start using more and more from the Ruby library. Only the simplest of applications would be helped.

Regards,
Dave Clark

[trogluddite]

Now it's possible that the Ruby community has done something strange in compile/link that I'm not aware of

For a bit of light bed-time reading, look up "Ruby Global Interpreter Lock" or "Ruby Global Virtual Machine Lock".
Now, I'll admit to not having understood half of what I read - but yes, it seems that Ruby does have a particular mechanism built into it that prevents multiple interpreters per thread (and also making use of multiple processor cores).
Whether this affects only multiple identical interpreters, I really couldn't say, but I have seen it explicitly stated that one interpreter per process is a specific limitiation of the standard C versions of Ruby, as it was not written with embedding in mind.

This is not necessarily the end of the world - there are other branches of Ruby that are working towards a version compatible with the current standards that do away with the global lock - for example, IronRuby, JRuby, Rubinius Hydra. At the moment, none of them are fully up to the latest Ruby standard, and there are complication due to the need to ensure that calls to C routines are thread-safe (part of the reason for the lock in the first place). But if DSPr were to upgrade Ruby in the future, moving to one of these properly concurrent versions might be a good option.

I would go even further and strongly advise DSP Robotics to support the *full official* Ruby installation

It's a pity that we don't just get Ruby as part of our OS installation the same way that Mac users do. From what I've read, the Mac Ruby implementation has now also lost its GIL, so is capable of fully concurrent processing too.

[DaveClark]

Hi Trog,

I don't believe that the Ruby GIL will have any effect on versions of Ruby --- for example version 1.91 and the theoretical 2.05 --- as far as causing limitations on use of different versions simultaneously.

The GIL, as I understand or misunderstand it, keeps a single interpreter from working on more than one task simultaneously. This is not the same thing as having two different DLL's with different names (e.g. 1.91 and 2.05) from being loaded, each with different DLL's (e.g. VST's) linked to it.

The GIL for each interpreter locks that particular interpreter; there is no reason I can think of to lock the other one. They shouldn't even be expected to know about each other, unless there are some common shared objects which are somehow locked (for example in Ruby lib). In the case of FS as it currently exists, the only possible shared common objects have to do with Win32API, so even this locking of lib shared objects would not be a problem for just running Ruby without Win32API, RubyGems, and so on.

Now it's possible that the Ruby interpreter could be doing something weird like writing a lock file to some globally accessible place, but again there isn't any reason I can think of to do that. The purpose of the GIL is to prevent a single instance from getting "confused," not to shut out additional instances of completely different interpreters, which is how this should be viewed.

In conclusion, I still don't see any problem with different versions of FS using new versions of Ruby with differently named interpreters, but remain perfectly willing to be convinced otherwise.

Regards,
Dave Clark

[trogluddite]

Thanks Dave, I'm tying myself in knots trying to understand the stuff I'm reading, so I appreciate the explanations.
Think I'm getting mixed up here between the .dll versions problem, and the shared namesoace thing. where it's sharing the same .dll that's the problem (I think?)

[VPDannyMan]

Out of curiosity, (not trying to rush anyone) is there any word on this Support? How are you making out?