Problems with COM port - (Fixed)

[vpnsoluk]

I am evaluating FlowStone and once i worked out how to use it I ran into some fundamental problems:

Firstly, COM ports with large enumerations don't work.
If I set the device to COM22 it fails, if I set the device to COM1 it works.

More of a problem, COM ports are simply broken.
If I set the baud rate to 921600 and use character mode, the data comes in at about 100cps (yes 1000 times too slow) and FlowStone crashes shortly afterwards. I guess internally there is a task switch for each character and windows simply can't work that way.

If I set a line terminator to ASCII 13 then I get just one line.

I've found that connecting the receive trigger to the transmit trigger (even though the incoming data is a stream and needs no trigger) then I get several lines - except it is the same one every time!!!

If I then put this data onto a string stack, FlowStone crashes after a few hundred (...um identical) lines.

Am I doing this right?

Cheers,
Jason.

[vpnsoluk]

I should have included, the logging needs an option for non-verbose.

Currently the received data is also logged which creates too much clutter in the log and in my case also causes a crash.

Surely there is no need to log received data if the data is also available on the output pin?

[admin]

Is there a reason why you use such high comport numbers?

What is the highest number you found it works?

We will certainly look into the high numbers issue and see if we have the same problem?

The limit of sending/receiving data in FlowStone is 100 times a second, so if you send your data one character at a time that is your maximum data rate! Which is why you should use strings, using string you will achieve much higher baud rates!

The Terminator on the Comport module is only for incoming data, ie. So we know when to output a string from the module. For sending there is no terminator necessary as you use a trigger to send the data string. That is not to say that the string may well have a terminator on the end so that the receiving Com port also knows when to output its string.

Baud Rate - the max baud rate of an COM port is 115k you are trying 921K ?

We use String Queues (Stacks) all the time to buffer incoming data until you are ready to process it. I have never seen it crash? Maybe you are running out of ram if you are just leaving it and not popping any data?

Also the data log on the Com port module is for debugging only, so you can see everything that is happening.

[vpnsoluk]

Hightest number I found that works is 1, I did not try any others.

High numbers are caused by Windows rubbish management of USB-Serial interfaces that means the next available com port is incrememted each time a new USB device appears on each USB port. 5 different usb devices pluged into 5 different usb ports (even one at a time over some time) would eventually result in 25 com ports being allocated. My laptop is up to COM23 at the last check.

You can manually re-configure each port when it is plugged in (if you ignore the warnings) but it does not resolve the problem. There is a registry hack you can do to reset the count.


Machine has 4Gb of RAM. Windows supports baud rate in excess of 2MBPS, so FlowStone should support that too.
It certainly should not crash if its buffers overflow, it should silently drop data.
I suggest you do some tests on fast com ports to see what I mean. Do you have any examples of string extraction as what I have tried so far fails to work - I get the same data over and over for about 5 seconds, then a crash.

[admin]

We have conducted intensive tests here on two PCs and although we have seen some strange windows issues FlowStone performs well and we didn’t have a single crash!

Actually Windows Com ports have a maximum baud rate of 128000 bps, this is the max and FlowStone fully supports this. However in our tests the max reliable data rate was 115.2K as higer than this and com port cables don't work! This is due to the single ended cable, which is why everyone says 115.2K is the Max data rate for RS232 com ports. Remember the actual data rate is the baud rate /10. So 115.2K = 11.5K (since each data word had 10 bits).

You can check this by going into the Windows Device Manager, Ports (Com & Lpt): Double Click on your com port, Port Settings, and you will see it goes from 75 – 128K.

You can also go into Advanced and change the port number 1 – 256, but if you choose a high number the Com port doesn’t work! This is a Windows issue not FlowStone!

<Edited>
As far as high data rates are concerned we did see some issues with lost data if you provided the Comport Module with data faster than windows can deal with it.

In our test today we were able to send the data as fast as we liked to the FlowStone Comport Module (1000 strings 100chars long – sent in <10ms), and it buffered the data and sent it out as fast as the baud rate would allow.

It appears that the receiving of RS232 data is where you can loose data not the sending. Basically if Windows is busy then data can be lost! For example if you display the incoming data on the screen in real-time windows struggles to keep up with the screen updates and can loose data. In our tests it took the same time to send the data @ 115.2K as 57600 about 18 seconds for 100,000 characters =5.5K words/second = 55.5Kbs.
I have included the FlowStone Tx and Rx test examples that we used to talk between two PC’s, so that you can test your system performance.

Basically if you are sensible and provide the data at a rate that Windows can deal with then there is no data loss and everything works perfectly.

We did also see one time when we sent too much data the Comport hardware driver crashed and we had to disconnect and reconnect it to bring it back to life.

The bottom line is if you want to send data faster than 115.2Kbs (11.5K actual), Com ports are not what you should be using! Maybe consider using TCP or UDP both of which are in FlowStone Pro.

[vpnsoluk]

Clearly you don't want to fix your software and this really is your problem. It is a pity as it otherwise would fit our purposes.

You are wrong about hardware limitations and wrong about the windows API and even wrong about the definition of baud.

To cover the last point first, Baud is the same as data bits per second. Or, for 8N1 Async data, 10 times the data rate in characters per second.

Older version of windows were limited to 115k baud, contemporary for the time. This was due to the slow ISA bus and limitations of semiconductor technology. And indeed this is reflected in the windows serial API where the enumeration of baud rates stops at 128Kbaud.

However, the max baud rate supported by Windows these days is way in excess of this. The API supports programmable baud rates (if you want to read up about BAUD_USER in the COMMPROP structure) as used by the fractional divisors in modern chips. Just because the windows port settings dialogue box does not list a baud rate does not mean its not available.

Also, for hardware limitations you are incorrect (btw I am an electronics engineer with 30 years experience). Even standard 16C550 compatible UARTS can run to 1MBaud, (though 921600 is often used due to available clocks).

Of course, that is not to say that a given serial port is using the standard driver or 16550 hardware and also making the assumption that the async link is RS232. The same Windows API would be used for any serial port.

Electrically RS232 can easily run at 1MBPS over short lengths dispite being single ended, just check out the datasheets from Exar and Maxim, e.g. SP3232. I've seen it run at 2MBPS without issues (see Maxim MAX13237). If you go differential you can go much faster. RS422 and RS485 will run async to at least 50MBAUD.

Of course all this is academic. The device I am talking to is really USB, there is no RS232. It is an emulation of a 16550 UART purely from a software perspective. It uses the USB CDC (UART) protocol which is implememted by usbser.sys, a standard windows serial port emulation driver. The baud rate is there purely as a software speed limit. Theoretically it could run at 480MBaud under Hi Speed USB, assuming the software at both ends can keep up.

We can test this (and I do, daily): I can write a short Python program that will print data at the rate required (921600) without any issues, and Python uses the Windows API. It has no clue how the hardware is really connected.

Take a look at a terminal emulator, e.g. Realterm or Putty, these will list the standard 16C550 baud rates. Though you can type in anything you like assuming the fractional divisor on the chip can approximate it.

[admin]

We would appreciate it if you could keep your posts professional and don’t keep going off in such an aggressive manner. We are all engineers here too and are trying to help you!

If you read our post again you would see that we are talking about exactly the same points?

On Windows traditional RS232 type comports are limited by the driver! Our definition of baud rate is exactly the same as yours? Windows can’t keep up with high data rates. Etc.

Of course PC’s can handle higher data rates, ie. Gigabit Lan etc. but this is handled at a low level and Windows, which is what we are talking about here, can’t handle these data rates at a high level.
FlowStone is both high level (Display, Gui, Keyboard, Mouse,etc.) and Low Level (Networking, Audio Engine, DSP, Video Engine etc. ). The only time speed issues occur is when you want to communicate between the low-level and the high level. Clearly no High level software can display data at > 50-100hz as that is the speed of the graphics refresh.

So if you have a different piece of USB hardware that requires a low-level driver to handle the higher data rates, perhaps you could take the time to explain:

1) The Application?
2) Exactly what hardware you are using?
3) How you want to process the data?
4) What the result or output should be?

and we will see if we can help you?

[vpnsoluk]

I am not being aggressive, just exact. That is just the way engineers are.

There is no built in limitation in the windows drivers other than the types of UART supported (i.e. the 16C550 and its friends). Strictly, the rate limitation is down the the serial chips internal buffer size, the maximum rate at which unmasked interrupts are processed by the CPU and if DMA is used by the driver.
All PCs these days use the chips I listed (or clones/emulations of them) and those chips are supported by the drivers.
Many PCs come with specific drivers for the motherboard vendor chipset, then all bets are off, so long as a minimum level (i.e. the 16C550 an clones) are supported.

If is your choice to believe what you want, but if what I state is not correct then perhaps you can explain why my PCs work at all?

To answer your questions:
In this case, the external hardware is an embedded computer that presents itself as a high speed USB peripheral. It declares itself as a USB CDC (UART) device. This is terminated by the PC's USB and exposed to the operating system using usbser.sys (a standard windows component) as a standard serial port (So limitations here are down to those imposed by USB) and seen by the OS as a standard (i.e. 16C550) COM port controlled just like any other.

If you want to build something similar, then use one of the USB-RS232 devices e.g. Prolific, using chips by SiliconLabs or FutureTech. Note the speed limitation is more in the RS232 line driver used in the specific OEM's device than limitations of drivers.

By the way, my £10 Prolific USB to serial converter works at upto 1MBPS as is used at the same time as the interface described below.

The custom peripheral prints state information over the serial port as a CSV data stream. I process the stream in real time and display the results graphically. After failing to achieve this in your software, I now have this working in Python successfully. I have 15 graphs on the screen with 18 traces recording a history of about the last 6 seconds of time.

I update the screen display 125 times a second, (yes, I know internally DirectX and the video driver maps this rate onto the true rate of the screen, but just like in gaming this is transparent) 125 times a second is used as a speed limit, set by the software, as the CSV lines update at 111.11 lines per second (the ADC sample rate)

I was originally wondering if your software would do this as i have done similar things in Labview in the past (but with far less graphs) That time the data was real time data send serially from a scope and used to do FFTs.

[admin]

Ok I see what you are trying to do now, I stand corrected on a couple of points:

The Comport driver in windows is limited to 128K in the device manager only!(we have several USB – Comport convertors and they are all the same 128K). However you can in Flowstone ask for baud rates above recommended 128K and you are correct they do go faster (much to my surprise, all be not reliably in this case)!

We did some more tests on the FlowStone comport only this time in an standalone EXE, not in the development environment, and yes it did work 100% up to the recommended 128K, so any previous dropouts must have been due to the Comport module in combination with the development Environment and not in the cables as I suspected. I assume you are using the FREE version of FlowStone?

Here are the results of our EXE test:

56K 100,000 chars 100% accurate = 19s ~ 55.5K Actual (including buffering)
115K 100,000 chars 100% accurate = 9.5s ~ 105K Actual (including buffering)
128K 100,000 chars 100% accurate = 8.4s ~ 119K Actual (including buffering)
256K 100,000 chars 90% accurate (some dropped strings) = 4.8s ~ 208K Actual
(including buffering).
512K – only 20% came through!

I must add still not a single crash yet?

As you can see the FlowStone Comport works perfectly for the standard recommended Baud Rates, once it is made into an EXE, but I see no reason why it couldn’t go faster. We will investigate this and see what we find?

Getting back to your application, you mention A/D and FFTs etc.

Maybe there is a better way to make you application? FlowStone is very good at processing real-time audio up to 192Khz 24bit etc. Also the FFT processing is optimized at a low level so it is very fast. It seems a bit counter intuitive to sample data and then try to squeeze it into a CSV comport stream. Why not just processes the real-time audio samples?
If you have analog signals it would be far more efficient to use the real-time audio features of FlowStone, plus there is even a low-level DSP coding section. If you can work with 96Khz sampling rates there are some very high quality low cost 8ch boxes from Tascam: US-1641 (actually 16 in, 4 out, 1 digital out).

Re-embedded systems we normally do one of two things:

1) Have an Embedded PC that does the data acquisition, DSP, and Display. Using a single FlowStone EXE.
Or
2) Have the Embedded PC do the DAQ, and the DSP, using a FlowStone EXE but then transmit the results/graphs via TCP over the network to a second PC anywhere in the world for display and control, again using a FlowStone app.

[vpnsoluk]

I can't fully explain what we are doing as that would break very restrictive NDAs, but suffice to say the ONLY thing i can do is 'print' the data. Printing it in CSV is very effecive as I can pass it to other software without alteration, e.g. MathCAD and Python. I can also read it as it flies up the screen. Because the data is real time and very random I can't do much to retrieve stuff that went on in the past without simply recording everything.

If I were to send it in a more compressed form (which I do occasionally, and have a very robust protocol for that purpose) then it would just need more processing to make it readable.

The data is not just from ADCs, it is more to do with the way that data is processed internally by the device in real time through filters etc, and the device is tiny, about the size of a stick of gum, so emulating it using a PC would not make sense. The USB type B connector is the largest thing on it!!

I've been processing the data in Python for about a year, but not graphed it in real time. I tried using 'standard' Python libraries for graphing and found it too slow, so saught an alternative. Since failing with FlowStone, I've done a bit more work and now have Python, running under windows doing the job perfectly, indeed better than i would have ever expected given the lousy real time performance of Windows. So my only reason for looking at your software in the first place was that I was just looking for a better way of post processing the data in real time without blowing our budget on a copy of Labview. A few hundred dollars compared to a few thousand is a good deal and DSP audio features of FlowStone seemed attractive.

Perhaps I'll keep an eye on what you are doing with FlowStone and give it a go in a year or so, for now its better to stick with what I know works.

Thanks,
Jason Morgan
Director, VPN Solutions Ltd