TProcess

BlitzMax Forums/BlitzMax Programming/TProcess

PhotonTom(Posted 2014) [#1]
Hi all,
I'm developing an application that uses an open source commandline tool but I cannot get Blitzmax to print out all of the output of the program that I would get if I started the same program from the command line. Removing the "If UploadProcess.pipe.readavail()" line causes more output of the program to be printed but also crashes the application. Is there any way around this or should I found an alternative module?
Thanks

Self.UploadProcess = createprocess("commandline app")
Local s:String
		
		Repeat
			If ProcessStatus(UploadProcess)=1 Then 
				If UploadProcess.pipe.readavail()
					s=UploadProcess.pipe.ReadLine().Trim()
					print(s)
				EndIf
			Else	
				Exit
			EndIf	
			
		Forever



BlitzSupport(Posted 2014) [#2]
Not sure where I got this, but this seems to work (Windows example -- uses ipconfig.exe):




Kryzon(Posted 2014) [#3]
Is there any way around this or should I found an alternative module?
It may be that you're not also reading from the StdErr output, which is another way applications let out more technical output (errors, logs etc.). Right now you're just reading from the StdOut output.

I used the following to read a command line application. I found it around these forums; I don't remember who is the author but I'm really grateful to him:




PhotonTom(Posted 2014) [#4]
Any way for it to process each character one by one, instead of a whole chunk of string text. The application I'm using has a text progress bar and at the moment it just appears to do nothing for a min or two and then shows the completed text progress bar.
Also using the err stream as well produces the full results but because of the chucked strings its all out of order.


Kryzon(Posted 2014) [#5]
Something which I had to add to the source of the application that I was using was a fflush() call right after it printed something. This fflush() function ensures that the information is output right then.
So I sought each printing command inside the source and inserted an fflush() call on the line below them, and rebuilt the software using the build instructions that the author published.

The lack of it was giving me a serious synchronization problem, with strings being received by BlitzMax in the wrong order.

Apart from that, you can't make the application print one character at a time unless you program it to do exactly that.
In a piece of C code the programmer would usually have a statement such as the following to print text to StdOut or StdErr:
fprintf( stderr, "Error %d, cannot bind", errno );
The entire string is sent out at once. The only way to interpret it character by character is on your end, after you read it in BlitzMax.


PhotonTom(Posted 2014) [#6]
OK that is fine about the characters but I don't really understand why running this program from command line results in each line printing in the correct order whereas running it with my application results in big chunks of printed strings.
This is particularly problematic as my program doesn't receive any err strings until the end, whereas on the cmdline they are inter dispersed between the normal lines.
Local s:String
Local a:Int = 0
While Process.pipe.readavail() Or Process.err.readavail()	
	If Process.pipe.readavail() Then
		s=Process.pipe.ReadString (Process.pipe.ReadAvail())
		If EmptyString(s) Then 
		
		Else
			Console.AppendText(s+" ~n ")	
		EndIf 
	EndIf 
	If Process.err.readavail() Then 
		s=Process.err.ReadString (Process.err.ReadAvail())
		If EmptyString(s) Then 
		
		Else
			Console.AppendText(s+" ~n ")	
		EndIf	
	EndIf	
	A4SHelperApp.Yield()
Wend


Commandline output:
C:\>arduinouploader standardfirmata\standardfirmata.ino 1 COM
8
Arduino Compiler & Uploader Version 0.8.8
(C)2013 Developed by Stanley Huang <stanleyhuangyc@...;, distributed under
 GPL license

Build Target: Arduino Uno (MCU: atmega328p)
Referenced libraries: [Servo] [Wire] [Firmata]

Compiling standardfirmata.ino.cpp...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata -I"standardfirmata" "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino
.cpp" -o "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino.cpp.o"

[Compiliation: 16%]

Compiling library [Servo]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Servo/Servo.cpp" -o "C:\Users\Tom\AppData\Local\Temp/
Servo.cpp.o"

Compiling library [Wire]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/Wire.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Wi
re.cpp.o"
avr-gcc -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/utility/twi.c" -o "C:\Users\Tom\AppData\Local\Te
mp/twi.c.o"

[Compiliation: 33%]

Compiling library [Firmata]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Firmata/Firmata.cpp" -o "C:\Users\Tom\AppData\Local\T
emp/Firmata.cpp.o"
^C
C:\A4S\ArduinoUploader>arduinouploader standardfirmata\standardfirmata.ino 1 COM
8 >>temp.txt
Build Target: Arduino Uno (MCU: atmega328p)
Referenced libraries: [Servo] [Wire] [Firmata]

Compiling standardfirmata.ino.cpp...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata -I"standardfirmata" "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino
.cpp" -o "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino.cpp.o"

Compiling library [Servo]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Servo/Servo.cpp" -o "C:\Users\Tom\AppData\Local\Temp/
Servo.cpp.o"

Compiling library [Wire]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/Wire.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Wi
re.cpp.o"
avr-gcc -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/utility/twi.c" -o "C:\Users\Tom\AppData\Local\Te
mp/twi.c.o"

Compiling library [Firmata]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Firmata/Firmata.cpp" -o "C:\Users\Tom\AppData\Local\T
emp/Firmata.cpp.o"

Linking all objects...

Generating program HEX (standardfirmata\standardfirmata.ino.hex)...

Compiliation successful completed!Starting upload for Arduino Uno via COM8...

C:\A4S\ArduinoUploader>arduinouploader standardfirmata\standardfirmata.ino 1 COM
8 > temp.txt
Build Target: Arduino Uno (MCU: atmega328p)
Referenced libraries: [Servo] [Wire] [Firmata]

Compiling standardfirmata.ino.cpp...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata -I"standardfirmata" "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino
.cpp" -o "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino.cpp.o"

Compiling library [Servo]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Servo/Servo.cpp" -o "C:\Users\Tom\AppData\Local\Temp/
Servo.cpp.o"

Compiling library [Wire]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/Wire.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Wi
re.cpp.o"
avr-gcc -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/utility/twi.c" -o "C:\Users\Tom\AppData\Local\Te
mp/twi.c.o"

Compiling library [Firmata]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Firmata/Firmata.cpp" -o "C:\Users\Tom\AppData\Local\T
emp/Firmata.cpp.o"

Linking all objects...

Generating program HEX (standardfirmata\standardfirmata.ino.hex)...

Compiliation successful completed!Starting upload for Arduino Uno via COM8...

C:\A4S\ArduinoUploader>arduinouploader standardfirmata\standardfirmata.ino 1 COM
8
Arduino Compiler & Uploader Version 0.8.8
(C)2013 Developed by Stanley Huang <stanleyhuangyc@...;, distributed under
 GPL license

Build Target: Arduino Uno (MCU: atmega328p)
Referenced libraries: [Servo] [Wire] [Firmata]

Compiling standardfirmata.ino.cpp...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata -I"standardfirmata" "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino
.cpp" -o "C:\Users\Tom\AppData\Local\Temp/standardfirmata.ino.cpp.o"

[Compiliation: 16%]

Compiling library [Servo]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Servo/Servo.cpp" -o "C:\Users\Tom\AppData\Local\Temp/
Servo.cpp.o"

Compiling library [Wire]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/Wire.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Wi
re.cpp.o"
avr-gcc -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Wire/utility/twi.c" -o "C:\Users\Tom\AppData\Local\Te
mp/twi.c.o"

[Compiliation: 33%]

Compiling library [Firmata]...
avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD
 -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cor
es/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Serv
o -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/
Firmata "arduino/libraries/Firmata/Firmata.cpp" -o "C:\Users\Tom\AppData\Local\T
emp/Firmata.cpp.o"

Linking all objects...

[Compiliation: 50%]

Generating program HEX (standardfirmata\standardfirmata.ino.hex)...

[Compiliation: 66%]

Compiliation successful completed!
[Compiliation: 100%]

Starting upload for Arduino Uno via COM8...

avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude.exe: Device signature = 0x1e950f
avrdude.exe: reading input file "standardfirmata\standardfirmata.ino.hex"
avrdude.exe: writing flash (11882 bytes):

Writing | ################################################## | 100% 1.90s

avrdude.exe: 11882 bytes of flash written

avrdude.exe: safemode: Fuses OK

avrdude.exe done.  Thank you.


My codes output:
Starting Upload on COM8 
Arduino Compiler & Uploader Version 0.8.8
(C)2013 Developed by Stanley Huang <stanleyhuangyc@...;, distributed under GPL license

 
 
[Compiliation: 16%]

[Compiliation: 33%]

[Compiliation: 50%]

[Compiliation: 66%]

[Compiliation: 100%]

 
 
avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude.exe: Device signature = 0x1e950f
avrdude.exe: reading input file "StandardFirmata\StandardFirmata.ino.hex"
avrdude.exe: writing flash (11882 bytes):

Writing | ################################################# 
 Build Target: Arduino Uno (MCU: atmega328p)

Referenced libraries: [Servo] [Wire] [Firmata]



Compiling StandardFirmata.ino.cpp...

avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cores/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Servo -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/Firmata -I"StandardFirmata" "C:\Users\Tom\AppData\Local\Temp/StandardFirmata.ino.cpp" -o "C:\Users\Tom\AppData\Local\Temp/StandardFirmata.ino.cpp.o"



Compiling library [Servo]...

avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cores/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Servo -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/Firmata "arduino/libraries/Servo/Servo.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Servo.cpp.o"



Compiling library [Wire]...

avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cores/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Servo -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/Firmata "arduino/libraries/Wire/Wire.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Wire.cpp.o"

avr-gcc -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cores/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Servo -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/Firmata "arduino/libraries/Wire/utility/twi.c" -o "C:\Users\Tom\AppData\Local\Temp/twi.c.o"



Compiling library [Firmata]...

avr-g++ -c -Os -s -pipe -fno-exceptions -ffunction-sections -fdata-sections -MMD -DARDUINO=105 -DF_CPU=16000000L -mmcu=atmega328p -Iarduino/hardware/arduino/cores/arduino -Iarduino/hardware/arduino/variants/standard -Iarduino/libraries/Servo -Iarduino/libraries/Wire -Iarduino/libraries/Wire/utility -Iarduino/libraries/Firmata "arduino/libraries/Firmata/Firmata.cpp" -o "C:\Users\Tom\AppData\Local\Temp/Firmata.cpp.o"



Linking all objects...


Generating program HEX (StandardFirmata\StandardFirmata.ino.hex)... 



Compiliation successful completed!Starting upload for Arduino Uno via COM8...

 
 



Kryzon(Posted 2014) [#7]
Please read my last post again, with emphasis on the fflush() part. It seems to be the same problem.
I don't know the exact cause, it must be a difference between MS-DOS and Blitzmax.


PhotonTom(Posted 2014) [#8]
Thanks for your suggestion but that is a lot of work (also I only really work with blitzmax so compiling a non blitzmax source would be a massive learning curve for me) and doesn't help when I use a program that isn't open source. Surely as it works fine on a command line there must be a workaround for blitzmax?


Kryzon(Posted 2014) [#9]
I thought the header line said it's distributed under GPL. That utility is completely open source.

In any case, I've done some research and the problem is caused by "output buffering." For efficiency, when the output of an application is piped or redirected, the Input/Output system of the OS buffers the content and releases it only when the buffer is full. This is what causes different streams of content such as StdOut and StdErr to be flushed at different times.
MS-DOS is a console, and as such the output is not redirected, so there's no buffering involved and everything is flushed right when printed.

- There's no way to request unbuffering from an external application such as your BlitzMax programme.
- If you want to print ordered content immediately, without buffering, you need to make changes to the source of the utility that's producing content. This is either adding fflush calls after every print command, or adding a pair of calls to setbuf() at the beginning of the utility source (which is much quicker to do).
The calls would be these, added around the beginning of the utility source code:
setbuf( stdout, NULL )
setbuf( stderr, NULL )

http://beej.us/guide/bgc/output/html/multipage/setvbuf.html


PhotonTom(Posted 2014) [#10]
Thanks again for the help :)
Yeah the program I'm currently using is open source but I've had this problem in the past with closed source programs and was hoping to fix this problem with those too, oh well. At least I have a workaround for my current project now.