Sending keystrokes to other applications
BlitzMax Forums/BlitzMax Programming/Sending keystrokes to other applications
| ||
| I want to control another program (MAME for those who know it), I searched and found this thread: http://www.blitzmax.com/Community/posts.php?topic=48863 which shows how to send info to Notepad Couple of things I'm not sure of from the example: Const WM_SETTEXT = $00C Extern "Win32" Function FindWindow( lpClassName$z, lpWindowName$z ) = "FindWindowA@8" Function FindWindowEx( hwnd1:Int, hwnd2:Int, lpsz1$z, lpsz2$z ) = "FindWindowExA@16" Function SendMessage( hwnd, msg, wparam:Byte Ptr, lparam:Byte Ptr ) = "SendMessageA@16" Function GetLastError() End Extern 'win=FindWindow( "SciCalc", "Calculator" ) win=FindWindow( "NotePad", "Untitled - Notepad" ) edit=FindWindowEx( win, Null, "Edit", Null ) SendMessage( edit, WM_SETTEXT, Null, "hi" ) So in the line win=FindWindow ("NotePad", "Untitled - Notepad" ) I'd obviously need to change this to the application I'm trying to control, so in my case the program name is "Mame" and the title bar will look something like: "MAME: Donkey Kong (US set 1) [dkong]" So if I change this to now read: win=FindWindow( "Mame", "MAME: Donkey Kong (US set 1) [dkong]" ) I don't understand the purpose of the next line however: edit=FindWindowEx( win, Null, "Edit", Null ) So I leave it as is. Run my program, and nothing happens. I've changed the SendMessage line to send a "5" to the application which should add a credit to the game, but nothing If I put a line in error = getlasterror() and then print this result, I get 1400 which if looked up on MSDN is: ERROR_INVALID_WINDOW_HANDLE 1400 (0x578) Invalid window handle Any ideas what could be going wrong? or is there an easier way to do this? What I've found on the forums so far is 8+ years old, not sure if there is any easier way to control other apps from within BM Thanks |
| ||
| The line edit = FindWindowEx(win,Null,"Edit",Null) will search through all the child windows of 'win' for a window control that has a class of 'Edit'. The 'Edit' class is the class name of the text area 'gadget' which is a child 'gadget' of the main notepad application/window. If you have Visual Studio installed then you may already have a tool called Spy++ as part of the installation. I'm not sure if comes with the express version however. If not, then you can download Spy ++ from here Its a great tool and can give you much more information that you need. Hope it helps. |
| ||
| Thanks, I downloaded Spy but can't get it working (yet). I'll keep playing around and see if I can work out what I'm doing wrong. |
| ||
| In Spy++... In the Search menu choose Find Window... a dialog window appears, or click the binocular icon in the toolbar. In the middle there an icon called 'Finder Tool', the icon looks like it has a target in the centre. You click and hold the mouse down over the target icon so the cursor turns to a target cursor. Keeping the mouse button pressed, drag the cursor over any window and the information in the Finder Window will tell you some brief information which ever window/gadget the mouse is hovering over, such as the window/gadget name and class name. Once the information is in the Finder Window, release the mouse button then press OK in the Finder window... The tree view will highlight an entry for that window. You can 'right-click' over that entry and select 'Properties' to get some detailed information about it. |
| ||
| Ah okay, I still haven't got Spy++ working (or my program) but I'll keep playing. I've made a few changes and have error = getlasterror() Print error win=FindWindow( "Mame", "MAME: Donkey Kong (US set 1) [dkong]" ) 'edit=FindWindowEx( win, Null, "Edit", Null ) Print win SendMessage( win, WM_SETTEXT, Null, "5" ) error = getlasterror() Print error Print win The resulting output is: 122 590924 122 590924 The 122 error code is a buffer undersize error, but I'm getting this before I do anything at all which is a bit weird as well as getting the error after I make the SendMessage call. The 590924 (correct me If I'm wrong) is just a unique identifier (handle) to the Window - so it's obviously finding it. When the program executes, the following line is working: SendMessage( win, WM_SETTEXT, Null, "5" ) but not in the way I expected, it's changing the title bar for the application. So I'm guessing I'm using the wrong "WM_" constant to do what I want. All the application (MAME) is doing as far as I'm aware is polling for keyboard input constantly, so I'll have to look more into the SendMessage syntax and the WM_ constants that you can supply it. Sorry if this looks like stupid comments/questions - I have zero experience with stuffing around with Windows APIs. Will keep playing. |
| ||
| Sorry if this looks like stupid comments/questions - I have zero experience with stuffing around with Windows APIs. Of course not! Thats what the forums are for - to get some help and advice. I'm only glad to help if I can. The GetLastError() is used by the whole thread that your code is running on. There's actually quite a lot of BlitzMax code that gets run well before the first line in your source gets run, so the GetLastError() value could be a left over result from code of one of the modules. Just for the record I get the same value if I print GetLastError() from the first line. I wouldn't worry about checking the result at the top of your code though, only after functions that you call that you know will leave a result that be can retrieved by GetLastError() - not all functions leave a result that way. SendMessage will only leave a result for GetLastError() if it fails due to a UIPI block ( incorrect privilege level ), and any result is also returned from the SendMessage function itself. it's changing the title bar for the application Thats the behavior I would expect to happen. Using win=FindWindow( "Mame", "MAME: Donkey Kong (US set 1) [dkong]" ) will give you a 'handle' ( the value of 'win' is the handle value ) to the top level main window and the behavior of sending a WM_SETTEXT message to the main window is to change its title bar text. Other type of class control, ie the 'Edit' control class will respond to a WM_SETTEXT message differently by inserting the text into the text field of that control. I would have thought you should send WM_KEYDOWN followed by a WM_KEYUP? There's also a SendInput function that sends simulated key-presses to the system. A possible down-side to SendInput is that thefunction sends to the top most window that has the focus. That may not be what you want. I'm not so familiar with how to send data to Mame. I would have thought that the Mame docs would tell you? so the above methods may not work if Mame isnt set up to respond to those types of window events being sent to it. I thought the windows version of Mame used DirectInput or has it been changed nowadays? |
| ||
| I used GetError before and after just see if the values were changing. In my original code it was, and it was because I wasn't getting the handle to the window, so I wasn't overly worried when it came back with two exact same values before and after my code ran, but thanks for the input - that makes a lot of sense. I'll try with the WM_KEYDOWN, WM_KEYUP as you suggested and report back. MAME is an odd one, I've not downloaded their code (not that I can really understand C++ anyway) but I do believe they switched to DirectDraw & DirectInput some time ago, though I'm not positive. This all came about from another forum where we were talking about teaching a computer to play a classic arcade game, yes it can be done, but I'd like to give it a go myself and my first thought was to give it a go with BM, without thinking the problem through I thought it would be relatively simple to send keystokes to any Windows application - but that may not be the case given what you've told me. Anyway, I'll try and have a bit more of a play/research this weekend. Thanks for all your help Col. Will definitely post back here if I get something working. |
| ||
Here is a more low level approach that uses the Win32 SendInput command.Extern "win32" Function SendInput(inputcount,inputdata:Byte Ptr,size) End Extern Type KeyInput Field inputtype Field vkeycode:Short Field scancode:Short Field flags Field time Field extra Field pad1 Field pad2 EndType Function PostKeyDown%(keycode) Global akey:KeyInput=New KeyInput akey.inputtype=1 akey.scancode=keycode akey.flags=4 Local res=SendInput( 1,akey,28 ) Return res EndFunction For i=1 To 100 PostKeyDown 100 Delay 100 Next |
| ||
| I would have thought you should send WM_KEYDOWN followed by a WM_KEYUP? There's also a SendInput function that sends simulated key-presses to the system. A possible down-side to SendInput is that the function sends to the top most window that has the focus. That may not be what you want. The WM_KEYDOWN / WM_KEYUP doesn't work, neither does the SendInput that SkidRacer posted (thanks for that btw). I'll have to have a look at the Mame Dev docs and see if there is any info there on how to accomplish what I'm trying to do - guessing it's not relying on DirectInput or key presses via SendInput |
| ||
| I always wanted to build mame as a BlitzMax module... |
| ||
| Well, MAME is completely open-source, after all, so maybe looking in there would help? (It's quite scary in there if you are not versed in C!) Arabia, what are you trying to accomplish? Russell |
| ||
| I have also been trying to send keystrokes to MAME for a long time. I was making an Arcade Frontend for arcade machines and wanted one of the buttons on the machine to act as Quick Save and another one as Quick Load. But no matter how much I tried, I could never have MAME answer to emulated keystrokes while every other emulator I tried did. I think you can compile MAME to use its old input system instead of the "new" DirectInput, and the old one apparently allows for emulated keystrokes. That solution does however not work for me as not all users using my Arcade Frontend will use the specially compiled MAME version. So I too would like a solution for this. |
| ||
| I've been reading through the docs and some source of MAME. MAME reads input via the WM_INPUT windows message. I thought if I create a WM_INPUT message and sent it into MAME that it would work. MAME does read it but it doesnt respond. It turns out that I was sending in a pointer to a RAWINPUT structure, but you need to send in a HANDLE to the RAWINPUT structure ( See Remarks on WM_INPUT linked page ). Now the fun part really begins :-) How do you create a windows handle to our structure. I tried using various Windows API memory allocation functions GlobalAlloc/VirtualAlloc plus others that are related and it still doesn't work. I then read up on how/what creates the WM_INPUT message in the system - it turns out that device drivers create this message before pushing it into the system. So what next? create a custom virtual driver to fake input? custom MAME build? |
| ||
| - double post |
| ||
| Arabia, what are you trying to accomplish? This all came about from another forum where people were discussing a program that a guy wrote to "learn" via AI how to play Super Mario Bros (NES version I believe), so talk went to whether or not a computer could play Donkey Kong. @col - what you just posted.... *whoosh* the sound of what you wrote going way over my head lol. |
| ||
| @Arabia Sorry about that. When working with the WindowsAPI you will need some good websites:- MSDN is the best place but its so massive it's sometimes difficult to find what you're looking for, stackoverflow.com and codeproject.com are great places to search for answers to a problem. I've usually found other programmers have either already found a solution to a similar problem or, if not, you'll always get an answer of some description. Anyway on-topic... On Windows... Mame will process keyboard input by handling the WM_INPUT messages that come to it. WM_KEYDOWN, WM_KEYUP, WM_KEYCHAR are not processed. This is known as 'the raw input' method of handling input because the data coming in via WM_INPUT is well... raw data from devices. It allows for multiples devices of the same type to be recognised and interpreted correctly. However, this means only devices drivers ( keyboard driver, joystick driver, mouse driver ) can generate the raw data. In my little test cases over the passed few days... I was using PostMessage to post the WM_INPUT message to Mame. The function parameters are described here. When posting/sending the WM_INPUT message the lparam parameter is a handle to a RAWINPUT structure. The RAWINPUT structure contains data that describes the raw data coming in and the device from where it came so that an app can interpret the data correctly. A handle in this case is a windows system handle. Handles are different from, and are NOT pointers. A handle is simply an integer value thats a reference value and only the 'handle system' would know what the handle value actually refers to. Its a little bit like having an integer value being used to reference a variable instead of using a variable name. As I said in the above paragraph, the device driver will create the RAWINPUT structure with its data and more importantly will create a handle to that structure and then post it into the windows system, there is no way to create a handle from our side of the system. So that only really leaves a couple of options that I can think of... Try to create a virtual keyboard device, similar to the on-screen keyboard thats available in windows, then post fake keypresses from it via the RAWINPUT structure of a WM_INPUT message. Or, as Mame is open source, you could create/build your own custom version of Mame to handle the other WM_* messages. But I feel that may open a can of worms and possibly create more problems than you can solve. Anyone else have any easier ideas? |
| ||
| It's funny that you mentioned the on screen keyboard, because I was wondering if you could control Mame via this and it turns out that you can. So is there a possibility of sending input to the on screen keyboard to accomplish what I'm trying to do? Not the perfect way, but if it works it would be fine. I don't really want to mess around with the Mame source, what I want to do is provide a program that plays a game and doesn't rely on someone having to install a "custom built" version of Mame. Besides, I don't know enough C to even had the faintest idea how to do it. |
| ||
| It's funny that you mentioned the on screen keyboard, because I was wondering if you could control Mame via this and it turns out that you can. I've just tried on Win7 and Win8 using Mame 0148 run from the command line using the -w switch for windowed mode and using the standard windows onscreen keyboard but nothing is happening in Mame. I'm making sure that Mame is the focus window but its not responding. Strangely enough it does send the WM_INPUT message :/ Am I doing anything different? |
| ||
| I've just tried on Win7 and Win8 using Mame 0148 run from the command line using the -w switch for windowed mode and using the standard windows onscreen keyboard but nothing is happening in Mame. I'm making sure that Mame is the focus window but its not responding. Strangely enough it does send the WM_INPUT message :/ Am I doing anything different? Odd. I just tried with .148 and got the same result (or lack of) as you. It does however work in .106 (which is a build used for setting records with Twin Galaxies). Obviously something was changed with the input between these versions. Only way to find out what would be to either compare the respective versions code or go through the update logs - not keen on either of these lol. |
| ||
SendInput works ok with .106 :
Strict
Extern"Win32"
Function GetLastError()
Function SendInput(Count,pInput:Byte Ptr,Size)
Function FindWindow(ClassName$z,WindowName$z) = "FindWindowA@8"
Function GetKeyboardLayout(Thread)
Function MapVirtualKeyEx(Code,MapType,Hkl) = "MapVirtualKeyExA@12"
Function GetWindowThreadProcessId(hWnd,ProcessId)
Function GetGUIThreadInfo(ThreadId,lpGUI:Byte Ptr)
EndExtern
' WindowAPI INPUT structure for the keyboard
Type INPUT_KEYBOARD
Field _Type:Int
' KEYBDINPUT
Field _VK:Short
Field _Scan:Short
Field _Flags
Field _Time
Field _Extra
Field _pad0
Field _pad1
EndType
' WindowAPI
Type GUITHREADINFO
Field _Size = SizeOf(GUITHREADINFO)
Field _Flags
Field _hwndActive
Field _hwndFocus
Field _hwndCapture
Field _hwndMenuOwner
Field _hwndMoveSize
Field _hwndCaret
' RECT
Field _Caret_Left
Field _Caret_Top
Field _Caret_Right
Field _Caret_Bottom
EndType
' MainCode
Local MAMEClass$,MAMEWindowTitle$,hWndMAME
Local ThreadId,KeyPressed
' Choose MAMEWindowTitle : Game Title
MAMEClass = "MAME"
MAMEWindowTitle = "MAME: Ms. Pac-man [mspacman]"
' Get MAME
' Can use FindWindow(MameClass,Null) if you don't know the windows title
hWndMame = FindWindow(MameClass,MameWindowTitle)
If hWndMame = 0
Notify "Cannot find MAME Window: " + MameWindowTitle + "~nhWnd: (" + hWndMAME + ")"
End
EndIf
' A key to simulate
KeyPressed = Asc("5") ' MAME - Insert Coin
' Create an instance of the INPUT structure
Local IK:INPUT_KEYBOARD = New INPUT_KEYBOARD
IK._Type = 1
IK._VK = KeyPressed
IK._Scan = MapVirtualKeyEx(KeyPressed,0,GetKeyboardLayout(0))
IK._Flags = 8
' Push MAME to the front to give focus otherwise SendInput doesn't work
ShowWindow(hWndMAME,9)
SetForegroundWindow(hWndMAME)
' Make sure MAME has focus before sending the key press
Local TI:GUITHREADINFO = New GUITHREADINFO
Local TimeOut = 500 ' Timeout after approx 1/2 sec or 500 tries
ThreadId = GetWindowThreadProcessId(hWndMAME,0)
While TimeOut > 0
GetGUIThreadInfo(ThreadId,TI)
If TI._hWndActive = hWndMAME And TI._hWndFocus = hWndMAME Exit
TimeOut :- 1
Delay 1
Wend
' Send a paint message and wait for it to return,
' then we know the MAME message is ready for input
SendMessageW(hWndMAME,WM_PAINT,0,0)
' Press the 'KeyPressed' key
SendInput(1,IK,SizeOf(IK))
|
| ||
| Well as I mentioned above, MAMEchanged a lot of stuff in one version. You can however compile MAME and use the old input method. So basically, Yes, they did change how input is handled. |
| ||
| SendInput works ok with .106 : I couldn't get it to work - but you obviously know a bit more about this stuff than I do. I'll give you code a go over the weekend, thanks very much Col, you're a champ! EDIT: Just gave you code a quick go. It works, but with weird behavior. y I don't have time at the moment, hopefully I can nut out what is going on. Where did you get your version of Mame 0.106 from? I should have mentioned that I'm using a version called WolfMame 0.106. Not that this should make any difference, the Wolf version is just a modified 0.106 which allows settings and stuff to be recorded for score verification - none of the core code should have been changed. Anyway, again, nice work getting this going for me. |
| ||
| So basically, Yes, they did change how input is handled. Yes, it certainly looks that way. I was trying with .148 as it's the latest version, and it requires input via WM_INPUT messages on windows. It turns out that Arabia is using an earlier version that doesn't rely on that method. Where did you get your version of Mame 0.106 from? I downloaded the first version I could find and it's from [a http://www.fileplanet.com/163620/download/MAME-32-Pro-Special!-0.106]http://www.fileplanet.com/163620/download/MAME-32-Pro-Special!-0.106[/a] EDIT: Just gave you code a quick go. It works, but with weird behavior. y I don't have time at the moment, hopefully I can nut out what is going on. If you get stuck, let us know... I'm still curious what the behavior is though :) |
| ||
| If you get stuck, let us know... I'm still curious what the behavior is though :) If I run your code "as is" I get nothing. If I double up the last line as in: SendInput(1,IK,SizeOf(IK)) SendInput(1,IK,SizeOf(IK)) It inserts a coin (adds a credit). If I don't use SendInput twice, I need to run the code twice to get a credit to appear. Not sure why this is, but if it works, then it's fine with me. Next step is to press the key "1" to start the game (1 Player start), so I changed the code to this:
SendInput(1,IK,SizeOf(IK))
SendInput(1,IK,SizeOf(IK))
IK._VK = Asc("1")
SendMessageW(hWndMAME,WM_PAINT,0,0)
SendInput(1,IK,SizeOf(IK))
SendInput(1,IK,SizeOf(IK))
I wasn't sure if the 2nd SendMessageW was required, but it doesn't make a difference either way, it doesn't start the game. So that is where I'm stuck at the moment. Mind you, I haven't actually had a lot of time to look at it yet, just 5 mins while on a break at work today. |
| ||
| I'll look into why that happens over the weekend. Some thoughts... Once the MAME window has focus then SendInput should be received the first time its called. There may be issues if your app takes focus again meaning that you need your code to give focus back to MAME. This is the problem with SendInput. The checks I put in there are to make sure the MAME window is the foremost one and make sure the MAME message pump is ready. The SendMessage( ...WM_PAINT... ) is there to cause your code to wait for the result of the SendMessage from MAME. This ensures that the message pump is awake and ready to process more messages. If you SendInput while the window is in a transitional state then it wont be processed ( say from transitioning from no focus to getting focus, or while resizing from being minimized ). Because Windows OS is multi-threaded this is very likely to happen. I also think maybe WM_PAINT is the wrong message here as the wm_paint is called when the app is minimized. You mentioned earlier about 'teaching the computer to play an arcade game', so this means you're going to need some pretty responsive game-play and the way things are at the moment I'm sure it will be buggy. |
| ||
| Also... For the keys that you want simulated you can also use the BlitzMax keywords KEY_*. For eg instead of 'IK._VK = Asc("1")', you can use 'IK._VK = KEY_1'. The values are the same. |
| ||
| Thanks col. Don't stress over it too much, I just wanted to see if it could be done with BlitzMax and turns out that it probably can, but it is going to be a hell of a lot more complicated than I anticipated. |
| ||
| I'm curious how would the computer 'see' what is happening on screen so it can react? :-) Oh and just realised the reason the "1" doesnt work for you is because you to need to alter two variables in the IK type. You need to change the IK._VK and IK._Scan variables. If you're going to use the code in a project then you're better to use a function similar to what skidracer demonstrates above. Something like: Strict Extern"Win32" Function GetLastError() Function SendInput(Count,pInput:Byte Ptr,Size) Function FindWindow(ClassName$z,WindowName$z) = "FindWindowA@8" Function GetKeyboardLayout(Thread) Function MapVirtualKeyEx(Code,MapType,Hkl) = "MapVirtualKeyExA@12" Function GetWindowThreadProcessId(hWnd,ProcessId) Function GetGUIThreadInfo(ThreadId,lpGUI:Byte Ptr) EndExtern ' WindowAPI INPUT structure for the keyboard Type INPUT_KEYBOARD Field _Type:Int ' KEYBDINPUT Field _VK:Short Field _Scan:Short Field _Flags Field _Time Field _Extra Field _pad0 Field _pad1 EndType ' WindowAPI Type GUITHREADINFO Field _Size = SizeOf(GUITHREADINFO) Field _Flags Field _hwndActive Field _hwndFocus Field _hwndCapture Field _hwndMenuOwner Field _hwndMoveSize Field _hwndCaret ' RECT Field _Caret_Left Field _Caret_Top Field _Caret_Right Field _Caret_Bottom EndType Function SendKeyToMAME(KeyPressed) Local MAMEClass$,MAMEWindowTitle$,hWndMAME Local ThreadId ' Choose MAMEWindowTitle : Game Title MAMEClass = "MAME" MAMEWindowTitle = "MAME: Ms. Pac-man [mspacman]" ' Get MAME ' Can use FindWindow(MameClass,Null) if you don't know the windows title hWndMame = FindWindow(MameClass,MameWindowTitle) If hWndMame = 0 DebugLog " Cannot find MAME Window: " + MameWindowTitle + " (hWnd: " + hWndMAME + ")" Return EndIf ' Create an instance of the INPUT structure Local IK:INPUT_KEYBOARD = New INPUT_KEYBOARD IK._Type = 1 IK._VK = KeyPressed IK._Scan = MapVirtualKeyEx(KeyPressed,0,GetKeyboardLayout(0)) IK._Flags = 8 ' Push MAME to the front to give focus otherwise SendInput doesn't work ShowWindow(hWndMAME,9) SetForegroundWindow(hWndMAME) ' Make sure MAME has focus before sending the key press Local TI:GUITHREADINFO = New GUITHREADINFO Local TimeOut = 500 ' Timeout after approx 1/2 sec or 500 tries ThreadId = GetWindowThreadProcessId(hWndMAME,0) While TimeOut > 0 GetGUIThreadInfo(ThreadId,TI) If TI._hWndActive = hWndMAME And TI._hWndFocus = hWndMAME Exit TimeOut :- 1 Delay 1 Wend ' Send a paint message and wait for it to return, ' then we know the MAME message is ready for input SendMessageW(hWndMAME,WM_PAINT,0,0) ' Press the 'KeyPressed' key Return SendInput(1,IK,SizeOf(IK)) EndFunction ' Insert coin SendKeyToMAME(KEY_5) Delay 1000 ' Start one player SendKeyToMAME(KEY_1) |
| ||
| I'm curious how would the computer 'see' what is happening on screen so it can react? :-) Haven't got that far yet, my idea would be to grab screen (window) grabs and then analyze them for objects, but that is a fair way off yet. The first thing is to just get Mario (Jumpman as he was known then) moving around, then work on him completing the 1st elevator stage which is the easiest board to complete in the game as you don't have to worry about enemy's. Having said that, it would probably be a better idea to start with Pac-Man since this game can be beaten by running patterns, so no worrying about scanning the screen for ghosts. Thanks for the code above, works like a charm :) |
   |