How to create keypress simulation program?

7

Main Problem

I'm trying to create a widely applicable keypress simulation program, but I don't know how to use Java so I can only search for Q&As on the site, which is inefficient. So I want to open a new post and deal with these problems all in once:

I need the following functionality:

  1. Press arbitrary keys (using key codes shown here) for example, some code like press[65] should press A.

  2. Press arbitrary mouse keys (including mouse down, mouse up and mouse click)

  3. Move mouse to a specific location

  4. rotate the scrolling of the mouse

  5. Screen shot

How to do these small functionalities? Thanks!


Some work I've done

One can use these as a reference:

JLink part:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];

move[{x_, y_}] := robotclass@mouseMove[x, y];
mouseleftdown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
mouseleftup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
mouseleftclick[] := CompoundExpression[mouseleftdown[], mouseleftup[]];
mouseleftdoubleclick[] := Do[mouseleftclick[], 2];

Mathematica Part:

MousePosition[];
CurrentValue[{"MouseButtonTest",1}]

Related

move cursor

left mouse clicks

key press

screen shots

Wjx

Posted 2016-08-25T14:15:16.910

Reputation: 8 808

related: search?q=robotClass and search?q=screenshot

– Kuba – 2016-08-25T14:17:03.743

What, that can't be found above, is missing? – Kuba – 2016-08-25T14:18:18.387

What's missing? – Wjx – 2016-08-25T14:21:16.910

What answers have you failed to find? – Kuba – 2016-08-25T14:25:26.330

rotate scrolling and press arbitrary mouse keys (except left mouse button). but just willing to merge them together so that further readers can get a full image of how to manipulate such things with this post. – Wjx – 2016-08-25T14:27:14.403

@Kuba up to now no Q&A include the content about scrolling or right key clicks. and unfortunately, simple alternation in the code apply for left nouse click will not work...... – Wjx – 2016-08-26T13:45:15.680

Answers

5

A summary to all the stuffs:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];
LoadJavaClass["java.awt.event.KeyEvent"];
LoadJavaClass["java.awt.Event"];

(*Action*)
mousemove[{x_, y_}] := robotclass@mouseMove[x, y];

leftmousedown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
leftmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
rightmousedown[] := robotclass@mousePress[InputEvent`BUTTON3UMASK];
rightmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON3UMASK];
midmousedown[] := robotclass@mousePress[InputEvent`BUTTON2UMASK];
midmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON2UMASK];
leftmouseclick[t_: 1] := 
  Do[CompoundExpression[leftmousedown[], leftmouseup[]], t];
rightmouseclick[] := 
  CompoundExpression[rightmousedown[], rightmouseup[]];
midmouseclick[] := CompoundExpression[midmousedown[], midmouseup[]];

scroll[x_] := robotclass@mouseWheel@x;

keypress = (Function[k, 
      robotclass[keyPress[Symbol["KeyEvent`VKU" <> k]]]] /@ {##};
    Function[k, 
      robotclass[keyRelease[Symbol["KeyEvent`VKU" <> k]]]] /@ {##}) &;

(*Sense*)
getpixelcolor[{x_, y_}] := 
  StringCases[robotclass@getPixelColor[x, y]@toString[], 
   "=" ~~ d : DigitCharacter .. :> ToExpression@d];

screeninfo = 
  "FullScreenArea" /. 
   Flatten@SystemInformation["Devices", "ScreenInformation"];
screenshot[range : {{x0_, x1_}, {y0_, y1_}} : screeninfo] := 
  GUIScreenShot[range];

The name of those functions already explains everything, so no need for extra explanations~ This piece of code has already include all the key features in 按键精灵, a keypress or mouse move simulator used to automate repetitive works, e.g. play HearthStone. But with the ability of Mathematica, I think this piece of code can do much more! :)

Wjx

Posted 2016-08-25T14:15:16.910

Reputation: 8 808

how do I get keypress to emulate "f" – Conor Cosnett – 2017-07-22T11:58:08.900