Close

A USB HID Keyboard, Mouse, Touchscreen Emulator

Close

Teensy Board

The Teensy is a tiny PCB board (35 x 18 mm) with a lot of hardware functionality. It can be programmed with the same developer environment as an Arduino board. Among the features of the Teensy is the capability of the microprocessor to emulate USB devices.

Programming

To program the Teensy's processor you need a micro USB cable and you have to install:

1. The Arduino compiler version 1.64. (Download)
2. The TeensyDuino libraries (Download)
   Direct links to version 1.23: MAC, Linux 32 bit, Linux 64 bit, Windows

The source code has to be written in C and the firmware that you write for the board is named a "Sketch".

You have to configure the compiler like in this screenshot:

You can also use the option "Serial + Keyboard + Mouse + Joystick" which will add an USB virtual COM port to the keyboard, mouse and joystick devices.
 

Problems with version TeensyDuino 1.23

When you have installed and configured the compiler you can already start writing code that emulates keyboard, mouse and joystick. You will find several example projects already installed on your harddisk (see menu "File" -> "Examples").

But there is a severe problem with the original code from TeensyDuino 1.23:

The mouse emulation does not work on Linux.
The cause is a severe design error in the Linux X11 server which is full of bugs and misdesigns.

The X11 server does not accept a USB mouse device that sends absolute coordinates.
But if you want to position the mouse at an exact location on the screen, the mouse positioning with relative coordinates is useless for several reasons:

1. As you don't know the current location of the mouse pointer on the computer that is connected to the Teensy you would have to first move the mouse to the top left corner of the screen and start a relative movement from there, which is ugly.
2. When the Teensy USB device tells the remote computer that the mouse has moved by 100 units it depends on the settings in control panel and on the operating system, how much the mouse really moves on the screen. It may move 53 pixels or 144 pixels!
3. When "Mouse Enhancement" is enabled in control panel it becomes even worse: The distance of movement will depend additionally on the movement speed.
4. Relative movement works on Linux only correctly in steps of 1 due to bugs/misdesigns in X11.
5. Summary: If you want to position the mouse pointer exactly with relative movement: FORGET IT !

For that reason PJRC has implemented absolute mouse positioning in the versions TeensyDuino 1.17 and 1.18. This works better but then some users complained that they now cannot move the mouse anymore relatively which they needed for their projects.

So in the TeensyDuino versions 1.19 to 1.23 PJRC implemented a USB device that allows relative and absolute movement in the same HID device. But Linux does not accept this HID device and the mouse does not move with absolute coordinates anymore.

SUMMARY:
Whatever version of TeensyDuino you use, you will never have a code that satisfies the needs of all users and all operating systems.

My changes

And here comes my project. The code that you can download above in the ZIP file is a modification of the Teensyduino 1.23 code. Copy the 5 files in the ZIP to the folder ArduinoCompiler/hardware/teensy/avr/cores/teensy3
I made the following changes to the original code:

1. The mouse device uses ONLY relative coordinates which will work on all operating systems (just like a "normal" real mouse).
2. I added a new touch screen device for absolute pointer movement that also works on all operating systems.
3. I did not change the commands of the Mouse class that you already are using in your current sketches, so you simply replace the files in the ZIP file above and recompile your sketch without changes.
4. I completely rewrote the Mouse class, cleand up the code and added more functionality.
5. I added lots of comments to the code that was sparsely commented before.
6. I have commented out the joystick device which I don't need. If you need it, read my comments in usb_desc.h how to enable it again.
7. I removed the SerialEmu device which I don't need. If you should need it, you have to change only a very few lines in usb_desc.h.

I hope that PJRC will integrate my code into future versions of TeensyDuino (from 1.24 on, which is not yet released).

For the case that PJRC does not implement my code, you find also the original code from 1.23 in the ZIP file. So you can compare what I have changed and you can also compare the orginal 1.23 code with any future version to see what PJRC has changed. (I recommend Araxis Merge to compare text files)

After applying my changes and connecting the programmed Teensy to a Windows computer you will see in control panel 3 new USB devices:

USB device descriptors

To create a new HID device in Teensy I had first to learn a lot about USB descriptors. There is no really well written manual about this complex stuff in internet. If you read the documents from USB.org you will get shocked about the horrible writing quality. You will probably not understand anything.

You find some more user friendly tutorials at Beyondlogic and Eleccelerator but even these stay cryptic for a beginner.

A VERY great help for Windows users is the program USBlyzer which exists as a 33 day trial version with full functionality. This USB Sniffer shows all device descriptos of all connected USB devices and the data packets sent through the USB cable(s).

To be sure that what I do is correct I copied the device descriptors of a real Genius optical mouse and a real touch screen: ELO TouchSystems CarrollTouch 4500U. The keyboard device stays unchanged.

Here are the HID device descriptors that I finally implemented:

As you see the mouse now allows relative movement of 16 bit (-32767 to 32767) rather than 8 bit (-127 to 127) in the original code which was far too few.

The touch screen uses coordinates from 0 to 10000 which allows to pass values in percent of the screen width/height with a precision of two digits after the decimal point. So to position the mouse at the bottom right corner of the screen the Mouse class sends the values 100.00% for X and Y which corresponds to a value of 10000 beeing sent through the USB cable. To position the mouse in the center of the screen the Mouse class sends 5000, 5000.

Note that it is completely irrelevant how many pixels the real monitor resolution is. This works on ANY monitor size because the operating system that receives the touch screen coordinates converts them into screen coordinates in pixels. All touch screens work like this.

I tested my project on Windows XP, 7, 8, 10 and on Linux (Suse, Ubuntu, Knoppix) where it works seamlessly.

First alternate Touchscreen Descriptor

Update february 2016:
Some people reported here on Codeproject that the above touchscreen descriptor does not work on OSX and Android. So I created an alternate descriptor that you must enable in the file usb_desc.h by changing a zero into a one in this line:

#define TOUCH_DEVICE   1

This will replace the above descriptor with this one:

I tested this descriptor successfully on Windows XP, 7, 8, 10 and on Ubuntu and Knoppix.

Please report results on OSX and Android!
 

Second alternate Touchscreen Descriptor

You can enable the third touchscreen descriptor in the file usb_desc.h with:

#define TOUCH_DEVICE 2

This descriptor is a multi-touch descriptor, but it uses only one finger.

ATTENTION:
This descriptor behaves very different from the previous two descriptors.
When you call Mouse.moveTo() the mouse pointer will NOT move on the screen. When you click with the simulated finger, the mouse pointer jumps to the new location and it may even disappear (Linux and Windows). The reason is that on a real touch screen you do not need the mouse pointer because you see where you are touching, so the operating system hides the mouse pointer.

This descriptor works immediately on Windows XP, 8, 10 and Ubuntu and Knoppix.
But nothing will happen on Windows 7 if you do not enable it first. Go to Control Panel -> Pen and Touch:

After the first installation (Windows says: "The device is ready now") it may be necessary to disconnect and reconnect the Teensy once to make it work.

Try which of the three touchscreen descriptors works for you by setting TOUCH_DEVICE to 0, 1 or 2.

Programming Mouse/Touchscreen Movement

The commands in your sketches can stay unchanged, but you get extended functionality:

Move the mouse pointer with relative coordinates (-32767 to 32767) via mouse device:

Mouse.move(x, y);

Move the mouse pointer with absolute coordinates in percent (0 to 10000) via touchscreen device:

Mouse.moveTo(x, y);

Example:
Mouse.moveTo(5000, 5000);
moves the mouse to the center of the screen (50.00%)

Mouse.moveTo(10000, 10000);
moves the mouse to the bottom/right corner of the screen (100.00%)

Move the mouse pointer with absolute coordinates in pixels on a screen of 1024 x 768 pixels resolution via touchscreen device:

Mouse.screenSize(1024, 768);
Mouse.moveTo(x, y);

Example:
Mouse.moveTo(1024/2, 768/2);
moves the mouse to the center of the screen

Mouse.moveTo(1024, 768);
moves the mouse to the bottom/right corner of the screen

As you see:
After calling Mouse.screenSize() the coordinates automatically change from percent to pixel.
You have to call Mouse.screenSize() only once (or never).

Doing a drag operation with the touchscreen device

Mouse.moveTo(startX, startY);
delay(10);
Mouse.set_buttons(0, 0, 0, 1); // finger down
delay(10);
Mouse.moveTo(endX, endY);
delay(10);
Mouse.set_buttons(0, 0, 0, 0); // finger up

This will result in a jumping drag operation rather than a slow drag operation.
To get a more realistic simulation you should write a loop which sends the coordinates in smaller steps.
For example to move from coordinate 0,0 to 100,100:

Mouse.moveTo(10,10); 
delay(5);
Mouse.moveTo(20,20); 
delay(5);
Mouse.moveTo(30,30); 
delay(5);
Mouse.moveTo(40,40);
etc.. 

Special Case: Apple Macintosh

The previous code of PJRC had a workaround for Apple Macintosh which puts a fix margin of 7.5% around the usable area of the mouse movement resulting in 85% of the screen area. See this posting.

I don't know if a touchscreen on MAC also needs this mouse workaround because I don't have a MAC to test it.
For the case that it should be required, I implemented four optional parameters for Mouse.screenSize() that allow to define this margin, but in a more flexible way than the original (very cryptic) PJRC code:

Mouse.screenSize(widthPixel, heightPixel, // in pixel
                 marginLeft, marginTop, marginRight, marginBottom); // in percent

Example for MAC:
Mouse.screenSize(1024, 768, 750, 750, 9250, 9250);

This line defines a screen size of 1024 x 768 pixels and a margin of 7.50% around the usable area. (100.00% - 7.50% = 92.50%).

This information is based on PJRC.
Can a MAC user please confirm that this workaround is really required?

Clicking

Due to the fact that there is now a mouse device that has 3 mouse buttons (left, middle, right) and a touch screen that has only one "button" which is the finger of the user, I added the option to click either with the mouse device or the touch screen device:

Mouse.click(button);

Or you can also use
Mouse.press(button); 
and 
Mouse.release(button); 

where button may be MOUSE_LEFT, MOUSE_MIDDLE, MOUSE_RIGHT or TOUCH_FINGER

If you want to to hold a button down (for example for a drag and drop operation):

Mouse.set_buttons(1, 0, 0, 0); // left mouse button down
Mouse.set_buttons(0, 1, 0, 0); // middle mouse button down
Mouse.set_buttons(0, 0, 1, 0); // right mouse button down
Mouse.set_buttons(0, 0, 0, 1); // finger touching
Mouse.set_buttons(0, 0, 0, 0); // release all buttons and the finger

Mouse Scrolling

And finally you can use the mouse wheel (-127 to 127) via mouse device:

Mouse.scroll(movement);

Please read the plenty comments that I have put in usb_mouse.h.

Download

Software  from  ElmüSoft

Desktop Organizer & Event Calendar My program PTBSync (in english + deutsch) has an event calendar / scheduler with lots of features, calculates the holidays of 70 countries, can display multiple gadgets on the desktop (current weather, birthdays, moon phase, menstruation calendar, world clocks,...) pins notes on the desktop, replaces the primitive Windows trayclock and adds a detailed tooltip, synchronizes with an atomic clock, and much more.

ECU Hacker & Tuner connects to ECU's (Engine Control Units) from motorbikes, cars and trucks, shows parameters, clears DTC's, has tuning for a Delphi ECU, has an ECU emulator, can analyze the CAN bus using cheap adapters, supports many protocols, has a sniffer and manual data injection for K-Line and CAN Bus, allows you to write your own macro scripts that communicate over K-Line and CAN Bus.

Oscilloscope Waveform Logicanalyzer displays oscilloscope waveforms and has A/D converters, noise suppression, precise time measurement, logic analyzer with decoders for UART, SPI, I²C, CAN bus (also CAN FD) and chip-specific decoders. Can split half-duplex communication. Transfers signals over USB (SCPI) from the oscilloscope to the computer and remote controls the oscilloscope. It is open-source and you can extend it with new features.

Subtitle Tuner calculates the display duration of the subtitles based on the count of characters in the text. This avoids that badly created subtitles disappear before you could read them. Can remove deaf texts, lyrics, italic/bold. Can resynchronize a subtile to a movie with different framerate or change the subtitle offset.

Secure Doorlock with RFID Cards is a project with an RFID antenna and a Teensy processor that opens a door with Mifare Desfire cards. The RFID cards of up to 64 users can be stored in the flash memory. It is extremely secure because it uses strong encryption. A big battery assures that the door can be opened during a power failure.

Remote Control your Computer with Infrared An infrared receiver and a Teensy processor allow to remote control the music player on your computer with any old remote control that you have from a TV, amplifier, DVD player, SAT tuner or whatever. The Teensy simulates a keyboard and sends keystrokes to the music player.

3D Render and Editor Control in C# A .NET control that can be easily implemented into System.Windows.Forms applications to display 3D graphs. The user can move and rotate the graph with the mouse and edit single points. Many display modes are available: 3D surface, 3D shapes, 3D lines, 3D objects, also animated 3D objects.

USB HID Touchscreen A Teensy processor emulates keyboard, mouse and touchscreen at the same time to remote control a computer for automation purposes.

You can write me in english, oder auf deutsch, o en español.