In this example, we create an emotion imitation application using our blocks from QTrobot Studio. Here is the scenario:
QTrobot looks for a person's face and recognizes one of these three emotions: happy, angry and surprise. Then QTrobot will imitate that emotion by showing the corresponding facial expression. If no person appears within some time, QTrobot looks around by randomly moving his head to left or right in search of a human face.
How does the QTrobot's emotion recognition work?
Now let’s see how we can easily implement our scenario in QTrobot Studio. There is QTrobot nuitrack interface which is running on the robot. Using QTrobot's camera, the
qt_nuitrack_app simply detects whatever somebody is standing in front of the QTrobot and reading his/hers face expressions/emotions. If we stand in front of the QTrobot, the
qt_nuitrack_app recognizes the face expression and it publishes the corresponding message to
/qt_nuitrack_app/faces topic. The topic uses a message of type
qt_nuitrack_app/Faces. Within this message, the value of faces, is an array of
FaceInfo (multiple faces):
we are interested in
emotion_surprise for our scenario. The value of these fields is the confidence level of detected emotion ranged from
1.0. Higher value represents higher confidence level in recognizing the corresponding emotion. In our example, we consider values above
0.9 as confident enough.
First we try to read a message from
/qt_nuitrack_app/faces topic using ROS Subscriber block and check if the message is not empty. After that, we extend it so that we read a detected emotion confidence level and react accordingly. Now we have our main building blocks, we implement the logic of the game as it is described in the scenario, and finally we put all the pieces together and adds some speech messages to make our games more interesting.
1. Read the detected emotion
We can use the ROS Subscriber block to read the message published by
qt_nuitrack_app/faces as it shown here:
2. Extend the blocks to extract emotion confidence level
In our scenario we are interested to know which emotion the user has shown and react accordingly.
Our ROS Subscriber block waits until a human facial emotion is detected (store it in
faces variable) or
10s passed without appearance of any person in front of the QTrobot. Then we check if any face has been detected by simple checking the validity of
faces. If valid, this variable holds an
array of detect
faces. In our scenario we are interested in the first detected
face. Thus, we need to get the first item of
faces array. We call this
my_face which is of type
FaceInfo. Then we simply check the confidence value of each interested
emotions. For example, if the confidence value of
emotion_happy is more than
0.9, we show
QTrobot Happy emotion. We do the similar thing for other emotions.
3. Make QTrobot to look around
Now we just need to implement the look-around part in case no one appears in front of the robot. We want to move only the robot
head yaw joint (left and right) to a random position. For this purpose, we use standard
random-integer block to create message with random number between
40 degree (for
HeadYaw joint). Then we simply publish it to
/qt_robot/head_position/command topic. Using the
random-integer blocks, we also show yawing face randomly every few times that QTrobot does not see any face.
Let see how we can implement it using our blocks:
5. Put it all together
Now we have all our building blocks we can put them together and finalize our scenario. We wrap all blocks by LuxAI
repeat until I press stop block. This repeat blocks keeps our game running until we stop it using Educator Tablet. Then we add a condition block (IF) to check the detection of the user. If the user is not present for
10 seconds, QTrobot will look around. Here is the complete source of our memory game: