Design and Implement an Android Application to Remotely Control a Mindstorm NXT Robot

Nowadays, robotics technology has become very popular in all fields of human life. Many robots have been already used in different applications such as indoor service, military undertaking, entertainment, healthcare service, etc. On the other hand, controlling robot remotely plays an important role in many areas when people cannot reach easily. Therefore, the aim of this work is to allow users to perform remote control experiments on a mobile robot through Bluetooth, Internet and SMS. The commands could be entered manually, using voice command, or using QR code. To limit the cost of the proposed system, the inexpensive (compared to more professional robot platforms) LEGO Mindstorm robot is used.


Introduction
Nowadays, robotics technology has become very popular in all fields of human life. Different robots have been already used in different kind of applications such as indoor service, military undertaking, entertainment, healthcare service, etc. This work designed, implemented, and evaluated a robot remote control system using a mobile phone. Specifically, an Android application was developed to control an NXT Mindstorm robot based on different techniques. To control the robot, five moves were achieved: FOREWARD, BACKWARD, TURN-RIGHT, TURN-LEFT, and STOP. The user has five options to control the robot: Manually, SMS, QR Codes, Voice Commands, and Internet. The diagram of the whole system is shown in Figure 1. Controlling robot remotely plays a very important role in different applications areas, spatially when people cannot reach easily. Therefore, the aim of this work is to allow users to perform remote control experiments on a mobile robot through Bluetooth, Internet and SMS. The commands could be entered manually, using voice command, or using QR code.

Related Works
In the past few years several researchers have explored the idea of controlling robots remotely. However, most of the related works do not focus on the methods of entering commands/orders to control the robot. In contrast, this work introduced several methods to enter commands. Each of these methods has its own advantage. On the other hand, unlike this work which it has an option to use Internet as a communication media to control the robot, most of the related works use only the wireless or Bluetooth to control the robot. For instance, authors of [5], [1] and [2] introduced systems to control robots using Bluetooth or Wi-Fi. The main advantages of using such systems are speed and reliability. However, the main drawback is the range limitation problem.. The author of [7] introduced a solution to control a robot remotely based on the traditional TCP socket. Comparing with this work, a different solution which it takes the advantage of new technologies (cloud and Firebase) was used to improve performance and reliability.

The Lego Mindstorm
The LEGO Mindstorm NXT 2.0 is a programmable robotics kit, which was launched in 2009. It contains 619 pieces that could be used to develop small programmable robots. The LEGO Mindstorm has a programmable brick computer which is the central processing unit of the mobile robot, and it is responsible for all I/O communication (wired or Bluetooth) [9] [3]. LEGO Mindstorm kits could be used to model different types of embedded. Moreover, different programming languages could be used to program the LEGO Mindstorm such as C, C++, C#, Python, and Java. The LEGO Mindstorm NXT 2.0 is shown in Figure 2. The Firebase Realtime Database, Figure 3, is a cloud-hosted database. Data is stored as JSON and synchronized in realtime to every connected client. Every time data changes, any connected device receives that update within milliseconds. [8]. In this work, the Firebase free plan, which is limited to 100 connections and 1 GB of storage [6], was used to achieve a realtime notification feature.

QR Codes
Quick Response (QR) codes are two-dimensional barcodes that can contain any alphanumeric text [10]. Data can be translated into a QR code by any QR generator, many of which are available free online. Decoding the information can be done with any mobile camera phone that has a QR reader, which is freely available online for most devices. A sample of a QR code is shown in Figure 3.

Fig. 3: QR Code Sample
QR code could be used as landmarks in indoor mobile robots to help robots become aware of their locations.

The Proposed System
The introduced Android application consists of five parts: 1. Control the robot manually using Bluetooth 2. Control the robot using voice commands 3. Control the robot using QR codes 4. Control the robot using SMS 5. Control the robot over the Internet The overall system design is shown in Figure 4.

Fig. 4: The System Schema
According to theproposed system, five commands (movements) were implemented to control the robot. These commands are explained in Table 1. The main parts that construct this work are demonstrated in the following sections.

Control the Robot Manually
By using this option, users can control the robot manually. Forward, Backward, TurnRight, TurnLeft, and Stop commands are sent directly to the robot. The Bluetooth was used to establish communication between the user (mobile device) and the robot.

Control the Robot Using Voice Commands
The user can also use voice commands to control the robot. The speech Recognizer control was used to build a simple speech recognition engine that is able to recognize the five moves that are listed in Table 1.

Control the Robot Using QR Codes
Getting a robot to know where it is and navigate to new locations is a tough problem that needs sophisticated machine vision. Robot navigation can be made much simpler and easier using QR codes as landmarks. Therefore, this part of the work takes the advantage of the QR codes to show how they could be used to navigate a robot. The QR codes for the five moves that are mentioned in Table 1 must be generated first. On the other hand, a QR code scanner was used to read the QR code value. Based on the value, the robot will move.

Control the Robot Using SMS
SMS is still one of the most reliable channels to communicate among users. Therefore, in this work, SMS technique is also used as a method to control the robot from anywhere in the world.
An SMS listener was developed and registered, so the application will receive a notification every time an SMS is received. The application checks the message content. If the value equals one of the Table 1 commands, the message will be considered as an order, and the robot will be moved based on the SMS text.

Control the Robot over Internet
Using Internet connection is another approach that was introduced and implemented in this work. Two Android applications were developed to archive this feature. Moreover, Firebase database was built as a shared database between the two applications. This process is explained in Figure 5.

Fig. 5: Over Internet Schema
App(B) will get a notification as soon as App(A) update the database. Therefore, when App(A) write a command to the database, App(B) will receive the command immediately.

Implementation And Results
App Inventor was used to develop the proposed android application. The main window of the android application is shown in Figure 6. Using buttons, the user can choose a way to control the robot.

Fig. 6: Application Main Activity
Once the application is launched, the first action that must be done is connecting to the robot. The user needs to press the 'Connect' button to select the Bluetooth device.
'Manually' button allows the user to control the robot using a joypad. Once the user press it, the directions options will appear so the user can choose a direction to move the robot, as Figure 7 shows. Five buttons are presented, and each one has an image on it to indicate its purpose.

Fig. 7: Manually Option
'Voice Command' button may use to command the robot using voice. Once the user select "Voice Command" button, the speaking window will appear (Figure 8), ask the user to talk. Depends on the user speech, the robot will be moved.

Fig. 8: Voice Commands Option
'QR codes' button can use to send commands using QR Codes To implement this part, first of all, QR codes need to be generated. A free online web service [4] was used to generate the main five commands. The generated commands are shown in Figure 9. These commands should be printed out so the user may use them later to control the robot.

Fig. 9: The Generated QR Codes
In addition, a barcode scanner needs to be installed on the Android phone. Once the user select 'QR codes', the barcode scanner app will be launched to get the value of the QR code ( Figure 10). Depends on the QR code value, the robot will be moved.

Fig. 10: QR Codes Option
'SMS' button is responsible of generating and registering an SMS listener. Once the user selects this button, the application listens and takes an immediate response to any incoming SMS. Depends on the SMS content, the robot will be moved.
Finally, to command the robot over the Internet, a checkbox was added to the main App. Window. To achieve this feature, a database needs to be created on the Firebase via Firebase website. Figure 11 shows the created Firebase database.

Fig. 11: Firebase Database
On the main application, the user may control the robot over Internet by check the "Over Internet" check box (Figure 12). When the user selects a move, the application will check if the check box is checked; if yes, the Firebase database value will be changed based on the used selection. On the other hand, once the Firebase database value is changed by main application, the second application (listener) will get a notification. Based on the notification value, the robot will be moved.

Using the Ultrasonic Sensor to Detect Obstacles
One feature that was added to the introduced system is detecting obstacle. To achieve this feature, ultrasonic sensor was used. When the robot's ultrasonic sensor detects an obstacle at a distance below 30 centimeters, the robot will stop moving automatically.

Results
The introduced application was tested, and the experimental results have shown that the proposed system is fully operational and robust. With all options, the robot was able to move and respond to the predefined commands (Forward, Backward, Right, Left, and Stop). The result have shown the effectivity of the proposed system, where it is easy and comfortable to use. Table 2 summarizes the performance of the five different methods that are used in this work to control a robot.

Conclusion
Mobile applications are ubiquitous. There are apps for just about everything, entertainments, socializing, dining, travel, philanthropy, shopping, education, navigation, and so on. In this work, a remote control mobile system was introduced based on the Android system. Using the introduced application, the robot can be controlled in five different ways (Manually, using SMS, using QR Code, using Voice Commands, and using Internet). To limit the cost of the proposed system, the inexpensive (compared to more professional robot platforms) LEGO Mindstorm robot is used. The results revealed the use of the application inventor and Firebase cloud database to implement a robot remote control as efficient with a low computational complexity.