Automative Mars Rover
A remote-controlled and autonomous Mars rover was assembled and programmed. The rover was designed to make autonomous decisions such as avoiding obstacles and moving to a target position. This was achieved by processing camera images on a DE10-Lite FPGA to detect obstacles and sending the information to a ESP32. This microprocessor then sends movement decisions to an Arduino to move the rover and report position information. A web interface allows for monitoring the rover’s position, battery usage and includes a video feed. This also allows for position remote control and adjusting object detection parameters in real time. The project can be break down to 5 subsystems, vision, control, command, drive, and power.
Introduction
Queue Overview
Queues are created using FreeRTOS functions to share data between tasks. They can either be FIFO queue with items placed and removed; or a mailbox with space for a single item that may not be removed when read. Video frame data is written to a buffer by the I2S driver which the TCP task accesses. A mutex controls which task is allowed to access this buffer.
Data flow diagram
Vision Subsystem
The Vision subsystem uses the video input from the D8M camera (OV8865). The DE-10 Lite FPGA aims to decrease the large data flow from the camera by finding just the coordinates of bounding boxes around ping pong balls to be sent to the ESP32 microcontroller. There are five ping pong balls that need to be identified, each having a distinct colour (red, pink, yellow, green, blue). The project code was built off Dr Edward Stott’s GitHub code.
Basic red detection image
Multiple bounding box detection
Dynamic Colour Detection
Each obstacle colour has its own HSV range, and it can be selected by user by the double slider. The dropdown menu allows user to choose the colour, sliders value would then change accordingly. The reset button would reset all HSV values of selected colour to the default value. By changing the configuration in real time, it helps improve accuracy under different light condition
Control Subsystem
Video Streaming
The aim is to enable video streaming by transmitting picture frames from the FPGA to the ESP32 before being transmitted by TCP/IP for displaying. The initial aim was to transmit RGB colour images with 4-bit colour depth (12-bit pixels) at VGA resolution (640 by 480 pixels), matching the VGA display output. Ideally minimal processing should be required by the Nios II processor, ESP32 and the Server to display the images.
To integrate the Video Streaming functionality into the current Video Pipeline, a second Avalon-ST source was added to the Image Processing IP block. The Video is sent to both outputs, but will only accept backpressure from one, selected through a physical switch on the FPGA.
As the bitmap takes up a majority of the TCP transmission bandwidth, sharing the same TCP port for communication of data and video limits the transmission speed. A separate TCP server port is created, dedicated for video streaming. However, during testing, it was found out that hosting the TCP server port in the Node.js main server alongside with other ports slows down the main server. Hence a separate Python TCP server is created.
Programming Structure
Using the FreeRTOS Kernel, different tasks can be created and assigned to run by the Kernel. Blocking events such as delays are used to transfer control back to the Kernel which can then assign a different task to unblock and run. The corresponding tasks and their priority are shown below
Automation
An aim of the rover is to be semi-autonomous, with the rover able to perform simple obstacle detection and position navigation. This makes use of available data from both the Drive and Vision subsystem, and complements the rover’s ability to allow the user make high level decisions through the video streaming and remote control.
To avoid obstacles, the ESP32 would need to obtain the position of any obstacles in the camera’s view. The ESP32 gets coordinates of bounding boxes for each of the ping pong ball colours via UART. Since the size of the ball on the screen is inversely proportional to its distance, and each ball is of the same width, it was determined that the distance of the ping ball could be calculated as 100𝑚𝑚∗256𝑝𝑥/𝑤𝑖𝑑𝑡ℎ, since it was determined that a 256px length ball on the screen was 100mm away from the rover. The angle of the ball is also calculated based on its x-position. It was also determined that the rover needs to have a ball move just out of the camera view for the rover to avoid hitting it. Hence the rover will determine whether to turn left or right after considering all obstacles that are within 600mm. These commands are directly sent to the same queue that the Drive UART communication reads from.
Obstacle Position Calculation
Positions of obstacles can be calculated given the direction and angle of obstacle from the rover and the rover’s current direction. The rover’s facing angle also needs to be known to calculate the x,y coordinates of the obstacles to be displayed on a 2D map.
The facing angle of the rover can be calculated by obtaining the slope of two consecutives position given that the rover is moving forward. Hence calculation of rover is only done when the rover is moving forward.
Moreover, due to measurement error, the calculated position of a same obstacle might vary a bit. Hence certain error tolerance is allowed, which is a 100mm square. If a previous-calculated obstacle of same colour is within error range of the new obstacle, the previous obstacle positions would be updated with the average of these two values, else a new obstacle would be created.
Command Subsystem
The command subsystem aims as a remote controller and dashboard monitor of the rover. It sends manual command to the rover and receives real-time data from other subsystems on board to display on the webpage. It communicates with the control subsystem via the Internet, TCP/IP is selected as the communication protocol.
Web Page Components
The 2D map is made from a scatter chart, containing data points of the obstacle positions, rover past and current position. As legend and x,y axis are disabled, the origin (0,0) is shown as a black spot on the map.
Bisque circles represent positions of obstacles, red circles represent positions of the rover, and the red square is the current position of rover. Rover position data points are connected with a grey line, sorted by time, to represent the trail of rover. Each data point on the map also has its tooltip showing the x,y coordinates, rover positions’ points also display the time data point was recorded.
Three control modes are available for the rover, Direction, Position and Exploration. Direction gives the rover exact instruction on moving direction, Position gives the rover a target coordinate and the rover would move to it, and finally Exploration allows the rover to move randomly around. Please refer to the Control Subsystem section for detail on board implementation for each mode.
In the position mode, user inputs the target x and y coordinate on the form. A blue dot would be shown on the map as a preview, only upon clicking the submit button the coordinate would be sent to the server. The stop button, as suggested by the name, would stop the rover. Clicking either button triggers an alert popped out to indicate data successfully sent out.
Drive Subsystem
The purpose of the drive module is to control and measure the distances of the mars rover, and control the direction it facing to. The control module will implement the instruction received at Tx Rx ports, and send the position back. The block diagram shows below is the overall structure of the control module.
The overall structure of the distance control can be seen in the graph below. There are 7 instructions can be read from serial communication port. For example, when the decoder received a stop instruction, the voltage in the motor will be set to zero, so the mars rover will stop.
One important feature of mars rover is distance control. When decoder received instruction such as “Moving forward 30 cm”, the desired distance will be recorded. By comparing the current location with the desired one, the error_y (difference between desired y and current y) will be recorded. When error_y is large, the voltage of the motor should be high, so it will approaching the desired location faster. When error_y is small, the voltage of the motor should be low, so the mars rover can have a better control of the speed, and not missing the target y.
