Looking at Camera Data

To retrieve camera data from the virtual sensor, we can use the following RACECAR
function:

image = rc.camera.get_color_image()

To display the camera data from the virtual sensor, we can use the following RACECAR
function:

rc.display.show_color_image()

This will open up a display window when the program runs. For Windows users, a very
common error may pop up: qt.qpa.xcb: could not connect to display
localhost:42.0

To solve this issue, open XLaunch and set the display number to 42.

Camera frames are structured as a 3D array, with each dimension representing the row,
column, and channel respectively. Camera images come in as the BGR color space, which
means that the first channel has a blue color, the second color has a green color, and the
third channel has a red color.

To find the shape of the image, use the command:

dimension = image.shape

Which returns the height, width, and channel depth of the numpy array.

We can extract the blue, green, and red pixel values using array indexing. Three sets of
square brackets represent the row, column, and channel of the value that we are grabbing.

# For example, let's find the pixel in the middle of the screen​
row = dimension[0] // 2​
col = dimension[1] // 2​
​

1
 # Extract the blue, green, and red values​
blue = image[row][col][0]​
green = image[row][col][1]​
red = image[row][col][2]​
​
# Display the color to the screen​
BGR_color = (blue, green, red)​
BGR_image = np.zeros((300, 300, 3), np.uint8)​
BGR_image[:] = BGR_color​
cv.namedWindow('BGR Color Display', cv.WINDOW_NORMAL)​
cv.imshow('BGR Color Display', BGR_image)​
​
# Draw a circle in the location of (row, col) on the screen)​
cv.circle(image, (col, row), 5, (0, 255, 255), -1)

Image Segmentation by Color (HSV)

https://math.hws.edu/graphicsbook/demos/c2/rgb-hsv.html

RGB is a color scheme that represents all colors on the color spectrum using a blend of red,
green, and blue values from 0-255. This is similar to mixing paint. Depending on the
amount of paint from 0-255 for each color, you would end up with a different color when all
is mixed together.
-​ For example: Red + Blue = Purple, or Red + Green = Yellow

In this lab, we want to detect an object using its specific color in the simulation. With RGB,
it might be difficult to deduce the exact color of an object (or create a threshold that
surrounds the object) since it requires a good understanding of mixing color between the
three channels.

HSV is a type of color space that decouples a few axes and makes it easier to identify raw
colors:
●​ Hue: Hue is the raw color of an object, with a numeric range of 0-180. The low
numbers represent reds, oranges, and yellows, and the higher numbers represent
blues and purples.*
●​ Saturation: Saturation represents the intensity of the color, with a numeric range of
0-255. At 0 saturation, the color is completely white, and at 255 saturation, the color
is most intense and matches the raw hue of the color.

2
 ●​ Value: Value represents the brightness of the color, with a numeric range of 0-255.
At 0 value, the color is completely dark/black, and at 255 value, the color is the most
bright and matches the raw hue of the color.

By observing the HSV color cylinder, we can separate each of the parameters of HSV into
two distinct decisions when deciding the color range an object can fall in.

Hue represents the raw color from 0 - 180. On the color cylinder, this is represented by
degrees. If we unwrap the cylinder, we can see that the raw colors fall into these
approximate values:

Saturation and Value are closely linked together and fall on the same axes. We can map
them out on a graph, with saturation (0-255) on the x-axis, and value on the y-axis. Then,
we can specify the following results from changing saturation/value:
●​ If saturation is low and value is low, the color will be black
●​ If saturation is high and value is low, the color will be black
●​ If saturation is low and value is high, the color will be white
●​ If saturation is high and value is high, the color will be the most intense color as

3
 selected by hue.

Using this logic, we can specify a “range” of colors that match with our detected object. If
the pixel color of our object is within the hue, saturation, and value thresholds we set for it,
we can say that it is (with high certainty), the object we want to detect.

To test this out, use the file hsv_tuner.py.

WINDOWS/LINUX USERS

wget
https://raw.githubusercontent.com/MITRacecarNeo/racecar-neo-oneshot-labs/refs/hea
ds/main/labs/lab2/hsv_tuner.py

MAC USERS

wget
https://raw.githubusercontent.com/MITRacecarNeo/racecar-neo-oneshot-labs/refs/hea
ds/main/labs/lab2/hsv_tuner_non_gui.py

Programming a Stoplight Detector

# Grab a frame from the color camera​

image = rc.camera.get_color_image()​
​
# Define lower and upper hsv bounds​
hsv_lower = (10, 50, 50)​
hsv_upper = (20, 255, 255)​
​
# Change color space from BGR to HSV​
image = cv.cvtColor(image, cv.COLOR_BGR2HSV)​
​
# Create a mask based on the hsv threshold​
mask = cv.inRange(image, hsv_lower, hsv_upper)​
​
# Display the frame to the screen​
rc.display.show_color_image(mask)

4
 # Crop the image​
image = rc_utils.crop(image, (180, 0), (rc.camera.get_height(),
rc.camera.get_width())

# To find contours:​
contours, _ = cv.findContours(mask, cv.RETR_LIST,
cv.CHAIN_APPROX_SIMPLE)​
cv.drawContours(image, contours, -1, (0, 255, 0), 3)

# To remove small contours (filter):​

CONTOUR_MIN = 30​
contours_filtered = []​
for contour in contours:​
​ if cv.contourArea(contour) > CONTOUR_MIN:​
​ ​ contours_filtered.append(contour)​
​
# Draw the contours to screen​
cv.drawContours(image, contours_filtered, 0, (0, 255, 0), 3)

# To find the largest contour (modify previous script)​

CONTOUR_MIN = 30​
max_contour = contours[0]​
for contour in contours:​
​ If cv.contourArea(contour) > CONTOUR_MIN:​
​ ​ if cv.contourArea(contour) > CONTOUR_MIN:​
​ ​ ​ if cv.contourArea(contour) >
cv.contourArea(max_contour):​
​ ​ ​ ​ max_contour = contour​
​
​
# Draw the contours to screen​
cv.drawContours(image, [max_contour], 0, (0, 255, 0), 3)​

5
​
# Find the center of contour and draw to the screen​
contour_center = rc_utils.get_contour_center(max_contour)​
cv.circle(image, (contour_center[1], contour_center[0]), 6, (0, 255,
255), -1)

racecar_utils function equivalent

# [FUNCTION] Update the contour_center and contour_area each frame

and display image​
def update_contour(img):​
global contour_center​
global contour_area​
​
# Crop the image to the floor directly in front of the car​
# image = rc_utils.crop(img, CROP_FLOOR[0], CROP_FLOOR[1])​
​
if image is None:​
contour_center = None​
contour_area = 0​
else:​
# Find all of the contours of the saved color​
contours = rc_utils.find_contours(image, COLOR_THRESH[0],
COLOR_THRESH[1])​
​
# Select the largest contour​
contour = rc_utils.get_largest_contour(contours,
MIN_CONTOUR_AREA)​
​
if contour is not None:​
# Calculate contour information​
contour_center = rc_utils.get_contour_center(contour)​
contour_area = rc_utils.get_contour_area(contour)​
​
# Draw contour onto the image​
rc_utils.draw_contour(image, contour)​
rc_utils.draw_circle(image, contour_center)​

6
​
else:​
contour_center = None​
contour_area = 0​
​
# Display the image to the screen​
rc.display.show_color_image(image)