The Falcon

Handbook

Version 1.0
Last Updated 2/04/2021
History/Change log
● 4/24/2020, Version 1.0
○ Expanded section and added examples under Positioning - Vehicles
● 4/17/2020, Version 1.0
○ Added more vehicle sizing examples under Shape & Sizing - Vehicles
○ Added more examples under Echo Points
○ Updated gif examples under Unknown Zones
● 4/5/2020, Version 1.0
○ Added more vehicle sizing examples under Shape & Sizing - Vehicles
● 2/5/2020, Version 1.0
○ Draft created
● 2/4/2021, Version 1.0
○ Updated image examples under Positioning - Vehicles
○ Added a Freeway Scene Section
○ Added Parking Lots Section
○ Added more GIF examples for Echo Points
○ Added section for How to Identify Bad Data
○ Updated images under Articulated Vehicles
○ Added section about Trailers
○ Added section about Dumpsters
Table of Contents

How to Annotate:
● Ground Points and Tires
● Cuboid Sizing:
○ Shape and Sizing - Vehicles
○ Positioning - Vehicles
○ Extremities
● Unknown Zones
● Freeway Scene
● Parking Lots
● Echo points
● How to label Shifting Data
● How to Identify Bad Data
● Scattered Data
● Articulated Vehicles
● Dumpsters
● Trailers
● Animals
● Advanced Concepts
Common Errors:
● Standard Sizing - How and when to do it
Guides, Tips, and Advice:
● Workflow
● Missing Cuboids - Using Radar Points
● Pedestrians sitting on the ground
● Leaving out extremities
● Objects entering/exiting a building with an Unknown Zone
● Bicyclist/Motorcyclist
How to Annotate
Ground Points and Tires

Include ground Grounds points are the green colored lidar points near the bottom of the
points if they are scene. The coloring is meant to help differentiate the ground from the
also part of the vehicles; however, they are not always accurate. Parts of the car, like
tire. Always size tires, might be colored green by mistake.
the cuboid to the
bottom of the You should always size the cuboid to the bottom of the lowest tire. Include
lowest tire. ground points if you are trying to include a tire in the process.

Incorrect: Correct:
Example 1: Cuboid Cutting off Tires
The following incorrect example shows ground points excluded from
cuboids. This is incorrect because it also excludes the tires of vehicles.

The uneven wheel placement suggests the vehicle is on a slope. In these

cases, we should be extra careful to size the wheels properly.

Note: Beware of vehicles that are sized to the bottom of the vehicle body
with no consideration for wheel height.

Example 2: Properly including tires

The following correct annotation includes the lowest wheel point and
ground points of the elevated wheel.

The vehicle is clearly on a slope, but we are taking extra care to include
the lidar points of the lowest wheel.
Vehicle vs Small 3 wheel motor objects are considered “Vehicles”. Key difference is
Motorcycle that the rider is not exposed.

Exceptions to the “exposed rider” rule are roofless vehicles and lawn
mowers.

Shape & Sizing - Vehicles

The cuboid edge Cuboids should be tightly sized on Vehicles:
should be no 1. Height: Tire to Roof
larger than 2. Width: Door to Door
0.15m or smaller 3. Length:
than 0.05m from 4. to Bumper
the hard line of
the lidar points.

DO NOT include Do not include the following extremities within vehicle cuboids
mirrors, (examples below):
antennas, open 1. Side Mirrors
car doors / 2. Antennas
trunks, roof 3. Open Car Doors / Trunks
racks, or any 4. Roof Racks
other extremities 5. Bike Racks
6. Back tires on cars
7. Other Extremities

Example 1: Tire to Roof

(a) Tire to Roof sizing focuses on tightly sizing the top of the roof to the
bottom of the tires. The set of images on the right show the corrected size
adjustments.

Remember to always check the camera view to confirm if an extremity

exists. If an extremity exists, please exclude it.*

(b) Here we have a very big gap from the edge of the LiDAR to the edge
of the cuboid. Camera view confirms that this cuboid could’ve been sized
tighter to the LiDAR.

Example 2: Door to Door

(a) Door to Door sizing should always be sized tight to the edge of the
“hard line” of the doors of the vehicle. In this case, the cuboid was
adjusted on the sides to align with the doors of the vehicle. Exclude any
noise points when necessary. The image to the right shows the correct
size.

There should be no space in between the vehicle edge and cuboid edge!!

Remember to always have APC turned on at 50 density. Use standard

sizing if LiDAR and camera view are insufficient.

(b)

In this example, the changes are very subtle. However, this change was
large enough that it was considered a critical error.

The cuboid width on the right was narrowed on both sides to exclude the
extra noise points and extra space right by the doors of the vehicle.

Example 3: Bumper to Bumper

Bumper to Bumper sizing shows us that cuboids need to be sized tightly
from the front of the bumper to the back of the back bumper.

Here we have APC turned on at 50 density. The side view shows this
vehicle was not sized properly according to the rules mentioned above.
The back bumper still has some room left that should not be there. In the
image on the right, the back of the cuboid was pulled in where the edge
touches the LiDAR.

Bonus points if you were able to identify extra space at the top of the
vehicle.

Example 4: A Combo of Any of the 3 Rules Above

(a) This example incorporates both the Door to Door and Tire to Roof
sizing. There is too much space in between both of those edges. On the
image to the right, the cuboid was adjusted to the edge of the LiDAR.

Example 5: Front Tire at an Angle

 At first glance in LiDAR view, the protrusion at the front of the vehicle
appears to be noise. However, upon further investigation, that protrusion
is the front tire at a slant or angle.

We do not include wheels that stick out in that manner. Please follow the
Door to Door sizing rule in this case.

Positioning - Vehicles

Positioning a vehicle does not necessarily require you to adjust the size of
the cuboid. Instead, use camera view and lidar context can help determine
how to properly position the cuboid to align with the lidar edge of the
object.

Stationary Vehicles

Consider this stationary vehicle below:

At first glance, this cuboid looks like it was annotated properly. However, if
you look closely, you can see that the cuboid edge at the top is not aligned
with the edge of the LiDAR and is instead annotated at an angle.

To fix this, use the arrow keys to adjust rotational heading. There is no
pitch and roll in falcon, so you do not need to worry about that. Pressing
the right arrow key will align the cuboid edge so that the sides will become
parallel with each other.

You will notice that there is now a considerable gap in between the cuboid
and LiDAR edge. Using the WASD key functions can help move the
cuboid to its proper location. Here, the D key was used to move the
cuboid. Keep in mind that no sizing adjustments have been made, only
positioning!

Dynamic Vehicles

Dynamic vehicles can be a bit more difficult to position because their

position can vary from each frame. The best way to make sure all LiDAR
points are captured correctly within the cuboid is to go through each frame
the vehicle appears in the scene.

The hot key B is very useful to display all dynamic frames in one image.
This is called dynamic APC. For another interactive explanation of
dynamic APC, please refer to the Workflow section in the Handbook.

In conjunction, using the hotkey O to display the entire interpolated path of

the cuboid can help make sure pathways are aligned correctly.

Take this dynamic vehicle for example:

In the cuboid above, it appears that all the LiDAR points were not entirely
captured. The front bumper is cut off and the back bumper has too much
space in between the cuboid edge.
The main principle of dynamic APC is to be able to visualize all moving
frames in one place. This can be done by placing the mouse cursor and
hovering across the pathway bar.

In the scene above, it seems that the frames where the LiDAR was not
contained is within frames 30 to 45.

The next steps in fixing this is to adjust the cuboid in each key frame the
error is present in. In this case, you would adjust the cuboid in all the
keyframes between frames 30 and 45. This fix here is simple; using a
combo of the WASD keys, you can adjust the positioning for that key
frame.

Since the bumpers were not properly annotated in the example above, the
W and S keys should be used. You may have difficulty containing all the
LiDAR points that are in frames between the key frames, so some back
and forth is expected.

The end result should look like this with all LiDAR contained within the
cuboid.
Extremities

The cuboid Cuboid size should NOT be influenced by extremities. Extremities

should be no such as:
larger than 0.15m 1. Side Mirrors
or smaller than 2. Antennas
0.05m from the 3. Open Car Doors / Trunks
hard line of the 4. Roof Racks
lidar points. 5. Bike Racks
6. Back tires on cars
DO NOT include 7. Other Extremities
mirrors,
antennas, open When you are less than 90% confident that lidar points represent an
car doors / extremity, include the lidar points. Use this chart to help:
trunks, roof
racks, or any
Has Camera No Camera
other extremities
Clear Lidar Confident the extremity Confident the extremity
exists. Do not include lidar exists. Do not include lidar
points. points.

Unclear If you can visually confirm No confidence that lidar

Lidar the extremity exists, Do not points represent extremity.
include lidar points. Include lidar points.

If Camera Context cannot

confirm existence and
unclear lidar points, include
lidar points since extremity
cannot be confirmed.

Some generic extremities:

Example 1: Clear Camera and Clear Lidar
 Considerations: Instructions consider loading dock of
semi-trucks/moving vans as extremities. Lidar suggests the existence of
loading docks. Camera confirms it’s existence.

We choose to exclude the lidar points that are associated with the loading
dock.

Conclusion: Cuboid will exclude lidar points that are suspected to be part
of the loading dock. If camera view is not present, points will be excluded
if clear with 90% confidence.

Unknown Zones

Unknown zones Unknown Zones are generally large cuboids that cover an area of interest.
are used to label They cover areas of low confidence like the inside of buildings where we
lidar points with are less certain about the existence of another label (like a vehicle) or
less than 90% where the LiDAR points might be less distinguishable from the
certainty of the surroundings.
proper label.
Conditions in where Unknown Zones are used:
● LiDAR points suggest an annotatable object but can NOT be
They confirmed in camera. Additionally, there are insufficient LiDAR
automatically points to confirm conditions for annotation with 90% confidence.
include ○ Cuboid should be sized to include all lidar points in
reflections, question
objects in ● Indoor areas where vehicles and pedestrians can exist should be
buildings, and labelled with an Unknown zone. These include, but are not limited
covered cars. to, the first floor of a multi-story parking lot, garages, and
 restaurants.
○ In these cases, do NOT annotate other labels inside the
unknown zone.
○ Avoid including the walls of the building or extending past
the first floor unless there is lidar evidence of potential
things to annotate. Include those lidar points in the
unknown zone, do not annotate them individually.
○ If an object is entering or exiting an unknown zone, it
should be annotated once it is outside the unknown zone.
There should be no frame where the object is not contained
in some cuboid.
○ Example 1, Example 2
● Reflections anywhere in the lidar scene
○ Unknown Zone cuboids can extend past inner walls to
include reflections. The Unknown Zone cuboid should not
extend past the exterior wall that faces the outside.
○ An exception to the rule is when reflections extend past the
exterior walls out into the streets and onto other cuboids.
Tightly size the Unknown Zone cuboid to include all
reflection points that extend into the street.
○ Example 3

Example 1: No Cuboids Inside an Unknown Zone:

Incorrect: Correct:

Cuboids should not exist inside unknown zones. Even if they appear in
Lidar.

Example 2: Use Unknown Zones for First Floor Parking Garages /

Buildings

We can confirm in camera that the structure is a parking garage.

We label parking garages and restaurants where pedestrians and vehicles
might be with unknown zones.
We do not need to include the side walls of the building or parts of the
building past the first floor.

Example 3: Use an Unknown Zone for Reflections

(a) An unknown zone is placed over a large area that captures all
 reflections in a given region. Remember that all unknown zones that
capture reflection points should be sized within the walls of a building the
reflections are contained in.

(b) Reflections can be present outside of buildings and into streets and
sidewalks. In these cases, annotate only the reflection points of the object
that is being reflected off of.

Size the unknown zone as tightly as possible making sure that any LiDAR
points are not excluded. These specific cuboids are allowed to overlap
over other objects as long as there are reflection points.

Cuboids Resting on a Hill or Incline

When Cuboids In certain scenarios, there may be insufficient LiDAR points to judge a
have insufficient large number of cuboids. In these cases, we should place the cuboids
lidar points and within an Unknown Zone.
are on a hill or
incline, label
cuboids to the Example 1: Cuboids resting on an incline.
best of your Because of insufficient lidar points, z-index between different cuboids can
ability and place be different. This is natural given the low confidence of the entire area to
an unknown zone begin with.
over the entire
area.
In cases like this:
1. Label all cuboids to the best of your ability using LiDAR points &
Standard Sizing Rules.
2. Avoid changing existing cuboid unless you have over 90%
confidence in the change. Different Z-indexes are OK.
Freeway Scene

❏ For sparse or ambiguous lidar points far from the SDC position
that are NOT ON the freeway check camera. If the lidar points are
not in the camera image (occluded behind buildings, barriers, etc.)
we can ignore.
 ❏ MAKE SURE to check context and scene layout if you think it is an
object of interest then mark with Unknown Zone.

Parking Lots

❏ Use Context and scene composition as much as possible when

annotating Parking lots.
❏ This means that you should pay close attention to where
and how the lidar points are appearing. Some useful
general rules to follow for Lidar points in parking lots:
❏ 1. Lidar for parked vehicles will appear in orderly
rows
❏ 2. The height of the lidar points will be about the
same
❏ 3. Cars at the end of the row are usually the most
visible and easiest to spot and become less visible
as each car behind the first.

❏ Vehicles in parking lots without camera image but have clear lidar
and context should be labeled as Vehicle. Based on context we
can confidently assume that the lidar points are cars
Echo Points

Cuboid should Echo points appear when the SDC picks up extraneous data
include echo around a dynamic object. Echo points can be included when annotating
points when dynamic paths.
necessary that
correspond to the However, the cuboid should follow the LiDAR track of the actual object not
object in motion the surrounding echo points. Cuboids should be sized to the frame of the
fullest LiDAR image of the actual object at its closest distance to the SDC.

To figure out if an object has echo points, go through every frame of the
moving object. A “ghost” projection of the LiDAR should appear right next
to the object as it gets closer to the SDC. This extended projection gives
the illusion that the object is larger than normal.
(a) The images above show echo of the same object one frame from each
other.

(b) In the series of frames above, echo is present around the vehicle. The
cuboid length was extended to include the surrounding echo points.

This would be considered incorrect because the cuboid does not

accurately reflect the true size and shape of the vehicle. Instead, the
cuboid includes the after image or “ghost” LiDAR points of the moving
vehicle.
(c) The gif above shows us how to properly exclude echo points. With the
aid of a camera view (if present), locate the densest LiDAR cluster that
best represents the actual object.

How to annotate dynamic objects with echo:

1) Locate the frame with the fullest LiDAR projection as illustrated in

the left image above. This will be the size of the object across all
frames.
2) Then, go through each frame and place the cuboid in frames
where the echo-protruding side from the object is at the closest
point to the SDC.
More Examples:

Incorrect:

(a) The GIF above shows the incorrect way of annotating a dynamic
vehicle. Some echo points were included inside the vehicle which
makes this annotation incorrect.

Correct:
 (a) The GIF above shows the correct way of including the valid dense
points of the vehicle and excluding the echo points.

How to Label Shifting Data

Shifting data Shifting data occurs when the SDC makes a sharp turn, drives over
should be speed bumps and/or potholes, or if there is a miscalibration of the LiDAR
included within data. LiDAR points within the scene “shift” and move at each frame with
the parameters of the movement of the SDC. This results in objects appearing larger and
the cuboid exaggerated in height, length, or width.

1. Dynamic: Size based on best lidar frame then exclude lidar points
if necessary. (Pathing should be smooth to properly measure
momentum)
2. Stationary: Include all points by annotating in APC.

All LiDAR points should be included when shifting data is present.

Example #1:
Considerations: Max APC is on. There is enough LiDAR on the backside
to determine the full width and height of the vehicle. Since there is an
insufficient amount of data on the frontside of the vehicle, we cannot
determine the complete length of the vehicle. Therefore, standard sizing
should be used at 4.75m.

Because shifting data is difficult to determine the true size and shape of an
object, all of the LiDAR should be contained within the edges of the
cuboid.

Example #2: Shifting data with extremities

 Considerations: Max APC is on. Shifting data is present as suggested by
the exaggerated LiDAR projection. There is an extremity on top of the
vehicle as shown in the camera view. According to the instructions,
extremities are to be excluded.

Since there is a clear view of the camera, we can actually size the cuboid
to the camera view while keeping the LiDAR shift in mind. Be careful to
not include any extraneous LiDAR that is not part of the vehicle.

Examining the Back View, there seems to be a sizable vertical shift.

However upon further inspection, that is not the case. Instead, what we
have here is extraneous LiDAR near the back end of the vehicle.

If we take a look at the Overhead View, we can clearly see two protruding
side mirrors which suggests the true width of the vehicle.

How to Identify Bad Data

Bad data occurs when you start to notice reflection points underneath an
object but it is neither rainy nor foggy in your current scene.

There are certain conditions for when we can consider the reflections
[without the rain] as bad data.

● When the points appear scattered [check scattered data

information below] and are mostly consisting of green points that
follow the object throughout it's available lidar points.
 ● When green points that are lower than valid white lidar points
[meaning the points are to be considered as ground points.
Sometimes these green points we can find in bad data can even
go lower than the ground if you check how the cuboid looks like on
camera view.] start to shift and the shift can be considered as bad
shifting data [.5m shift or more].

Example of a Bad Data

(a) The GIF above shows the abnormal data. The lidar points are all
white and not green lidar points and it is stacked on top of each
other. To find this kind of data, turn on MAX APC - density 50 to
see all of the points clearly.
 (b) Turn on Dynamic APC [hotkey B] and scrub through the frames so
you can see the movement of the points in each frame. This way,
you can confirm if bad data is present.

Scattered Data

Shifting data Scattered data appears when LiDAR points are “stretched out” giving the
should be appearance of a wider, taller, or longer object. Scattered data is a type of
included within noise. This is not to be confused with shifting data, however they are very
the parameters of similar in appearance when Max APC is turned on.
the cuboid
Scattered data and shifting data differ in the following ways:

● Scattered data is captured when additional LiDAR points appear in

the surrounding area of the true shape of the object.
● The true placement of the LiDAR remains constant throughout all
50 frames whereas size is distorted across all 50 frames.

● Shifting data will vary in position per frame after the data collection
process.
● Shifting data will capture the true shape of the object but the
position of the object will drift from its original position.

To put it simply, scattered data collects extra LiDAR points around the
object but the position remains the same. Contrastly, shifting data
maintains the true size of an object per frame but its position moves
across all 50 frames. Thus, resulting in an exaggerated shape when Max
APC is on.
It is also important to note that scattered data does not necessarily affect
every cuboid in the scene compared to shifting data.

Example #1:

Considerations: Max APC is on. Standard sizing has been applied due to
insufficient LiDAR points at the front and backside of the vehicle.

As a general guide, if scattered data is unclear and does not allow for
proper cuboid sizing, use standard sizing. Additionally, if a camera view is
present, use the camera image to help size the cuboid.

Because this vehicle has scattered data, it is difficult to determine the

actual shape. However, we do have a camera view to help us determine
the width and height. The cuboid width is sized to standard sizing and just
so happens to align with the camera view. As for height, it was adjusted
based on the camera.

It is best practice to default to standard sizing and then adjust accordingly

to camera view, if there is one, while capturing as much of the LiDAR as
possible.
Articulated Vehicles

Articulated vehicles are two (or more) separate vehicles that are
attached together such as:
● Semi-trucks

Each section of this semi-truck should be annotated separately with the

“VEHICLE” label.

The front section of the semi-truck (where the driver would be in) should
be sized from the front bumper to back of the middle tires (without the
payload, the semi-truck would still have the middle tires).
The back section of the semi-truck (the payload) should be sized from the
end of the trailer to the end of the front section of the semi-truck.
Note: remember to exclude extremities (especially loading docks)
The overhead view, in the end, will look like this:

Note: it is okay if the two cuboids overlap just slightly. The two sections do
not have to be perfectly touching each other.

How to Size Articulated Vehicles with Unclear Lidar Points

(a) The image above shows that the object is an articulated vehicle
based on the camera view.
 (b) But checking on the lidar view, it is hard to determine the pivot
point. However, we can distinguish the head of the truck to the
trailer of the truck so we can safely separate them.

Correct Sizing:

(c) The first cuboid is sized to cover the head of the truck up to where the
pivot points should normally be (exercise best judgement) while the
second cuboid covers the entire trailer. Again, we can see slight overlap
that is necessary for these objects.

NOTE:
In cases where an articulated vehicle is visible in the camera but no
indication of where the connector between the front and back section is,
use ONE vehicle cuboid is used to annotate the whole thing.

(a) LiDAR view

(b) Camera view

● Vehicles with an attached object

 Each section of this articulated vehicle should be annotated separately
with a “VEHICLE” label similarly to how it is shown below.

Trailers

Trailers without wheels should be annotated and labeled as UNKNOWN

ZONE since they can potentially be towed by a truck.
Example 1

(a) The GIF above shows a missing trailer/container that is evident in

camera and lidar view. This should be annotated and labeled as
UNKNOWN ZONE and size up to its lidar points since in this case
we have dense points.
 For sizing:
a) Size it with standard length, if there are few lidar points and no
camera view
b) Size it based on camera view for length [if applicable]
c) If object exceeds standard sizing, then size it up to the available
points

Dumpsters

New Update! Annotate and label dumpsters as UNKNOWN ZONE.

Animals

ANY animal that Shape:

picks up LiDAR Animals should be labelled from head to tail, including legs. Only include the
points will use an body and legs of the animal, while EXCLUDING the head and tail.
“Animal” label
(Distance does Orientation:
NOT matter). Orientation/ directional heads of animals do not matter.
Example 1:
Unclear LiDAR and unclear Camera view

Example 2:
Clear LiDAR and Clear Camera view
Advanced Concepts

Using Camera/ ● When looking at a task, try using LiDAR View and Camera View to
LiDAR context to correctly position objects
help locate
missing objects
and their position

○ If we look closely in the second image, we can see that the

Vehicles are actually BEHIND the cones (we are able to
see the cones, meaning the Vehicles are behind it).
● This means that Vehicle 2 should actually be on the same
horizontal axis as Vehicle 1.

Common Errors:
Standard Sizing

Condition for using General Guidelines:

Vehicle standard 1. Use standard sizing in any dimension when you are less than 90%
sizing sure of that dimension (Length, Width, Height).
2. Position the cuboid as best you can given Camera Context and
existing LiDAR Points.
1.8W x 4.75L x 1.85H
3. Do not forget to take the scene into account

Has Camera No Camera

Clear Lidar Size to Lidar Size to Lidar

Example 2 Example 1

Unclear Size to Camera if Camera Standard Sizing

Lidar is good, otherwise use Example 1
Standard Sizing.
Example 2 (camera good)
Example 3 (camera not
good)

Clear Lidar Points = “More than 90% confidence in the exact size of the
cuboid”

Example 1: Unclear Lidar & No Camera / Clear Lidar & No Camera

Considerations: With max APC - LiDAR points show the outline of the
vertical back of the car as well as wheels. This gives enough context to
size the cuboid “tire to roof”.

The LiDAR points display half of the horizontal back of the car, as well as
some of what appears to be the wheels on the side. However, it does not
necessarily give us enough evidence to determine the sides of the car.

We see no evidence of a front bumper in LiDAR.

For Height: LiDAR points give sufficient evidence for the original width
and height of the cuboid. Therefore, we size to the LiDAR points.

For Width and Length: Vehicle length is uncertain since we cannot

clearly see a front bumper. Therefore, we should default to standard
length of 4.75m.

The Vehicle width is uncertain since we cannot see a clear back outline or
sides of the car - hence we fit standard sizing width 1.8M over existing
LiDAR points.

Note: Using width standard sizing and sizing based on lidar both get us
close to 1.8m width.
Example 2: Has LiDAR & Camera / Unclear LiDAR & Camera

Considerations: Camera view is present and APC is on - LiDAR points

sparsely outline the vertical back of the car in Side View. LiDAR points in
Side View also indicate what appears to be the front wheel; however, we
cannot be certain.

We do not have enough information in LiDAR to find the roof of the car, so
we cannot size ”tire to roof”. Camera view appears accurate, so we start
with standard sizing and size to camera as best as we can. We include
what we might consider to be the front tire.

The LiDAR points display the horizontal back of the car in Overhead View.
In Back View, they sparsely outline the shape of the car. The Overhead
View gives us enough confidence to size to Lidar. The width sizing
conveniently is the same as standard sizing.
We can see no evidence of a front bumper in LiDAR.

For Height: LiDAR points do not show a roof position and give <90%
confidence for tire position. Camera is available and seems accurate, so
we size for Camera Accuracy while including all LiDAR points in the tire. If
there was no camera, we would use the standard sizing for height of
1.85m.

For Width: LiDAR points show the back outline of the car in Overhead
View. The outline gives us enough confidence to size the width to LiDAR.
This is conveniently the same as 1.8m width standard sizing.

For Length: LiDAR shows us the back outline of the car; however, it does
not give us any information on the front of the car. This does not give us
enough to size “bumper to bumper”. Hence, we should use 4.75m length
standard sizing and align the cuboid to the back outline bumper.

Example 3: Unclear Lidar & Camera

Considerations: Camera view is present but occluded in all frames and
APC is on - LiDAR points outline a roof in Back View and Side View, but
do not outline tires. Hence, we cannot confidently size “tire to roof,” so we
should use standard sizing. Camera view is too occluded to help
determine size.

Even with APC on, LiDAR points only show one side of the vehicle. We
cannot find the other door, so we cannot size “door to door.” Hence, we
should use standard sizing.

Even with APC on, LiDAR points do not show the front or back bumper.
Since we cannot find the bumper, we cannot size “bumper to bumper.”
Hence, we should use standard sizing.

In the scene view around the car, we can see the bush in front of the car.

For Height: LiDAR does not show enough data to find the tires. Hence,
we cannot size “tire to roof”, so we should use 1.85m height standard
sizing. We position the cuboid to align with what we can see is the roof.

For Width: LiDAR only shows one door. The other door has no lidar
points. Hence, we cannot size “door to door”, so we should use 1.8m
width standard sizing. We position the cuboid to align with what we see as
one door.

For Length: LiDAR does not show the front bumper. Hence, we cannot
size “bumper to bumper”, so we should use 4.75m length standard sizing.

We see the scene has LiDAR points not associated with the car at the
front of the cuboid. We position the cuboid with these points in mind, even
if those push these cuboid past the back of the car.
Guides, Tips, and Advice:

Missing Cuboids

Using Radar points General guideline: if a vehicle can be seen in

camera view but there are no LiDAR points,
turn on radar to see if there are any radar
points nearby for the vehicle.

If there are no radar points nearby, the

vehicle should not be annotated.

If there are radar points nearby, there is a

two-step process of labeling the vehicle.
1. Use a standard sized vehicle to
position for camera accuracy.
2. Use camera view to adjust sizing of
the vehicle for any visible dimensions

Here is an example of spotting a missing

vehicle in camera

Here is an example of labeling the missing

object that had no LiDAR but had radar
Pedestrians sitting on the ground

Use the “pedestrian” ● If a pedestrian is sitting on the ground, bench, ledge, then use
label. the “pedestrian” label.

UPDATE!!
Use the “PEDESTRIAN
LYING DOWN” label for
pedestrians laying on
benches and laying or
sitting on the ground.
Excluding Extremities

The loading ramp of large vehicles such as

these are considered extremities.

Note: most of these vehicles have a large

storage unit in the back

The red circled part of this large vehicle is a

loading ramp that should be excluded.
An extremity is present on top of this vehicle.
We exclude all extremities when sizing
vehicle cuboids.

In this case, we use Side view and Back

view to help us find the true shape of the
vehicle.

If we look carefully in the camera view and

the side view, we can see what looks like a
protrusion on the back side of the vehicle. It
would make sense to exclude the back
protrusion if it was a loading dock. However,
since it appears to be an average sized truck
according to the camera view, we can safely
include the entire length of the vehicle.
Objects entering/exiting a building with an Unknown Zone

Rule: any objects inside a building should not In this GIF, the vehicle is entering a building.
be annotated even if it has LiDAR and/or The moment the vehicle’s LiDAR is fully
camera view. inside the Unknown Zone, the annotation of
the vehicle is finished.
Following this rule, dynamic objects
entering/exiting a building should be
annotated while the object’s LiDAR points are
outside of the unknown zone.
Other_Human

Any humans that do not fit into the Shape:

Pedestrian, Motorcyclist, or Cyclist labels. Use the fixed shape cuboid to annotate the
person
INCLUDES skateboarders, skaters,
Cuboid MUST BE centered on the person’s
non-motorized scooters, Segway’s, and
hips and placed around their torso/trunk
pedestrians in wheelchairs, or lawn mowers.
If you cannot see the pedestrian’s hips, do
EXCLUDES babies in strollers. your best to estimate where their hips are
Other_Human label is only used if the
pedestrian is self-propelling. Arms and legs are irrelevant to the
pedestrians’ fixed size cuboid.

Orientation:
Can be disregarded when labeling other
humans.
Bicyclists/Motorcyclists

Bicyclists: ● Bicyclists tips

Only annotate ○ Clear LiDAR points, pedal-to-pedal sizing
bicyclists when there
is a rider ON the
bicycle

Motorcyclists:
Only annotate
motorcycles when
there is a rider ON
the motorcycle

Do NOT annotate
motorcycles when
riders are NOT
present (i.e Parked
motorcycles)
Workflow
Workflow

Step by step guide for reviewing an entire task:

● Open the task
● Press R, T, Y (optional), L, C
○ R - Turn off Radar
○ T - Top down view
○ Y - Change LiDAR points color (optional)
○ L - Change cuboid color by label
○ C - Camera View
● While the frames are loading, look at the different Camera Views (Camera 1-6)
○ Get context on the entire scene
■ Where are you / What type of scene is this?
■ City Scene Tips
○ Pay close attention to the sidewalks/parking
lots/alleyways/intersections and buildings
■ Residential Scene Tips
○ Pay close attention to driveways/behind
fences/intersections/
■ Night Scene Tips
○ Pay close attention to headlights/sidewalks
● Once all the frames are loaded, scroll through the entire scene in each Camera
View
○ Determine if the task might have shifting data and/or scattered data
■ How to tell if the task has Shifting Data?
■ If the SDC makes any sort of turn, then there is a high chance of
shifting data
■ How to tell if the task has Scattered Data?
■ Scattered Data can happen whenever we do not have a “tell” on
when a task will have Scattered Data.
■ The only way to tell is by clicking on the individual cuboid
○ Click on Cuboid > Turn on Dynamic APC > Hover over
frames
■ Does it look like the object is moving (when it is
stationary?)
● If yes, then it is Shifting Data
● If no, then it is Scattered Data
■ Example of Scattered Data
○ Note down any obvious errors (eg. backwards cars, incorrect labels, obvious
incorrect sizing and positioning, missing objects, etc.)
■ Where?
■ Look at the bottom right of the task -> Click “Make Changes” ->
Use the feedback section
■ Example:
■ Missing vehicle in Camera 0 - Frame 27
■ Vehicle 1111 sizing error in Camera 4 || Vehicle abcd positioning
error in Camera 3 || Missing Pedestrian Camera 6 - Frame 13
○ Look for moving objects on the road (distant or occluded vehicles)
■ Example
■ Dynamic object behind Vehicle 1111 (check for LiDAR/Radar
points)
○ Look for moving objects on the sidewalk (distant or occluded pedestrians)
■ Example
■ Missing pedestrian Camera 6 - Frame 48
 ○ IF the task has too many obvious errors that will take more than 2 hours to fix,
then reject the task.
○ IF the task does not have too many obvious errors, then continue with the task.
● Go back into LiDAR View, turn on APC by pressing the “P” key and work by
camera sections

○
● Start working in “Section 1”
○ Fix cuboids that you’ve noted down for Camera 0//Section 1
○ Click on the cuboid CLOSEST to the SDC in Section 1
■ Stationary or Dynamic?
■ Stationary (Turn on APC, “P” Key):
○ Correct Label?
○ Correct Sizing?
○ Correct Orientation?
○ Correct Positioning?
■ Dynamic (Turn on Dynamic APC, “B” Key):
 ○ Correct Label?
○ Correct Sizing?
○ Correct Orientation?
○ Correct Positioning?
○ Smooth Pathing?
■ Fix cuboid (if needed)
■ When finished, press Shift + Spacebar
○ Repeat for the entirety of Section 1 with the next closest cuboid
● Start working in Section 2, Section 3, Section 4
○ Same steps as above
● Last Check - Scroll through the entire scene toggling APC on/off
○ Turn on Quality Assistant (“Q” Key)
■ Go through the list for any unchecked cuboids
■ If cuboid is unchecked, click the search icon and fix (if needed)
and press Shift + Spacebar
○ Look for missing stationary objects
■ Go down every single road and intersection
■ Check driveways for parked cars
■ Check at stop lights for stopped cars at intersections
■ Check parked lanes, with Radar on, if vehicles are getting
picked up
○ Look for trails of dynamic pathways that weren’t annotated
○ Check the Unknown Zones
■ Reflections
■ Is this UZ covering all points of the reflection?
■ Is the UZ around long enough to cover the entire reflection?
■ Overlapping
■ Are the objects actually overlapping by more than 20%?
■ Is the UZ around during the time that the objects are
overlapping?
■ Objects inside of buildings
■ Are these objects annotated and covered with an UZ?
○ If yes, then delete the individual object(s) and check if
the UZ is labeled correctly.
○ If objects are not annotated and not covered by a UZ,
then make an UZ that covers the entire section that
objects (while excluding the building walls)
 Reviewing Stationary, Dynamic and Pedestrians

Reviewing a Stationary Cuboid:

● Turn on APC (“P” Key)
● Click on the cuboid
○ If there is a camera view, then check the label; if there isn’t a camera view,
then use the lidar view to check the label.
● Determine if cuboid is “Normal”, Shifting Data or Scattered Data
○ If “normal” > size with standard rules
○ If Shifting Data > include ALL LiDAR points
○ If Scattered Data > size based off Camera View (if applicable) and use
standard sizing
● Check the Side View
○ Vehicles:
■ Is there clear enough LiDAR points to size from bumper to bumper?
■ If yes, size based off of LiDAR points.
■ If no, then use standard sizing for length.
■ Is the vehicle being obstructed?
■ If the vehicle still has clear LiDAR, size based off of LiDAR and
use standard size when needed.
■ If the vehicle does not have clear LiDAR, use standard size.
● Switch to the back view (v)
○ Vehicles:
■ Can you see both sides of the side view mirrors?
■ If yes, exclude them.
■ If not, use standard size or size to LiDAR.
● Open Quality Assistant Panel (“Q” Key)
○ Double check to see if ALL stationary objects have been checked off

Reviewing a Dynamic Cuboid:

● Go to LiDAR View
● Toggle Cuboid Trails On (“O” Key)
● Click on the cuboid
● Turn on Dynamic APC (“B” Key)
○ Look at the side view for this cuboid
 ■ Are all the points included inside of the cuboid?
■ If there are points sticking out, scroll to the problem area by
hovering over the frame bar.

■ Is the cuboid affected by echo or noise?

■ If echo, follow the real body of the vehicle.
■ If noise, exclude those LiDAR points.
■ Can you see the general shape of the object fairly clearly?
■ If the shape of the object looks blurry and unclear, check the
positioning to make sure all LiDAR points are lined up.
○ Scroll through the pathway by hovering over the frame bar, OR press K to view
all the frames at once.
■ Do the points hit the same part of the car?

○ Look at the back view for this cuboid

 ■ Are all the points included inside of the cuboid?
■ If there are points sticking out, scroll to the problem area by hovering
over the frame selector window.
■ Vehicles:
■ Can you see both sides of the side view mirrors?
■ If yes, are they both included in the cuboid? Are they
symmetrical on both sides of the cuboid?
■ If no, is the cuboid tight with the sides without mirrors?
Are you sure that the mirrors aren't visible the entire
scene?
■ Pedestrians:
■ Positioning
■ Are all the points tight, and do they resemble a person?
■ Sizing
■ Pedestrian’s cuboid a fixed size (0.5m x 0.5m x0.5m)?
● Look at the object’s path
○ Check entire path
■ Are there any kinks in the pathway? Is the pathway mostly smooth?
■ If kinks, smooth them out if possible.
■ If the vehicle is turning, is the curve smooth?
■ If no, use more keyframes if needed, while utilizing interpolation
to help create a smooth curve.
○ Scroll through the pathway by pressing + and -
■ Watch the cuboid in LiDAR view - does the motion make sense?
■ Watch the cuboid in Camera view - does the motion generally match
what you see in camera view?
○ Use +/- to navigate and use predicted frames to look for missed lidar points in
later/earlier frames.
● Open Quality Assistant Panel (“Q” Key)
○ Double check to see if ALL dynamic objects have been checked off

Reviewing a Pedestrian
● Go to LiDAR View
● Toggle Cuboid Trail On (“O” Key)
● Click on the pedestrian cuboid.
● Look at the side panel.
 ○ Turn on Dynamic APC (“B” Key)
■ Check if LiDAR points are all contained within the cuboid across all
frames.
● Tip: Use grid view (“K” Key) to see all frames at once and adjust
keyframes with lidar points that are not aligned with the rest.
■ Make sure cuboid sizing is fixed (0.5m x 0.5m x 0.5m)
○ Press “T” for top-down view
■ Check the smoothness of the pedestrian path.
■ Check z-index in the side panel for smoothness.
● If SDC goes over bumps/dips, then z-index may be jagged.
■ Press “O” to stop showing all frames.
■ Use +/- to navigate and use predicted frames to look for missed lidar
points in later/earlier frames.
● Open Quality Assistant Panel (“Q” Key)
○ Double check to see if ALL pedestrians have been checked off

Reviewing a Unknown Zone

● Check for areas that need Unknown Zones (i.e. buildings with low windows)
○ Press T to go into the Top-down view.
■ Check inside buildings for lidar points that look like annotatable objects.
(Car dealerships, shops with large glass windows)
■ If you suspect reflection is present. Press C to go into Camera
view.
■ Confirm the presence of mirrors, windows, or any large
reflective surfaces (i.e. side of a large bus) for suspected
reflections.
■ Turn on APC
■ Is the reflection dynamic? If so, the UZ should cover all dynamic
points and mark as stationary
■ Is the reflection stationary? If so, the UZ should also be
stationary across all frames.
○ If more reflections are present then make the UZ bigger
to include those reflections.
○ Overlapping
■ Are the objects actually overlapping by more than 20%?
■ Is the UZ covering entirely both overlapping cuboids while they are
overlapping?
■ Is the UZ around during the time that the objects are overlapping?
○ Objects inside of buildings
 ■ Are there objects that are annotated inside of buildings?
■ If so, delete them and put an UZ over the building (while
excluding the walls).
■ If not, just put an UZ over the building (while excluding the
walls).
○ Covered Vehicles (vehicles covered with a car cover/tarp)
■ Is the vehicle covered with a car cover/tarp?
■ If so, label it as an UZ and cover all LiDAR points.
■ If not, label it as a vehicle and follow sizing rules.
● Open Quality Assistant Panel (“Q” Key)
○ Double check to see if ALL unknown zones have been checked off