Chronological Review
Scheduling Overview
The project included various milestones the
needed to be completed separately before the entire truck code could be put
together into a working system. As
can be seen in Gantt chart (Table 1), there are multiple equally critical paths
that the project schedule followed.
The three main setup pieces: a functioning GUI that could be used to
send parameters and course descriptions to the robot, a robust vision algorithm
that could interpret the pixel data given from the camera and give pylon sizes
and locations, and the navigation or control algorithm that would be use the
data given from the vision and the encoder to successfully navigate around the
course. Those three main
components were implemented concurrently, but after an initial learning and
design phase. This process also
led to an interesting event when all of these pieces were completed it only
took a few days of work to put them all together just in time for the first
real practice competition in the Garden Court competition area.
Design, Learning, and Brainstorming Phase
We started the semester with multiple
meetings and various design brainstorming activities together as a team. These meetings were really the
foundation for all of our algorithms and design decisions for the rest of the
semester. During these meetings we
completed the business portion documents that required discussion of important
design elements for our particular project. We decided during these meetings:
·
We would overall design the pylon
recognition system using color segmentation
·
The navigation should drive
directly towards the pylon in order to easily maintain on course and make the
best possible vision calculations
·
The GUI should include parameters
for all of the useful parameters in the program including access to change the
speeds and to change the camera registers
As individual team members we dedicated an
individual person to be main caretaker of one of each of these three critical
tasks, with the forth person on our team receiving the assignment to be the
hardware master that could be the reference that the rest of us would consult
while we were building our pieces.
This separation of the tasks led to a little bit of time during the
middle of the semester when we were together less often to collaborate since we
were all busy working on our own individual tasks.
Pylon Recognition Vision Software
Our vision specialist quickly came up with a
fantastic and simple way to build an algorithm in simple code that could be
easily ported to any language. The
vision algorithm design used a method that built on edge detection and color
detection methods but before doing any of those the image was first put through
a sort of preprocessor. First, on
the first pass through the image the algorithm completely skipped the one-third
or so of pixels and the bottom one-third pixels, leaving only the middle third
of pixels to
analyze. Also on this
first pass of the data it kept running totals of the average color of each
column in this middle third of the photo.
We were able to assume the pylon would always be in this middle third of
the photo since a few of the course specifications including flat or level
ground and the fact that the camera is mounted on the top of the truck and
always approaches pylons from the same vertical angle. With the average color of each column
the width and x-coordinate of the largest matching pylon color area could then
be found. We used a fairly broad
range of colors for both the orange and the green pylons in the vision
software. We counted on the fact
that when we turned the corner around a pylon and completed a turn that the
next pylon would be in the camera view.
Because the pylons were brightly colored under most lightly situations
they would easily make the columns in which the pylon is located the largest
width that correctly matches the color thresholds for the current pylon.
On a second pass through the image all of the
pixels to the left and right of the width of the pylon were discarded and the
full height of the image was then analyzed in the section. The algorithm then calculated average
color of each row in that vertical segment of the photo, producing a single
one-dimensional array of data containing the colors of the height of the
pylon. The height was then
calculated by simply taking the largest color matching section on that 1
dimensional array. Even a single
pixel that did not match the correct color would make a disjointed section;
doing so provided an easy way to filter out any reflection in the floor since
the pylon always included a small white base that would disrupt the continuous
pylon image that can occur when viewing the pylon on a reflective surface.
For the first practice competition in the
Garden Court area, we actually used entirely the width and x-coordinate to make
all of the navigation decisions.
This worked fairly well and we considered removing the rest of the
vision algorithm to make it even faster. However, we found that because of the
different lightly situations the second part of the vision algorithm
significantly increased our ability to accurately decide the correct height of
the pylon.
Robot Control and Navigation
A fundamental piece of the project was
actually controlling the steering angle and driving speed of the robot in order
to successfully navigate the complicated pylon courses. After deciding early in the semester
that we would drive directly towards the pylon we simply had to build an
algorithm that could use the vision data to perform that principle action. First we created a few principle
parameters that would be used throughout the control code. This including the parameters for
driving straight speed, turning speed, number of encoder ticks to count per
radian, and the distance before pylon to start turning. We were able to first
successfully run the control code through the given simulator. Which allowed us to do some initial
adjustments to our design and test our idea of driving directly towards the
pylon.
We implemented this design with a simple state
machine. During iterations through the main loop it would call the control
state machine after analyzing a new frame of data from the camera. The state machine consisted of 6-7
states that the truck could be in at any given moment. The states included:
1.
Pylon searching. During this
state it was assumed that the pylon was in the given view. However it would only move out of this
state when the vision software successfully recognized a pylon. If no pylon was currently recognized
our final design simply had the truck continue straight at a slower speed while
giving the vision algorithm more passes to analyze the data.
2.
Pylon following. During this
state the truck would make simple steering adjustments in order to maintain the
pylon x-coordinate plus the width of a pylon in the center of the truck. Bigger distances from the center of the
screen would mean larger adjustments of the steering angle during this
state. The truck would continue in
this state until the distance from the pylon was calculated to be less the
threshold set in the global parameters as the distance before the pylon to
turn.
3.
Setting up the turn. There were two hard coded states that
depended only on the parameters set for the truck. These two steps including turning slightly away from the
pylon in the opposite direction then needed to be turned around the pylon and
then driving straight a set distance in order to line up with the pylon nearly
perpendicular from the line at which it was driving towards the pylon. If the vision consistently gave
accurate values for the distance of the pylon these state worked perfectly
every time. We assumed during this
state that we would lose site of the pylon and we therefore did not use any of
the vision calculations while in these states.
4.
Dead-Reckoning turn around the
pylon. Upon entering this state we simply saved the current Encoder value plus
the number of ticks we needed to travel to complete the turn. This extra number of ticks was
calculated by the angle given in the course description. When we approached a pylon of the same
color as the previous pylon we added an extra 10 degrees to every turn due to
the manner that we were entering the turns from an angle slightly less the
perpendicular to the line straight from the previous pylon. When the truck
approached a pylon of a different color then the previous pylon we added an
extra 45 degrees to the turn in order to compensate for the fact that in this
case the truck was crossing over the straight line between the two pylons and
actually entered the turn much further back in the turn. The number of ticks per radian was set
by a global parameter that we also could change through our gui. This variable of the ticks per radian
needed to be adjusted when we tested new faster speeds navigating the course. When the calculated number of ticks had
been completed the truck then moved back into the pylon searching state.
External GUI
The GUI was another important part of the design
as it was our means of communicating with the truck without having to pre-program
everything into the actual code on the truck. This allowed for much faster testing of new ideas and gave
us the ability to make adjustments based on test. This turned out to be an even more critical component then
we even originally planned since the lighting conditions varied extremely in
the garden court even from one minute to another. Our goal was to parameterize every useful piece of our robot
through the GUI so that we would never have to re-download the software to the
truck. By the time we were working
at the final competition we could change the speed, angles, course description,
any camera setting (luminosity turned out to be the most useful camera
settings), current state, and of course send a single command to immediately
stop the truck.
Another important feature of our GUI was the
ability to both download a full image from the truck and the ability to just
download the current values calculated by the vision algorithm and displays the
exact location on the GUI of calculated pylon width, height, and location. This was invaluable in making sure the
vision algorithm was working correctly and testing new camera settings in order
to learn that turning the luminosity higher when looking towards the large
natural light windows. Our GUI also
had a place that could be used to manually control the truck steering angle and
driving speed.
Putting it all together and testing
During the coding and after we completed each of
the three main separate tasks our hardware specialist helped us all make sure
that we were all using the structures and communication methods that would work
together throughout all of the different pieces. Over the course of a couple of days we put all of our code
together on the truck with a little bit of coding glue that was required to
connect the pieces together.
Overall putting the pieces together turned out to be a fairly
straightforward process since we had all made thorough and compatible designs
of the individual pieces. We were
the most successful team at the first practice competition, being able to
successfully navigate all of the courses presented by the competition
moderators. Our final design ended
up being only 752 lines of code. (Of course the truck comes with many more
lines of code in its complete software build. But our main.c, vision.c and
control.c with headers only add up to 752 lines of code.) An additional 404
lines of code was written using GTK and pylon for the GUI. We feel this simplicity in the design
was a key factor in our early success during the semester to navigate the
courses.
The last two weeks of the project were mainly
spent on tuning and upgrading our algorithms to work in the bright one-sided
light of the garden court and finding ways to increase the speed while still
having time to capture frames and compute the location of the next pylon. We did this testing mainly just with the
parameters in the GUI. These tests
were performed both in the lab and in the actual garden court during times when
we found that area available.
What we would do differently
In the end there were a few things we would
do differently. In regards to control, if still allowed, build in some
estimated distances checking and use a PID loop for velocity control. In the
vision code we would read vision frames straight out of the original frame
memory location instead of copying the data
Really analyze our vision algorithm, it
seemed to be an issue when running the truck faster, possibly the vision was
not getting enough frames per second to handle faster speeds. We would finish a
more thorough debugging tool for the GUI and also we wanted to get our code
loaded into the ROM memory, so that we can change batteries without having to
re-download.
Project Assignment Breakdown
Our project had start date of Monday, January 12,
2009 with the scheduled completion date of Thursday, April 2, 2009. A complete breakdown of assignments as
they were scheduled is given in Table 2.
We were comfortable with the workload assigned to each person. The hours in Table 2 give estimated
workloads in hours of each member of the team throughout the semester. Each member of the 4-person team would
be required to spend around 500-600 hours on the project. The actual hours spent were far less,
but since we actually assigned the pieces according to days (8 hrs = 1 day) the
assignments in Table 2 remained proportionally correct to the actual workload
required by each person in order to successfully complete the project by the
scheduled completion date.
Table 2: Who Does What
|
Resources and Assignments |
Start |
Finish |
Work |
|
Brandon |
Fri 1/16/09 |
Wed 4/1/09 |
560 hrs |
|
Write design doc |
Fri 1/16/09 |
Mon 1/19/09 |
16 hrs |
|
Write Design doc for GUI |
Tue 1/27/09 |
Wed 1/28/09 |
16 hrs |
|
Learn GTK |
Fri 1/23/09 |
Tue 1/27/09 |
24 hrs |
|
Learn Python |
Fri 1/23/09 |
Thu 1/29/09 |
40 hrs |
|
Create GUI layout |
Fri 1/30/09 |
Tue 2/10/09 |
64 hrs |
|
Research Serial Comm |
Wed 2/11/09 |
Thu 2/12/09 |
16 hrs |
|
Connect GUI to communication methods |
Tue 2/17/09 |
Wed 2/18/09 |
16 hrs |
|
Working E-Stop |
Thu 2/19/09 |
Fri 2/20/09 |
16 hrs |
|
Drive from keyboard |
Thu 2/19/09 |
Fri 2/20/09 |
16 hrs |
|
Research Camera Registers |
Tue 1/27/09 |
Thu 1/29/09 |
24 hrs |
|
Set camera registers from GUI |
Thu 2/19/09 |
Mon 2/23/09 |
24 hrs |
|
RGB565 to RGB888 |
Tue 2/3/09 |
Tue 2/3/09 |
8 hrs |
|
RGB888 to HSV |
Wed 2/4/09 |
Thu 2/5/09 |
16 hrs |
|
Research image formats |
Tue 2/3/09 |
Fri 2/6/09 |
32 hrs |
|
Test good velocity settings |
Thu 2/19/09 |
Mon 3/2/09 |
64 hrs |
|
Integrate vision and control code |
Thu 3/12/09 |
Fri 3/20/09 |
56 hrs |
|
Course control code |
Thu 2/19/09 |
Thu 2/26/09 |
48 hrs |
|
Navigate simple pylon course |
Mon 3/23/09 |
Tue 3/24/09 |
16 hrs |
|
Tune vision algorithm |
Wed 3/25/09 |
Fri 3/27/09 |
24 hrs |
|
Navigate complex pylon course |
Mon 3/30/09 |
Wed 4/1/09 |
24 hrs |
|
Luke |
Fri 1/16/09 |
Thu 4/2/09 |
528 hrs |
|
Write design doc |
Fri 1/16/09 |
Mon 1/19/09 |
16 hrs |
|
Make presentation |
Tue 1/20/09 |
Wed 1/21/09 |
16 hrs |
|
Research OpenCV |
Fri 1/30/09 |
Mon 2/2/09 |
16 hrs |
|
RGB565 to RGB888 |
Tue 2/3/09 |
Tue 2/3/09 |
8 hrs |
|
RGB888 to HSV |
Wed 2/4/09 |
Thu 2/5/09 |
16 hrs |
|
Research image formats |
Tue 2/3/09 |
Fri 2/6/09 |
32 hrs |
|
Implement vision algorithms |
Mon 2/9/09 |
Mon 3/9/09 |
168 hrs |
|
Vision competition |
Tue 3/10/09 |
Tue 3/17/09 |
48 hrs |
|
Port vision algorithm on car |
Tue 3/10/09 |
Wed 3/11/09 |
16 hrs |
|
Test good velocity settings |
Thu 2/19/09 |
Mon 3/2/09 |
64 hrs |
|
Integrate vision and control code |
Thu 3/12/09 |
Fri 3/20/09 |
56 hrs |
|
Navigate simple pylon course |
Mon 3/23/09 |
Tue 3/24/09 |
16 hrs |
|
Tune vision algorithm |
Wed 3/25/09 |
Fri 3/27/09 |
24 hrs |
|
Navigate complex pylon course |
Mon 3/30/09 |
Wed 4/1/09 |
24 hrs |
|
Verify Vision is using enough fps |
Thu 4/2/09 |
Thu 4/2/09 |
8 hrs |
|
Jaren |
Fri 1/16/09 |
Wed 4/1/09 |
590 hrs |
|
Write design doc |
Fri 1/16/09 |
Mon 1/19/09 |
16 hrs |
|
Download base code |
Thu 1/22/09 |
Thu 1/22/09 |
8 hrs |
|
Run Hello World |
Fri 1/23/09 |
Mon 1/26/09 |
16 hrs |
|
Display Video via USB |
Fri 1/23/09 |
Mon 1/26/09 |
16 hrs |
|
Research Serial Comm |
Wed 2/11/09 |
Thu 2/12/09 |
16 hrs |
|
Connect to car via wireless |
Fri 2/13/09 |
Mon 2/16/09 |
16 hrs |
|
Connect GUI to communication methods |
Tue 2/17/09 |
Wed 2/18/09 |
16 hrs |
|
Working E-Stop |
Thu 2/19/09 |
Fri 2/20/09 |
16 hrs |
|
Control Velocity |
Tue 2/17/09 |
Wed 2/18/09 |
16 hrs |
|
Control Steering |
Tue 2/17/09 |
Wed 2/18/09 |
16 hrs |
|
Drive from keyboard |
Thu 2/19/09 |
Fri 2/20/09 |
16 hrs |
|
Research Camera Registers |
Tue 1/27/09 |
Thu 1/29/09 |
24 hrs |
|
Set camera registers from GUI |
Thu 2/19/09 |
Mon 2/23/09 |
24 hrs |
|
RGB565 to RGB888 |
Tue 2/3/09 |
Tue 2/3/09 |
8 hrs |
|
RGB888 to HSV |
Wed 2/4/09 |
Thu 2/5/09 |
16 hrs |
|
Read Encoder |
Tue 1/27/09 |
Thu 1/29/09 |
24 hrs |
|
Create Math Tables |
Tue 1/27/09 |
Wed 1/28/09 |
16 hrs |
|
Test Control on Block |
Mon 2/23/09 |
Tue 2/24/09 |
16 hrs |
|
Test good velocity settings |
Thu 2/19/09 |
Mon 3/2/09 |
64 hrs |
|
Port C# to C for control code |
Tue 3/3/09 |
Wed 3/11/09 |
56 hrs |
|
Integrate vision and control code |
Thu 3/12/09 |
Fri 3/20/09 |
56 hrs |
|
Course control code |
Thu 2/19/09 |
Thu 2/26/09 |
48 hrs |
|
Follow a pylon |
Thu 3/12/09 |
Fri 3/13/09 |
6 hrs |
|
Navigate simple pylon course |
Mon 3/23/09 |
Tue 3/24/09 |
16 hrs |
|
Tune vision algorithm |
Wed 3/25/09 |
Fri 3/27/09 |
24 hrs |
|
Navigate complex pylon course |
Mon 3/30/09 |
Wed 4/1/09 |
24 hrs |
|
Steve |
Fri 1/16/09 |
Thu 4/2/09 |
536 hrs |
|
Write design doc |
Fri 1/16/09 |
Mon 1/19/09 |
16 hrs |
|
Setup Website & maintain website |
Thu 1/22/09 |
Fri 3/23/09 |
36 hrs |
|
Test Control on Block |
Mon 2/23/09 |
Tue 2/24/09 |
16 hrs |
|
Test good velocity settings |
Thu 2/19/09 |
Mon 3/2/09 |
64 hrs |
|
Download simulator |
Fri 2/6/09 |
Fri 2/6/09 |
8 hrs |
|
Write control Algorithm |
Mon 2/9/09 |
Thu 2/26/09 |
112 hrs |
|
Hook up control code to simulator |
Fri 2/27/09 |
Mon 3/2/09 |
16 hrs |
|
Port C# to C for control code |
Tue 3/3/09 |
Wed 3/11/09 |
56 hrs |
|
Integrate vision and control code |
Thu 3/12/09 |
Fri 3/20/09 |
56 hrs |
|
Course control code |
Thu 2/19/09 |
Thu 2/26/09 |
48 hrs |
|
Follow a pylon |
Thu 3/12/09 |
Fri 3/13/09 |
36 hrs |
|
Navigate simple pylon course |
Mon 3/23/09 |
Tue 3/24/09 |
16 hrs |
|
Tune vision algorithm |
Wed 3/25/09 |
Fri 3/27/09 |
24 hrs |
|
Navigate complex pylon course |
Mon 3/30/09 |
Wed 4/1/09 |
24 hrs |
|
Verify Vision is using enough fps |
Thu 4/2/09 |
Thu 4/2/09 |
8 hrs |