Penn Electric Racing
Penn Electric Racing is a team that participates in Fomula SAE Electric, a competition where students develop custom Formula-style Electric racecars. You can also learn more about our team on the Penn Electric Racing website. I've been on the team for the past five years, working on a wide variety of projects and doing whatever was necessary to get the racecar ready for competition. I've probably spent more time on Penn Electric projects than I have on every other project I've done at Penn combined, so it's impossible to cover everything. This summarizes my main contributions to the team over the years, but obviously the racecar is a huge team effort and I can't emphasize enough how much time everyone else has put in to make each racecar a reality. It has been a pleasure working with everyone and I'll always remember my adventures on Penn Electric Racing.
- 2018-19 (REV5): Working on Autonomous, Debugging Motor Controllers, Preparing for REV5 Testing
- 2017-18 (REV4): Reviewing PCBs, Developing Motor Controllers, Struggling at Competition
- 2016-17 (REV3): Leading the Software Team, Creating STM32F7 Libraries, Winning at both Competitions
- 2015-16 (REV2): Failing at Data Acquisition, Implementing Code Generation, Data Analysis
- 2014-15 (REV1): Building a Charger, Designing a DC-DC Converter, Graphing live CAN data
2014-2015
Before arriving at Penn, a friend had mentioned that he was on a club that built electric racecars. At first, I thought they were probably little RC racecars, but then I got to see the actual racecar they had built and I joined the team as soon as I could. Penn Electric Racing has been around for over three decades, working on everything from solar cars to an electric scooter. At the time I joined, the team was quite small and had just reorganized to start competing in the FSAE electric car competition one year before. The first racecar (REV0) hadn't been finished the year before (FSAE has a steep learning curve) and I immediately started helping to get it driving. My first project was getting the charger to work, specifically a box with a custom circuit board that was responsible for both communicating with the battery management system on the car and the charger itself. I wrote a seven segment display library, learned about CAN and also did the low voltage and high voltage wiring. Eventually I got it working and it's been in use for the last four years with only a couple wiring fixes as things broke over the years. We're finally upgrading it for REV5 this year, because we're switching to a new power supply.
A couple weeks after that was finished, the car was ready to drive and I got to see REV0 drive for the first time, not counting the few minutes it barely drove at competition with most things disabled. My next project was using a sketch of a schematic to design my first PCB for the DC-DC converter of REV1 to convert the 300V battery voltage into 12V for the low voltage system.
As we started testing REV0, we started gathering a significant amount of data and needed a way to view it, so I quickly threw together a basic graphing program. Over time, I've gradually added features, including streaming data as the car is driving, exporting subsets to excel and new file format support.
I also worked on a wide variety of other projects to get the car ready for competition, while also busy manufacturing a Stirling engine for a class. After a 20 hour drive to Lincoln Nebraska, we arrived and as a freshman I mostly just watched as the older members made it through the detailed inspection process. It was great to see the car driving well and in the end, Penn Electric came in 1st place, winning in 7/8 categories.
2015-2016
The next year started with a quick push to get a data acquisition system up and running to record data from many strain gauges that would be placed on various suspension linkages and also GPS and IMU data. I started creating a new, more flexible version of the grapher from the year before, but had to stop to focus on the firmware. In the process, I wrote code to read sensor data and designed a binary file format to store it efficiently. However, the PCBs kept getting delayed and we started running out of time to collect data before the design for our new car REV2 had to be completed. When they finally arrived, I had a day to get them working before we went testing and even then we didn't have all the sensors integrated we wanted. I did what testing and debugging I could in the little time we had and everything seemed to work, but when we tried to look at the logs the next day, we realized that a bug had corrupted the data. This was quite disappointing, but I did learn a lot from the project, both in terms of project management and how not to design a file format. A year later I made a new binary format, using the lessons I'd learned to prevent similar issues.
We did go testing again, but the lack of many strain gauges and GPS signal issues meant the data still wasn't that useful. I wasn't involved in the GPS circuit design for this system, but a couple years later when I needed to integrate a similar GPS chip my senior design project (SeaSearcher) I was extra careful to ensure in the design and layout we wouldn't have similar issues.
I didn't work on much else that semester, because I was busy working on Robockey and this was also the time when I became the first computer science major on the team when I finally decided to add computer science as a second major. I did a few other smaller things, like testing different screens to figure out which would be most sunlight readable for the dashboard. I also later wrote code for the dashboard to show the driver information and handle button inputs, excluding the display code.
I also designed the PCB for the REV2 power distribution unit. It houses 4 18650 cells on the back to power the car before the high voltage system is turned on and switches to the DC-DC converter when it is on. In total, the board can output up to 20A on its 8 outputs.
Another project I spent a lot of time on was code generation. The electrical lead at the time, Parth, had been at a company where he'd seen code be generated automatically to send CAN messages and thought this could make it much easier to manage all the data being sent around the car. I planned out the design and remember having a conversation for something like 8 hours straight to make all the details made sense. The program I eventually finished generated C++ code to handle communications around the car from a few .xml files. It supports a variety of features, including persistent variables that are updated in an EEPROM, snapshots of all variables whenever an error occurs and an integrated system for viewing all of the data live on the desktop program using MODBUS. It was working, but we didn't want to risk porting all the code on the car too close before the competition, so it was only running on the dashboard.
When some old code for logging the wireless data had problems with the updates we'd made to the car, I wrote a new program to replace it. I was able to complete this in a day and a half thanks to all of the libraries I'd written for code generation.
Code generation also made it possible to quickly implement an in memory data logger on the dashboard to get data without worrying about dropping wireless packets. This allowed us to measure acceleration tests and conduct other tests more accurately. It only could store about 32KB of data, but we didn't have time to implement an SD card logging system and it was enough to for the first time measure our 0-60 time of 2.7 seconds.
As we started testing the car, drivers found that the linear mapping between the pedal angle and the torque request wasn't ideal for all the events. For example, skidpad is done at low speeds, where it is important to have more fine control and high torque isn't necessary. I was tasked with creating customized pedal maps for each event as a function of pedal angle and racecar speed. Below is the endurance mapping, which allows the driver to take advantage of regenerative braking to recover energy at higher speeds. Regen increases at higher speeds due to the increased wing downforce and has to go to 0 below 5mph to be rules compliant.
An interesting phenomenon we noticed at low speeds was that the racecar would start speeding up and slowing down. Initially it seemed to be a some type of software issue, but eventually we concluded it was actually a result of the driver sliding back and forth into and off of the pedal as the car accelerated. It was partially resolved with pedal map changes, but it wasn't a big problem anyway, because the racecar isn't really designed for moving at slow speeds like that.
Only a few days before the competition, our motor controller blew up, and a team member flew to Canada to borrow another team's. We only got it working the night before the first day of competition and I was tired from programming during the drive, so I slept through most of it. Nevertheless, we came in 2nd place in the FSAE Lincoln 2016 competition, winning in several categories. We won acceleration with a record time (in the USA) of under 3.8s for 75m. The powertrain also failed half way through the endurance race, costing us a lot of points. The power distribution board and graphing software I had worked on were both mentioned by the judges in the closing ceremony as examples of great work the team had done.
2016-2017
This was the first year Penn Electric Racing really had a software team. I created a software challenge for recruiting that involved analyzing some of the team's old data files and interviewed people who wanted to join the team. Throughout the year I ran meetings to teach the new members everything they could want to know about the car. I also made sure we were staying on schedule, while keeping up with all the things I had to work on. It was also a challenging year, because we decided to design a four wheel drive racecar for the first time and to build custom motor controllers. Half way through the year, it became clear the racecar wouldn't be done in time, so we scrambled to make another much more similar to REV2 and push back four wheel drive a year. At the same time for unrelated reasons our electrical lead left, leaving Johnathan and I to lead the electrical (hardware + software) team together. Nevertheless, we got REV3 done in time and successfully competed at FSAE Lincoln as in previous years, in addition to attending Formula North for the first time.
For a project in one of my robotics classes (MEAM520), I worked with a couple friends on the team to make a robotic arm draw our team logo using a combination of geometry and inverse kinematics. The arm had an LED on the end that drew the picture and by turning off the lights and using a long exposure, we got the image below. It would have looked a lot better if it weren't for the fact that it was an old robot with bad joints.
The LPC1768 microcontrollers we had been using for the past few years had worked well for the team, but were getting a bit old and there were several newer and faster options with new features available. Mostly over the summer, I worked on designing a development board for a possible replacement called a SAME70. It had things like hardware support for CAN FD, which would allow us to send 8x as much data around the car, and a variety of other features. I managed to fit everything into the same form factor as the mbed boards that were available for the old microcontrollers and that meant they could easily fit into a breadboard for testing. The design included lots of extra hardware, such as CAN transceivers, a USB port and a hub to connect it to both a serial port and USB on the microcontroller. It also included power supplies, an RTC cell, an SD card slot and a few other features. It was a big challenge to route, because there were so many components and it was only a 4 layer board. Unfortunately the board was never manufactured, because we decided to use STM32F7s instead, because they seemed to have better libraries available.
Early in the year, I designed and implemented a new binary file format that could store data more efficiently and be parsed more quickly than our old formats. This new .perdat format incorporated all the lessons I had learned the past year when our log files got corrupted and was based around having a couple simple low level functions to do all the reading and writing so that even if something went wrong, the rest of the file could still be parsed. Our onboard logging system and computer software were both updated to support the new format and it has resulted in files that are roughly a third the size and can be parsed over ten times as fast. It can even achieve a better compression ratio when placed in a zip archive. The format also supports adding different types of metadata to allow us to add more features in the future. It has been working well since it was introduced. In addition to designing the format, I worked on structuring the code in a convenient way to manipulate the streams of data and that code has worked well for a variety of purposes over the years. A couple years later, we are now thinking about making minor changes to the format to simplify the code, but it will still largely remain the same.
Before I could start working on code for the new microcontrollers, I needed to get a new build system working, because we didn't want to keep using our old system, which was the mbed online compiler. This was because it lacked good code completion and couldn't use a debugger because it was just a website. After some research, it seemed like VisualGDB was the best option, because it worked well and we were already using Visual Studio for the computer software.
Once the IDE was setup, it quickly became clear that while the STM32F7s may have had had better libraries than the SAME70s, they still weren't great, especially compared to the mbed libraries we'd started to take for granted. One line of code to initialize a serial port turned into a mess of closer to 30 or 50 lines of code working with the STM32F7 Hardware Abstraction Layer (HAL). Therefore, we decided to build our own peripheral libraries to manage all the complexity of these libraries and in some cases, such as timers, bypass them entirely to get more flexibility and better performance. The hardest part was the learning curve getting started, especially because a single wrong or missing line would make nothing work. However, once I figured out how to configure a DMA serial port things got easier and I implemented around ten different peripheral libraries, including GPIOs, UART, SPI, I2C, ADC, CAN, timers and others. I also created a timer based scheduling API for short duration tasks that didn't make sense to be run through FreeRTOS. By the time I was finished, the libraries were even better than the old mbed libraries, because of our threading guarantees for some operations and better performance thanks to the use of DMA wherever possible.
Since we had decided to postpone four wheel drive by a year and to build REV3 to be very similar to REV2, we made the decision to reuse the electronics. That meant the code would only need minimal changes for some new rules year. However, shortly thereafter I learned the hard way the downsides of not having full control over your build system. Some internal mbed change over the past year broke our code so badly it couldn't even make it through static initialization. I spent a while trying to diagnose the problem, but it kept moving around and it was a challenge to find the issue without a debugger. After a day making little progress, I decided the best solution was to upgrade the code to use the new code generation system, which would eliminate roughly half the existing code, including the parts that seemed to be causing the issues. I had started updating code generation for the new STM32F7 microcontrollers once we'd decided to switch, but had thankfully decided to maintain backwards compatibility to the LPC1768 microcontrollers.
I considered just updating the battery management system, because the other boards hadn't been flashed since the previous year, but decided against it. This was a great decision because at competition we had to make a couple changes that would have been impossible with the build issues. It took several days and very late nights to port everything, especially with a few little edge cases that weren't really designed with code generation in mind. Nevertheless, I was really glad to have done it, even though it cost a day or two of testing, because this finally eliminated a lot of legacy code and also made a huge difference the next year. It also made adding variables around the car easier (just a single line of XML!) and prevented future issues manually serializing data. In addition, it made it possible for us to start viewing the data on each board from a computer connected to any other board.
Once all that refactoring was done, I had a few days before competition to continue improving our desktop software, called the Car Data Processor. Up to this point, we were still streaming data over UDP to the old CAN Message Grapher from my freshman year to graph data, so I decided to integrate live graphing directly into the CDP. In addition, I continued improving the register view to make it easier to view the increasingly large number of variables sent around the car. As part of this, I removed all the MODBUS code, because it was too inefficient, and replaced it with a custom binary serial protocol similar to the custom file format. Since then, we've continued using the Car Data Processor as our primary method of debugging any issues in communications between the boards and for monitoring the status of everything.
This year we competed at Formula North in Barrie, Ontario for the first time, in addition to FSAE Lincoln. It was only a nine hour drive, which made the journey much more enjoyable than the drive to Nebraska. We were a bit ahead of the trailer with the racecar, so we even got a chance to stop at Niagara Falls.
Unlike previous years, I was one of the two people leading our team through electrical tech inspection. The judges went through their eight page inspection checklist, asking many detailed questions about our car. We came prepared with hundreds of pages of datasheets printed out, pictures of everything that would be hard to take apart and all the design files we might need. Even with that preparation inspection wasn't easy.
Having two days to get through inspection may seem like a lot, but it leaves very little margin for error. After electrical inspection comes mechanical inspection, which is followed by the rain test, tilt test, and others. Some combustion cars who get to skip electrical inspection and rain test still struggle to finish everything in time Several years we've had late nights trying to fix things and this year was no exception. A power distribution board problem had to be quickly diagnosed and fixed. We finished electrical tech on the second morning and managed to finish everything else that day.
REV3 performed as well as we could have hoped and we came in first place at Formula North. A couple weeks later we drove to Lincoln, where we again had to go through the same inspection process again, except it was even more thorough. There weren't any major issues though and this year we also finished the endurance race we hadn't made it all the way through the year before. We even came in first place at Lincoln.
2017-2018
When my senior year started, I knew senior design would be a significant time commitment, but I planned to do an easier project so I would have plenty of time to work on the racecar. Then SeaSearcher happened. I really enjoyed the project and I don't regret any of it, but it was challenging staying on top of Penn Electric Racing at the same time, especially with all the other classes I was taking. After pushing back building a four wheel drive racecar to REV4, it seemed like we'd be done early and have plenty of time for testing. Unfortunately, that didn't happen for a variety of reasons.
Before the start of the year, I decided to create a physics simulation of the car for a variety of reasons, mainly to see how we could use torque vectoring with four wheel drive to get faster lap times. I got a bit sidetracked trying to get OpenGL to work nicely in C# and ended up creating a nuget package with a control I created. The goal was to make something similar to the OpenTK control that already existed, but purely in WPF without Windows Forms as a dependency. I succeeded to an extent and published my code on GitHub. However, just like the Windows Forms solution, it was still slow and a bit hacky, because WPF is based around DirectX and this meant that every frame had to be rendered to a buffer in main memory and then copied back to the GPU. That's one of the reasons we've started using gtkmm in C++ more recently. I never got that far with the simulation before I got busy with everything else as the semester got going. However, some of my ideas and lessons I learned were quite useful in developing the much more extensive autonomous racecar simulation the following year.
I was responsible for doing final reviews of all the circuit boards for REV4 and eventually I had to do the final fixes myself, so the boards wouldn't be too far behind schedule. This order included everything but the motor controllers and there were still some mistakes when we got them back, but considering we had switched microcontrollers and debuggers among other things, they worked surprisingly well.
The year before, I had started working on the firmware for our custom motor controllers, but didn't get that far. It became my top priority this year, because we needed to validate the motor controllers worked before making revisions and ordering new PCBs. I didn't know a ton about field oriented control before the project, but I quickly learned a lot after doing research and watching Dave Wilson's Teaching Old Motors New Tricks series a few times. Eventually I finished implementing the controller and thoroughly tested the boards, fixing several hardware issues along the way. I also gradually optimized the main control loop to run roughly 5-6x as fast as the original implementation, because it needs to run at over 32kHz. Note the motor in the video is one of the old motors we used to use, because the custom motors hadn't arrived yet. The initial testing was also done off a 30V, 150W power supply, so it couldn't go very fast.
REV4 deadlines started slipping early in the year, but things didn't get really bad until February when we heard from the judges that our battery pack fusing solution was insufficient and we needed to find a way to add 690 fuses to our pack to individually fuse the cells. It took months to solve the issue using wire bonding, but by that point it was only a week or so before Formula North. The reset of the car was barely done by then as well. As I watched the other subteams fall behind, I decided to focus more on senior design and that caused me to fall behind with updating the code for the new microcontrollers. I did test the boards to some extent, but didn't do as much as I should have. Thankfully I had upgraded the old code to use code generation the year before, which meant large portions of the code could remain unchanged. In addition, new members had made decent progress on several device libraries. Still, end of year projects meant I didn't have time to start until finals were practically over and refactoring all the firmware took nearly two weeks, because there were thousands of lines of code to update and minor issues here and there to fix. I also had to update code generation to support having four versions of the same board, because we had four custom motor controllers. At that point the car was pretty much ready, but the accumulator had problems. In addition, there were several software issues to debug, as expected when making such drastic code changes. I spent two days working with Dylan to debug a series of issues communicating with the LTC6804 chips that monitored the battery pack, before we finally got them working properly. It was time to leave for Formula North, but the car still wasn't quite ready. This time we didn't stop at Niagara Falls.
After we arrived, we started doing some final debugging before inspection. As we started going through the inspection checklist, it soon became clear we needed to stop worrying about tech inspection and start worrying about whether the car would be ready to drive at all. We hadn't even turned on the battery pack yet and initially it appeared to be a hardware issue, but it turned out to be a software bug. We gradually made progress and eventually we were ready for the car to spin the wheels for the first time. However, the CAN bus kept getting flooded by messages from random boards and eventually we figured out this was because std::atomic doesn't work on embedded devices, even though they have the necessary instructions to implement this functionality. Reverting a commit from months before where I had "upgraded" critical sections to atomics resolved the issue. We finally got the car to turn on and the PCM to enter ready to drive mode. However, the motor controllers kept faulting whenever we pushed down the pedals. We ran out of time to resolve the issue, leaving everyone disappointed after staying up to 4am in a freezing parking lot, when we finally had to call it a night. The next day I spent another couple hours on it before we had to leave.
Still, we had another competition at Lincoln and two weeks to get the car working, although most people, including myself, had to leave for summer internships. I did fly back early to get an extra 4 days to work on the racecar though. That time ended up mostly being spent on the accumulator, because it kept having mechanical problems. I had made some improvements to the motor controller while I was away, so I remained optimistic they would work at Lincoln. We also finally obtained a 300V power supply, which I used to start testing while others worked on the battery pack.
However, we were still a long way from being ready for Lincoln and were running out of time. The car was driven by a professional driver, while we took flights, which also meant we had a couple days waiting before we could do any further work on the car. When we arrived at competition, we started going through electrical tech inspection, but again things went slowly. At one point I even wasted an hour by forgetting to plug in the high voltage disconnect for a test, because I was tired and thinking about so many other things. Still, this time we passed electrical tech and finally had a car with four wheels controlled by the pedals. The mechanical team quickly got the car through mechanical tech and then it was time for the rain test. We had spent a significant amount of time waterproofing the car and adding extra tape here and here, but someone didn't connect a high voltage connector properly, causing a short to ground when it got wet and we failed the test. After a few hours of finding the problem, drying REV4 and rewaterproofing, we were ready to test again and that time we succeeded.
We didn't have any issues at the tilt test, but the brake test proved to be a challenge. The motor controllers still weren't working anywhere near as well as they should have and we'd run out of time to debug them. We were also running on two wheel drive, because disabling the motor controller allowed us to get the car working faster for rain test. The first time the car had even driven with tires on all four wheels was the first attempt at brake test, but it was too slow for us to lock all four wheels. In comparison, last year they had to tell us to slow down because we overshot the stopping area. I made a few tweaks the firmware, such as enabling the third motor controller, even though the racecar would no longer have symmetric tire forces. At 4:25pm on the second to last day of competition, 5 minutes before brake test would close for good, we passed.
That left us about three minutes to sprint to other autocross event on the other side of the competition area. We very nearly made it, but were only in time to see the last car complete the event. At that point, only endurance was left and we started preparing. We got the fourth motor controller working and started the race well, albeit a bit slowly, but faster than during brake test. However, one lap in a latching hardware fault circuit tripped, because it hadn't been calibrated properly and that was the end of our race. We placed worse than any year other since REV0, because we hadn't completed any of the events. Still, we had some good conversations about what went wrong and made plans to make sure we don't let the same thing happen for REV5.
2018-2019
Over time, as the team has grown and I have gained more experience, my role on the team has evolved. I now spend more of my time helping with decision making and reviewing other people's work, not to mention answering a ton of questions. This year more than ever made it my goal to grow the software team and improve our new member retention rate to help replace me when I leave, while also trying to hand off my responsibilities. A significant amount of that involved restructuring onboarding meetings to make them more interesting and relevant, but another important part was actually changing the build system. Our team has been mostly Windows based for the last several years, mainly because most people use SolidWorks and Altium, which only work on Windows. However, many computer science majors like using Linux and macOS and had trouble installing Windows. Damian figured out how to replace our entire VisualGDB build system and visual studio with cmake and Clion. This even allowed us to finally add a remote continuous integration system. At the same time, Jack has been developing a cross platform version of the Car Data Processor in C++, so everything on our team is cross platform and runs in C++. Daniel and Ben started working on a bootloader to allow us to flash the entire car at once over the CAN bus.
When I got back on campus, I was disappointed to see that B11 was about the messiest I had seen it and I decided to do something about it. We try to do a thorough cleaning each year, but we really did it this year. Everyone who was around then worked together to clean up the room. By the time we were finished, it was more organized than I've seen it any of my five years on the team.
Once I could actually find space in B11 to work in, I immediately started trying to figure out what we could do to get better motor controller performance. It took a while, because there were so many things that could be wrong, but everything seemed fine. Eventually I started looking at the angle sensor data in detail and found that our sensor was very nonlinear due to bad alignment between the sensor and the magnet on the motor. Note the actual error is 1/6 as bad as it looks, because the motor has 6 pole pairs and from the controller's perspective rotates 6 times per actual revolution.
We decided the best way to fix it was to switch to a new angle sensor and move the magnet somewhere we could ensure the alignment of everything was precise enough. However, that wasn't something we could do quickly, so I needed a solution in the mean time. I implemented a dynamic calibration system that estimates the angle error at each angle and uses that to correct the measurement. It's still not perfect, because the nonlinearity varies with speed, but it nevertheless made a huge difference.
With these changes and a few more controller tweaks, we went testing a couple more times and finally got an acceleration time comparable to what we had achieved with REV2/3.
It started getting too cold to test and we had to start taking apart REV4 to build REV5, so I switched my focus to an Autonomous Racecar Simulation that I worked on for a couple of classes. We're also starting to prepare to compete in the autonomous divisions of the Formula Student competitions in Europe and I hope my work will be useful to the team in the future.
I also started working on some minor improvements to our code, such as finally getting rid of some compiler warnings. As part of this, I implemented a new heap allocator, because the FreeRTOS one didn't work properly before the OS had been started and this was causing some issues with interrupts.
I've tried to stop doing much electrical work on the team, but I still reviewed the boards and ended up having to order them when deadlines started slipping.
Shortly thereafter, the second panel arrived with the new motor controllers. However, there was a supplier link mistake, because we'd finally stopped doing the bill of materials manually, and this meant the wrong microcontroller was placed. After a long day of drag soldering, they were all fixed.
I had also made a couple board improves, including external CAN FD transceivers so we could log even more data, and trace debug support. There was a battery management system issue that has caused the microcontroller to reset every few minutes or so. We couldn't figure out the problem, so I just implemented a quick workaround to recover from resets without stopping the car. However, in just a couple hours with the new trace debugger, Tony and I were able to find the problem. It turned out to be an issue with the timing library I had implemented a couple years before. When a thread used a delay function that used this API internally, there was a small chance of an interrupt delaying the thread until after the time it wanted to wait had elapsed. The library handled time wrap arounds properly everywhere, except the one place where it was added to the delay queue. It ended up being a quick 1 line fix.
As we finish building and debugging REV5, which is currently on track to be done in the next few weeks, the team is again starting to look ahead at making our future racecars autonomous. If all goes well, we will hopefully starting collecting data this summer.