Interested in Impira's Software Engineer internship? Learn more and apply today.
As a student who transferred to computer science (CS) from another major in my junior year, I had only one CS-related internship prior to my time at Impira. That internship abruptly moved to being remote due to COVID-19, and that change greatly impacted my experience. I had fewer and fewer opportunities to interact with the other members of the team and I was working on a project that I wasn’t necessarily passionate about. Coming into another (mostly) remote internship, I had concerns that this summer would be a repeat of that past internship experience.
However, I quickly realized that my Impira internship was going to be different. Right off the bat, my mentor, Gus, a software engineer at Impira, spent a ton of time helping me set up my dev environment. Throughout the first few weeks of the internship, I had the opportunity to talk to and meet everyone on my team one-on-one over Zoom. This made the internship experience feel a lot more personal, and I always felt like someone would be there to help if I had any kind of problem or had any questions.
At the beginning of the summer, I voiced my concerns about whether I’d have ample time to find a project I really wanted to pursue. Gus was happy to help, and during my onboarding in the first two weeks, I was given various small tasks to complete in different parts of the codebase to give me a better idea of what kind of work I might be interested in doing. When I was finished with those, Gus gave me a couple proposals for an intern project to focus on for the rest of the summer, according to the interests I expressed throughout the first two weeks. He gave me an additional few days to poke around in the code for each project and decide which one I was more interested in. Having so much time to explore my options was extremely valuable in allowing me to pick a project that genuinely fascinated me.
Project: A redesign of Turbine
I was tasked with a partial redesign of Turbine — our in-house task scheduling system that is responsible for asynchronous processing of assets. This is achieved by importing dynamically loaded libraries (DLLs) into the core Turbine server process running on one of our machines. The process has one thread per CPU core, and each one pulls tasks from an internal task queue and individually executes the task’s code.
However, since all threads in a process share the same address space, this approach is prone to stability issues if individual tasks accidentally access memory used by other tasks or the core Turbine code, or if tasks crash the entire process. The goal of this project was to design and implement an approach which isolates tasks within processes and handles crashes correctly.
Over the course of 12 weeks, I implemented two different approaches to this problem. Michael Andrews, Impira’s chief architect and Turbine’s main developer, proposed one approach, and I came up with a proposal for the second approach.
Michael’s approach (Exhibit A) involved replacing the individual threads with full-blown processes, such that there is one process per core, as well as a supervisor process that manages the task queue and handles crashes. These processes have a shared memory region that holds the task queue. This allows each process to individually pull tasks from the queue and execute them.
As a result, tasks are spread out over multiple address spaces, allowing for a moderate degree of process isolation. So, if a task crashes, only one child process crashes, and the supervisor process can clean up and restart processes if necessary. DLLs are loaded into each process as necessary, allowing a single process to have many DLLs cached so that future tasks using the same DLLs can be loaded quickly.
My approach to addressing the problem involved forking a worker process to run each task. This means that each task gets a dedicated process, allowing for maximum process isolation. Once a worker is finished running a task, it joins a worker pool and can be reused to run tasks that use the same DLL. This restriction on using the same DLL is an optimization to avoid loading DLLs unnecessarily, since doing so is a fairly expensive operation. If a worker hasn’t been used for a while, it’ll exit in order to save resources. If a task has been running for an abnormally long period of time, the supervisor will send a signal to the worker to pause that process to give CPU time to other tasks.
My internship came to an end before I was able to run full performance tests on both approaches, but preliminary testing showed that both implementations performed comparably when run with many tasks with the same DLL. Both approaches successfully solved the problem of fault handling on crashing tasks and improved process isolation. The second approach is also especially useful for implementing extensions, such as limiting the amount of memory used by each process to help with some memory leakage issues.
A look back: Thoughts and takeaways
Over the course of the internship, I was really able to apply the material I learned from my university courses to a degree I was never able to do at any previous internship. This is partially due to the patience and acceptance of my mentors who were open to the ideas I had for improving the system, and allowed me the freedom to alter the structure and scope of my internship to maximize my experience. Michael Andrews was especially supportive of my approach to the process isolation problem and gave ideas for improving it, and Gus was flexible enough to let me reallocate time during my internship to fully pursue the design and implementation of my solution.
My project was special because I was given the opportunity to work on a real system and make changes that would actually be deployed and utilized. I also had the chance to create my own design for a system and iterate on it — something that no other internship in the past had allowed me to do. I was given full autonomy and ownership to apply the approaches I wanted to try, to design components in a way I thought was best, and to create the tests I thought were necessary. Additionally, I was able to practice using the concepts I learned in courses in the past, like concurrent programming and process management, as well as learning to use C++ in ways I had never imagined possible.
I’d like to thank Michael Andrews for being receptive to my ideas, for always encouraging me to improve my designs and think outside the box, and for being an exceptional systems programming mentor who would always provide insight for solving even the toughest of issues.
I’d also like to thank Gus for facilitating a smooth integration into Impira’s team, and for giving me great advice about anything related to working at Impira. He always worked to make sure my internship was yielding the most possible value for me.
Ready to jump in? Apply for the Software Engineer Intern position today.