BACKEDGE CAPITAL
While working on my Master's in Computer Science in 2023, I interned at a small trading firm called BlackEdge Capital. This experience was my first exposure to developing low latency server-side code,
which is an experience I thoroughly enjoyed and plan to pursue. The team had me tackle two projects: developing an application which could take a custom network protocol specification as input
and would generate a Wireshark packet dissector as well as C struct definitions for each packet segment in the protocol; and a low latency application which would listen for incoming packets from
the exchange, tag an interesting subset of these packets, and relay the tagged packets to a client-side application.
Beyond just enhancing my Python, C, and C++ skills, the packet dissector generator application taught me the importance of rapid iteration. There were 3 languages that the Wireshark dissectors could be written in. I spent a sizeable amount of time researching each of these languages online to determine which would best suit our needs, however, even after researching I didn't feel confident, I could draw any firm conclusions. Consequently, I decided to just develop a MVP for a basic dissector using each of the 3 languages. The MVPs proved far more informative than researching each of the languages online, and it ultimately led us to select C as the language in which the dissectors ought to be written in. Now, I was faced with another daunting challenge of creating an application to generate Wireshark dissector code and C++ definition code for a wide variety of network protocols. The specification for this application was very vaguely defined, so instead of trying to formulate and account for all possible edge cases, I assessed what protocols the team was most interested and tailored my application to be able to work for this handful of protocols. As the team requested additional protocols, I slowly expanded and iterated the application. Developing in this way allowed me to make incremental progress in areas that generated the greatest value for the team. Once again, iterative design allowed me to take a loosely defined problem, and be able to solve it incrementally, to yield the most value in the shortest amount of time.
The second and more time-consuming of the projects was the development of a low latency packet tagger. Written in C++, this project proved an incredible learning experience for me and reaffirmed my desire to work in the development of low latency server-side code. The BlackEdge team showed me coding paradigms, data structures, and methods which allowed for many optimizations to be made at compile time which resulted in significant boosts in performance. They also taught me the importance of understanding the tradeoffs between developer time and program performance. If an application needs to be able to process a packet, tag it, and relay it to the client in the order of microseconds, this can be done leveraging optimized C++ code. However, if you need something that needs to be able to react in nanosecond latencies, you could spend countless developer hours trying to tune your C++ code to perfection, but at the end of the day, you will get a far more significant performance boost from creating a dedicating piece of hardware, like an FPGA, than you ever could with a C++ program. The time and money spent setting up this hardware solution will have significantly higher returns than time spent trying to hyper optimize a C++ program. For this reason, it is very important to distinguish the “hot paths”, the functions that if they become more performant directly lead to greater profits, from less important execution paths, so that developer time can be utilized as efficiently as possible. This project was helpful for me in keeping in mind my own coding efficiency and helped me to focus more on the balance between coding something to perfection or maximizing my output to yield the most profit as efficiently as possible.
I had a phenomenal time interning at BlackEdge Capital and am incredibly thankful for all the people there that allowed me to learn alot and have a ton of fun during the summer of 2023.
Beyond just enhancing my Python, C, and C++ skills, the packet dissector generator application taught me the importance of rapid iteration. There were 3 languages that the Wireshark dissectors could be written in. I spent a sizeable amount of time researching each of these languages online to determine which would best suit our needs, however, even after researching I didn't feel confident, I could draw any firm conclusions. Consequently, I decided to just develop a MVP for a basic dissector using each of the 3 languages. The MVPs proved far more informative than researching each of the languages online, and it ultimately led us to select C as the language in which the dissectors ought to be written in. Now, I was faced with another daunting challenge of creating an application to generate Wireshark dissector code and C++ definition code for a wide variety of network protocols. The specification for this application was very vaguely defined, so instead of trying to formulate and account for all possible edge cases, I assessed what protocols the team was most interested and tailored my application to be able to work for this handful of protocols. As the team requested additional protocols, I slowly expanded and iterated the application. Developing in this way allowed me to make incremental progress in areas that generated the greatest value for the team. Once again, iterative design allowed me to take a loosely defined problem, and be able to solve it incrementally, to yield the most value in the shortest amount of time.
The second and more time-consuming of the projects was the development of a low latency packet tagger. Written in C++, this project proved an incredible learning experience for me and reaffirmed my desire to work in the development of low latency server-side code. The BlackEdge team showed me coding paradigms, data structures, and methods which allowed for many optimizations to be made at compile time which resulted in significant boosts in performance. They also taught me the importance of understanding the tradeoffs between developer time and program performance. If an application needs to be able to process a packet, tag it, and relay it to the client in the order of microseconds, this can be done leveraging optimized C++ code. However, if you need something that needs to be able to react in nanosecond latencies, you could spend countless developer hours trying to tune your C++ code to perfection, but at the end of the day, you will get a far more significant performance boost from creating a dedicating piece of hardware, like an FPGA, than you ever could with a C++ program. The time and money spent setting up this hardware solution will have significantly higher returns than time spent trying to hyper optimize a C++ program. For this reason, it is very important to distinguish the “hot paths”, the functions that if they become more performant directly lead to greater profits, from less important execution paths, so that developer time can be utilized as efficiently as possible. This project was helpful for me in keeping in mind my own coding efficiency and helped me to focus more on the balance between coding something to perfection or maximizing my output to yield the most profit as efficiently as possible.
I had a phenomenal time interning at BlackEdge Capital and am incredibly thankful for all the people there that allowed me to learn alot and have a ton of fun during the summer of 2023.
FACTSET RESEARCH SYSTEMS INC.
In the summer of 2022, I worked as a Software Developer Intern for FactSet Research Systems Inc. FactSet aggregates, processes, and serves large amounts of financial data in near real-time,
so that traders and investment professionals have the data they need to make informed decisions. The internship was in-person in Chicago, and I made a habit of going into the office every day, to
maximize my exposure to the company and enhance my learning experience. Additionally, I made it my goal during this internship to gain experience in developing highly performant and concurrent
systems, since these are challenges that I find particularly interesting.
My intern project was to develop a web application using Typescript and Vue.js, which would make it far easier to configure and debug an internal data processing system. What made the system difficult to use was that it spanned multiple technologies, running on AWS EC2 instances, getting data and configurations from PostgresSQL databases, and storing files and logs in AWS S3. My web application would abstract away all these technologies and provide a single uniform interface where the system could be configured and debugged. The primary challenge of developing this application was devising a system for globally ordering updates to the database. My experience in the Distributed Systems class at Michigan helped me to overcome this challenge by simply versioning all updates to the database and ensuring strict consistency. The multiple instances of my web application communicated this versioning value by storing it in a central location in the PostgresSQL database. Additionally, to limit conflicts I added several useful features to the application such as auto refresh and a conflict resolution system.
I finished the web application a month early and wanted to continue striving towards my initial goal, so I relentlessly asked my manager to find me tasks that would allow me to work with concurrent and performant code. He delivered and tasked me with refactoring some multi-process Python code, as well as implementing and testing a cross-thread logging feature for a high-performance C++ program that could not afford milliseconds of latency increase. The Python refactor was challenging, since testing my code changes took around 20 hours, so I made sure to develop an extensive unit testing system. The unit tests allowed for most of the 20-hour tests I executed to succeed and were essential to my successfully completing the project. The C++ program I worked on was very different from other systems I worked with and required that I learn how to use Perforce for version control as well as deepen my experience in developing multi-threaded code.
Overall, this internship at FactSet was a great experience and allowed me to achieve my goal of gaining experience in developing highly performant and concurrent systems.
My intern project was to develop a web application using Typescript and Vue.js, which would make it far easier to configure and debug an internal data processing system. What made the system difficult to use was that it spanned multiple technologies, running on AWS EC2 instances, getting data and configurations from PostgresSQL databases, and storing files and logs in AWS S3. My web application would abstract away all these technologies and provide a single uniform interface where the system could be configured and debugged. The primary challenge of developing this application was devising a system for globally ordering updates to the database. My experience in the Distributed Systems class at Michigan helped me to overcome this challenge by simply versioning all updates to the database and ensuring strict consistency. The multiple instances of my web application communicated this versioning value by storing it in a central location in the PostgresSQL database. Additionally, to limit conflicts I added several useful features to the application such as auto refresh and a conflict resolution system.
I finished the web application a month early and wanted to continue striving towards my initial goal, so I relentlessly asked my manager to find me tasks that would allow me to work with concurrent and performant code. He delivered and tasked me with refactoring some multi-process Python code, as well as implementing and testing a cross-thread logging feature for a high-performance C++ program that could not afford milliseconds of latency increase. The Python refactor was challenging, since testing my code changes took around 20 hours, so I made sure to develop an extensive unit testing system. The unit tests allowed for most of the 20-hour tests I executed to succeed and were essential to my successfully completing the project. The C++ program I worked on was very different from other systems I worked with and required that I learn how to use Perforce for version control as well as deepen my experience in developing multi-threaded code.
Overall, this internship at FactSet was a great experience and allowed me to achieve my goal of gaining experience in developing highly performant and concurrent systems.
INVIA MEDICAL IMAGING SOLUTIONS
During the summer of 2021 I worked as a Software Developer Intern for Invia Medical Imaging Solutions. Invia’s main product is an application which reads in heart scans stored in large databases
and performs processing on the images to reconstruct a 3D representation of the heart. Once the images are processed the product then allows doctors to easily disect and
view slices of the heart, and will highlight key information about the slices that makes them easier to interpret.
Over the course of the summer I used C++/CLI, C#, SQLite, and the .NET Framework to solve the issue of high latency when copying a database across a network. There are particular types of scans, which generate a large amount of very small image files, which are difficult to transfer across a network since for each image transferred there is overhead when connecting to the destination computer, and incurring this overhead for each image transferred led to immense latency. I worked with the excellent Invia engineers to solve this problem, by creating a service, which could identify these particular images and package them into a zip archive. As a result of this, I had to add functionality to the product to process a database that has zip archives in it, and also process the images within the zip archive. By the end of the summer, the product was capable of reading in database files which were zipped, and transferring a database across a network became 10x faster.
This experience taught me a lot about what it is like to work on a professional software development team, developing windows services and applications, multi-threading, and several programming languages I was unfamiliar with. The project manager gave me a lot of freedom to employ my own problem-solving and creative abilities in solving the many challenges I encountered, which allowed me to take ownership and pride in what I did. This internship was an incredible learning experience for me and it was a lot of fun to work with the Invia staff and contribute a product feature that will actually be shipped.
Over the course of the summer I used C++/CLI, C#, SQLite, and the .NET Framework to solve the issue of high latency when copying a database across a network. There are particular types of scans, which generate a large amount of very small image files, which are difficult to transfer across a network since for each image transferred there is overhead when connecting to the destination computer, and incurring this overhead for each image transferred led to immense latency. I worked with the excellent Invia engineers to solve this problem, by creating a service, which could identify these particular images and package them into a zip archive. As a result of this, I had to add functionality to the product to process a database that has zip archives in it, and also process the images within the zip archive. By the end of the summer, the product was capable of reading in database files which were zipped, and transferring a database across a network became 10x faster.
This experience taught me a lot about what it is like to work on a professional software development team, developing windows services and applications, multi-threading, and several programming languages I was unfamiliar with. The project manager gave me a lot of freedom to employ my own problem-solving and creative abilities in solving the many challenges I encountered, which allowed me to take ownership and pride in what I did. This internship was an incredible learning experience for me and it was a lot of fun to work with the Invia staff and contribute a product feature that will actually be shipped.