Background

From May 2019 to January 2020, I worked at Netskope on the Data/Infra team, both in Santa Clara and in San Francisco. I learned a lot – both academically and professionally. I redesigned and partially rewrote a core part of their infrastructure, which eventually broke out into its own component that would be scaled up over time. This infrastructure was written in Go to replace their aging infrastructure in Python.

Internship

During the internship, I worked on the component called QueryService – a component that retrieves database data for UI components. In particular, I wrote out a crucial piece of infrastructure to support new databases (as we were migrating from the aging Mongo cluster we had) to support both Clickhouse and Google Cloud Platform (Bigquery). This piece of infrastructure was written in Go, which is a departure from the company’s main language of Python.

When I started over the summer, I wrote the entire redesign of the service and checked in with several teams to find their pain points in using the in-house API. With the feedback, I generated pages of new API documentation. While this final API did not make it past the planning stage, the 19-page document serves as a great guide for future improvement to their current service.

During the fall, the engineering team required better metrics, so we looked at Clickhouse and Bigquery as good aggregation tools. Using the code I wrote over the summer, we ported it over to be called as a separate service (from the current Python infrastructure), and we quickly scaled it up to support various existing queries (subgroupbys, aggregations, etc.). The main part I worked on during the Fall was dynamic database selection, where several preaggregated tables and collections would be available, and a request coming in would be matched to the table that would take the least computational time very quickly. This allowed us to not have to query the raw tables for all aggregations, which saved a tremendous amount of time.

Takeaways

Food is great. I’ve never really realized, but it is pretty privileged to work in the Silicon Valley with perks that employees take for granted. People complain about the lunch selection, but it’s a great departure from worrying about the next meal in a college setting.

Communication with teams halfway across the world is difficult. In the fall, my main team was located in India, so getting around a 8,000 mile barrier with a 12.5 hour time difference was pretty difficult. Calls were scheduled for any time between 8am and midnight. There were many times when I stepped in the office, pulled code from the repo, and found that something was broken. Since the team was asleep, I would have to work out the issue for a few hours before committing any useful work. Because of this, I came to appreciate working in an office and being able to call someone over at any time.

Overall, the experience was really great – I learned a lot about industry technologies – Kubernetes, Docker, etc. – and worked with many different kinds of people from different backgrounds. Working in San Francisco was a great experience as well – while it was a lot to handle between a full load of classes and a 12 hour work week, it was definitely rewarding.