In Memory of Travails

Hello, my name is Gabriel. I work at auction.com and I'm going to be talking about how we improved the performance of GraphQL resolvers for our GraphQL service. We had a problem with our subscription deployment, where we were experiencing a high number of restarts due to memory allocation failures. This prompted us to investigate and optimize the memory consumption of our resolvers. To assess the performance, we set up a local environment with Kafka, Graph, and a client that connected 4000 WebSockets to Graph. After running the test, we found that we were only able to process and distribute 16 messages to our clients. Yay. The memory consumption graph showed peaks and valleys as messages were delivered. Three distinct phases were observed: idle, Sockets connected with no messages, and messages being processed. We decided to optimize the context, which contains request-specific information and backend details. Since subscriptions primarily involve sending Kafka messages, we realized that the message itself often has all the necessary information. Therefore, we only create backends when a call is made to them. We optimized our backend creation process using the proxy object, which allows us to create backends lazily only when they are accessed. This resulted in less memory consumption without changing the code or the schema. The less memory consumption is evident in the second phase of the recording, where the plateau formed from having multiple contexts is significantly lower. Most of the savings were achieved by reducing temporary objects and using native iterators instead of Lodash calls for converting key names to Snakecase. All of a sudden, the performance increased by 18%, resulting in increased memory consumption. Upgrading to newer versions of GraphQL Redis subscriptions did not have a significant impact on memory usage. However, optimizing the conversion of key names to snake case by using memoization improved computational efficiency. Our performance significantly improved after implementing snake case. However, memory consumption remained high. To address the memory leak, we introduced auto-scaling and restarted the service every night. Additionally, we optimized the code generation process to improve memory consumption. We explored using heap snapshots in Google Dev tools to analyze and reduce memory consumption. By identifying unnecessary objects and removing them, we were able to free up memory and improve performance. We patched the location object to improve performance and reduce memory consumption. We also optimized data loaders to avoid N+1 queries and improve efficiency.