LLM Powered Migration of UI Component Libraries

Hello, everyone. Thank you for joining my talk today. In today's session, I will be talking on how at Zalando we used LLMs to power the migration of component libraries in our application from one component library to another. Before we begin, a quick introduction. I am Nawal. I am a principal engineer at Zalando. Zalando, if you're not familiar, is one of the leading fashion platforms in Europe, and it partners closely with thousands of brands worldwide.

My team is responsible for building the platforms, toolings, and component library that is used by other engineering teams within Zalando to build their own external B2B applications. As a part of operating and maintaining these core platforms, we encounter different kinds of migrations in our day-to-day work. One core thing about all these migrations that we carry out is that we not only have to carry the migration for our own core platforms and toolings, but when we plan the migration, we have to plan for each and every application that is running on our platform.

Now talking about the migration problem that we encountered recently, in Zalando for our B2B external applications, there were two different component libraries that were being used. These component libraries had totally different structure. They had different interfaces, they had different components, and they had different design philosophies. First we tried with code mods. Code mods are like the traditional approach to handle this kind of migration. Code mods work by directly modifying the abstract syntax tree or AST for short.

They had different interfaces, they had different components, and they had different design philosophies. Maintaining two different component libraries was a bit expensive because we had to duplicate our effort, and it also reduced the consistency of the UIs for our partners. Partners would see different UIs based on which application they were on. As such, we want to unify on a single component library. We wanted to migrate all applications using the component library you see on the top to a component library at the bottom, the target library.

However, this migration from the source component library to the target component library was a very big effort. If we talk about in developer days, it was expected to take more than a year to complete, and as such, we wanted to see how we can automate this migration and make it faster. First, we tried with code mods. Code mods are like the traditional approach to handle this kind of migration. Code mods work by directly modifying the abstract syntax tree or AST for short. A simple AST for an import statement you can see on the right-hand side.

As you can see, if you want to change the name of the package from the old UI to the new UI, all you have to do is find out where in the AST is the value of the old UI and then convert it to the new value and then convert it back to the code. However, for our use case, what we found out was that the code mods were not a good fit. There were many variations in our code base, making code mods cumbersome and challenging to create for each use case. The challenge was the complexity of migrations, especially when handling different variations and logic in the code. This led us to explore a tool that could understand the code relations and logic, which brought us to LLMs, or large language models, a suitable solution for our migration needs.

First reason was that there was a lot of variations in our code base and if we had to create code mods to cater each and every use case, code mods quickly became very cumbersome and challenging. For example, if I take the typography element, which is let's say the most simple element that we have, even in this element, there were many different variations and teams were using different practices. Some teams used the element straight away, while some assigned the typography to a shorter value and then used it. Additionally, some teams used the heading object and assigned it to a variable before using it. This diversity in practices made handling variations challenging.

Another limitation we faced with code mods was the complexity of migrations. For instance, when migrating the select component from the source to target library, it wasn't just about migrating the component itself but also ensuring the data within the component was correctly transferred. The select options, generated by business logic, required careful migration to preserve the logic. Moreover, the variability in code structures, such as variables, name functions, and arrow functions, added layers of complexity. Code mods struggled to handle these diverse scenarios effectively, leading us to seek a tool that could comprehend code relations and logic autonomously.

We turned to LLMs, large language models, as a potential solution for our migration challenges. While we had experience with LLMs through tools like Copilot UI, using them at this scale for migration was uncharted territory for us. Initial doubts surrounded the capability of LLMs to handle our specific migration needs, especially considering our in-house component libraries. Questions about how to utilize LLMs effectively and the expected accuracy and consistency further fueled our exploration. To test the viability of LLMs, we opted for a focused three-day exploration to address these concerns methodically and evaluate the suitability of LLMs for our migration project.

Additionally, the libraries or component libraries that we had were in-house libraries and LLMs were not trained on it. So another question was how we can use it? Finally, what kind of accuracy or consistency we can expect from this LLM? So instead of diving straight into implementation, what we decided was let's have a three-day exploration. We will sit down as a team and try to answer these open questions one by one and assess whether LLMs can be used for our purpose or not.

While we had carried out a lot of activities during our experimentation phase, I would like to focus for this presentation on a series of iterative experiments that we carried out to improve the accuracy of the output and ensure that LLMs can transform our components correctly. For the first iteration, what we did was we just passed the component source files to the LLM. Then we gave LLM a test file containing different usages of the components with varying complexity and asked it to migrate it. We wanted to see how well it could do. The results were not very good. LLM could not transform the complex usages. It could transform the simple usages, but it could not transform the complex usages. The issue was after researching that we asked LLM to do too many things in one go or in one pass. For example, read the source file, extract the interface, create the transformer instructions on its own, then do the migration. Taking care of too many steps reduced the accuracy and caused the LLM to fail.

Then we tried a second iteration where we tried to simplify the amount of data and the content which we sent to LLM by not giving the source file, but giving the detailed component interface. This interface contained all the information LLM would need to know about the component like the properties, the sub components, the default values and so on, and then asked it to do the same thing. However, here also LLM did not do a good job. It could transform simple files or simple usages, but it could not transform more complex usages. One reason was that LLM was missing a lot of context here. To give one example, in the source library, the button component had three different sizes, small, medium, large, whereas in the target library it only had two sizes, small and large. So if LLM encounters a button with medium size, what should it do? Where should it put it, small or large? These kind of questions LLM was not able to answer and that's why there were issues.

Then we started the third iteration where we put the same prompt as before, but we also gave a step-by-step migration instructions which told the LLM how to handle this case and also cleared any ambiguity that LLM had. The results were definitely better this time. However, there was still some inconsistency. Some of the more complex business logic was not migrated. Some of the things like cleaning up the old imports or unused variables was also not correctly done. At this stage, we started researching like what can we do to improve the output? For example, we looked around and looked at different research papers and so on and what we figured out is that if we provide patterns that could actually increase the accuracy of the LLM and that's exactly what we did.

Along with the previous context, we also passed a set of transformation examples which were just before and after examples and we passed that. And with this thing, the accuracy of LLM was pretty good. It had a 90% accuracy and we were quite satisfied with it and then decided to proceed further. A small note on accuracy like we used a matrix to calculate how well LLM was behaving and this accuracy matrix was just number of lines that were correctly migrated by LLM divided by the total lines of code migrated. By correctly, I mean like if there was import not migrated correctly or some unused variables or some bugs, those were excluded and then we calculated this matrix.

Now, once the prompt structure or the was defined, this is how the final prompt looked like. It had four sections. The first three sections like I talked before is just interface mapping instructions and examples that were generated and the final section was the file which was needed which was to be transformed. One thing to note here is that we kept the interface instructions and examples at the top because these did not differ from file to file. For one component, they were always the same and this way, we could this way, this could be cached and we could save some cost on the LLM side due to prompt caching. Once the prompt was there, we decided to build a small tool which was a CLI tool. This tool, what it did was at the first step, it scanned through the whole repository to find out all the files that contained the components that we wanted to migrate.

Then it built the prompt with the components like I showed in the previous slide and the file and then submitted it to the LLM for transformation. LLM will take the file, transform it and send it as a response which was then extracted by the tool and it applied it to the existing file and committed it and then a review was created for the engineer to verify, test it and then merge and deploy. Now talking a little bit about the technical details, we used a lot of different tools. These tools were open source but they actually helped us reduce a lot of effort and also a lot of time. We used a tool called continue.dev. Continue.dev made it very easy to create the prompts. All we had to do was give the source files to the tool and then ask it to generate the instructions interface and examples and it was easily able to do it. Then we also used LLM SDK by Dataset. This had a very nice conversation API that allowed us to interact with LLM and also continue responses if it got cut off and so on. Finally, we also used a tool AIDAR which was also AI-based. This tool helped us in automating the git commits and it also helped in creating a nice commit message and description so that the user who is going to review the pull request, they know what has been done and it makes the request reviews easier for them. For the tool, we also had a regression testing framework in place.

What we had was a golden set of migration examples before and after and whenever we made changes to the prompts, we tried to run the tool again on the before examples and then compare it with the after examples. We compared it through manually or either through LLM and we said when this was used to see if there was any regression or decrease in accuracy of the tool. It was also very helpful if we wanted to test out a new model and see how that model was behaving. So this regression setup was very useful for us.

Another thing we had to do to maintain the accuracy of the LLM output was to manage the context size of the prompts that we sent to the tool. First, we wanted to try to run the tool with as many components as possible. However, what we noticed is that if the context size becomes very large, the accuracy of the output started degrading quite a bit. So what we, after a bit of trial and error, figured out is that around 40,000 token size was a good size for the prompt and that's what we used for our tool. So we migrated a small set of components, ensuring that the token size was under 40,000 token.

Now, let's talk a little bit about what was the outcome. We were able to migrate 11 plus complex B2B applications. These applications were quite complex applications. Each of these applications migration was expected to take between one to four months and LLM assisted in this migration by performing the most repetitive and the common transformation or migration tasks, freeing up our design product and engineering team to focus on more challenging tasks, polishing the UI, refining bugs, testing, and then deploying and merging. LLM also carried out the transformations or migration with a very high accuracy of 90 percent and above. And many of these, and there were no specific issues here, it was few bugs here and there, but most of the things were just unused input. So this ensured that our engineers did not need to spend a lot of time fixing issues caused by LLMs. The cost per repository for us was 40 USD. If you look at how much effort the LLM saved us during this migration, this was a very good return on investment.

On the other hand, we also faced one incident here. This incident was due to the fact that when LLM migrated the data expected by a component, the LLM made a mistake in that migration. However, this issue was very hard to catch and it also got missed during the review, resulting in the code being deployed to production and the users saw a broken page. Two learnings from this incident for us were, first, LLMs can generate a bit plausible code, which looks plausible, but it can be incorrect. And that's why it's very important that whenever you have something like this, some code generated by LLMs, you should review it very carefully.

Another issue was that when LLMs migrate the code, it can create a knowledge gap because the users have not migrated it themselves, so they may lack a little bit of familiarity with the library. And this actually may cause incident or it may cause debugging difficulty, and so on. And what we recommend here is that a team should basically be as close to the code as possible, try to onboard and try to understand what is happening, so these kinds of issues don't happen.

Giving an example of how the migration would look like, as you can see, this is an example of a PR, which was basically made by the LLM. You can see it's a very big PR, so you can imagine the amount of effort it saved us by the automated migration carried out by the LLM. It also did a very fine job of migrating the different components. One thing interesting is that if you look at the select component, the select component had a bit of logic inside it. For example, it had a map which actually iterated over select options, and there was also a text component inside the select option. The LLM was able to migrate all of this successfully. It extracted the options list and converted it into an object. It also extracted the function and put it in the label field. So, as you can see, LLM could very successfully carry out multi-step complex transformations without any extra or additional information. Then, there are a few more examples I would like to share, where LLMs did a quite good job.

This was a notification pop-up, and in the source library, the notification pop-up had a timeout of 4.5 seconds, but in the target library, there was no notification pop-up timeout. It had to be explicitly specified. While we didn't instruct LLM to do it, it automatically took the default value of the source and added it here. So, it can be seen how smart LLMs are in some cases. Also, it fixed an issue in one of the function names. This issue was, I don't know for very long, but it still was able to fix these kinds of issues.

Talking about the challenges, hallucination and ignoring instructions are some of the most common concerns when we use LLM, and we also saw the same issue. One example is this exclusive field which was not there in our instructions for example, but LLM still added it. Another issue is that the output is not always deterministic. Something which was working today may start or stop working tomorrow. For example, if you look at the image on the bottom, you can see that it changed the name of the function from a correct one to an incorrect one. Something like this does not happen every time, but it may happen, let's say, one every 10-15 runs.

We had the temperature set to zero, but it still happened, and these are the things we ask our engineers to keep an eye on during the reviews. Another issue was that we had to manage the prompts very carefully. If the input was very big, then the accuracy of the prompts started decreasing and LLMs could not transform it very correctly, and hence, we had to be careful about it. Another issue was that the responses were quite slow, especially when compared to codemods. Each response from LLM took around 30 to 200 seconds per file, and while this was not an issue during the migration since the migration tool could run in the background, it however slowed down our iterative and experimental phases because of this time taking.

There were also some project-related constraints due to which we could not use the tool to its full capacity. For example, there were time constraints due to which we could not migrate the test cases. There were some differences in the components where each of them had needed to be handled on a case-by-case basis, and there were a lot of variations in the code basis which made the tool usage more challenging.

Now, talking about lessons learned, LLMs are quite effective. Through this project, we could see that LLMs were quite effective for our migration, and now, we have been using LLM in all our future migrations also. The iterative and experimental approach that we applied during this project proved to be very useful, and this is the template we use whenever we have some open questions on clarity about usage of LLMs in our project. LLMs can migrate, but they don't transfer the knowledge, and this can, like I showed before, can cause issues in debugging or can cause even incidents in the worst case.

Our recommendation to teams is that they stay on top of the code, know what's happening, and if there is something new, try to onboard themselves to that codebase. As shown in the previous slide, there were a lot of unexpected issues caused by LLMs, and teams need to ensure that there is some engineer who is reviewing each and every change and reviewing it carefully, because LLMs still can get a lot of things wrong. We used a lot of tools, and these tools were very helpful to reduce our time and also reduce our effort. These tools are also like a very fast-moving landscape.