LLM Powered Migration of UI Component Libraries

Hello, everyone. Thank you for joining my talk today. In today's session, I will be talking about how at Zalando we used LLMs to power the migration of component libraries in our application from one component library to another. Before we begin, a quick introduction. I am Nawal. I'm a principal engineer at Zalando. Zalando, if you're not familiar, is one of the leading fashion platforms in Europe and it partners closely with thousands of brands worldwide.

My team is responsible for building the platforms, toolings, and component library that is used by other engineering teams within Zalando to build their own external B2B applications and these external applications are then used by the brands or our partners to interact with Zalando and manage their business on Zalando. As a part of operating and maintaining this core platform, we encounter different kinds of migrations in our day-to-day work. These migrations can range from simple version upgrades, for example, updating from React 18 to 19.

However, they can be more complex migrations like migrating from one component library to another or migrating from one UI framework to another or updating the branding theming of our applications. One core thing about all these migrations that we carry out is that we not only have to carry the migration for our own core platforms and toolings, but when we plan the migration, we have to plan for each and every application that is running on our platform. So that means that for these migrations we have to think of how we can automate at scale and carry out the migration more efficiently.

Now talking about a migration problem that we encountered recently. In Zalando for our B2B external applications, there were two different component libraries that were being used. These component libraries had totally different structures, they had different interfaces, they had different components and they had different design philosophies. Maintaining two different component libraries was a bit expensive because we had to duplicate our effort and it also reduced the consistency of the UIs for our partners. Partners would see different UIs based on which application they were on.

As such, we want to unify on single component library. We wanted to migrate all applications using the component library which you see on the top. Let's say the source library to a component library at the bottom, let's say target library. However, this migration from the source component library to target component library was a very big effort. If we talk about in developer days, it was expected to take more than a year to complete and as such we wanted to see how we can automate this migration and make it faster.

First, we tried with codemods. Codemods are like the traditional approach to handle these kind of migrations. Codemods work by directly modifying the abstract syntax tree or AST for short. A simple AST for an import statement you can see on the right-hand side. As you can see, if you want to change the name of the package from old UI to let's say new UI, all you have to do is you have to find out where in the AST is the value of the old UI and then convert it to the new value and then convert it back to the code. So the codemods are actually very efficient and making the transformations and they are very fast and deterministic and they are a pretty good tool for handling many different migrations.

However, for our use case, what we found out that the codemods were not a good fit. There were a few reasons for this. First reason was that there was a lot of variations in our code base and if we had to create codemods to cater each and every use cases, codemods quickly became very cumbersome and challenging. For example, if I take the typography element which is let's say the most simple element that we have and even in this element there were many different variations and teams were using different practices. Some teams used the elements straight away. However, some teams assigned the typography to a shorter value and then used it. Some teams also used the heading object and assigned it to a variable and then used it. Some teams did not use the content as text component but they just used it as a property. So as you can see, there were a lot of different variations and getting to each and every variation quickly became very challenging.

Another limitation what we noticed for codemods was that we had to do a lot of complex migrations. If you look at the image on the right-hand side, you can see that when we have to migrate the select component from the source to target library, we don't only have to migrate the component but also the data itself in that component. For example, here the select options is a variable which is generated by a business logic of iterating over the options and that means that if we have to apply codemods, we have to be careful that the business logic stays intact. Also, there are other variations that are possible here. For example, here is a variable but it can be a name function, an arrow function and so on. So as you can see, the challenge here was that there were so many different variations and complexities that codemod were a bit impractical and what we wanted was some tool that could understand on its own the relation between different parts of the code and the different logic that is happening in the code and then we combine it with our instructions and the tool can do the migration.

So a good fit for that was LLMs. And LLMs or large language model was something that we were using already for our coding purposes through the co-pilot UI or similar tools. However, we had never used LLMs or large language models at this scale and we had a lot of open questions around this. Can LLM really handle this? I mean we had the idea that theoretically it could work but we were not sure if it can handle this kind of migrations. Additionally, the libraries or component libraries that we had were in-house libraries and LLMs were not trained on it. So another question was how we can use it. Finally, what kind of accuracy or consistency we can expect from this LLM. So instead of diving straight into implementation, what we decided was let's have a three-day exploration. We will sit down as a team and try to answer these open questions one by one and assess whether LLMs can be used for our purpose or not.

While we had carried out a lot of activities during our experimentation phase, I would like to focus for this presentation on a series of iterative experiments that we carried out to improve the accuracy of the output and ensure that LLMs can transform our components correctly. For the first iteration, what we did was we just passed the component source files to the LLM. Then we gave LLM a test file containing different usages of the components with varying complexity and asked it to migrate it and we wanted to see how well it could do. The results were not very good. LLM could not transform the complex usages. It could transform the simple usages but it could not transform the complex usages. The issue was, after researching, that we asked LLM to do too many things in one go or in one pass. For example, read the source file, extract the interface, create the transformer instructions on its own, then do the migration and taking care of too many steps reduced the accuracy and caused the LLM to fail.

Then we tried a second iteration where we tried to simplify the amount of data and the content which we sent to LLM by not giving the source file but giving the detailed component interface. This interface contained all the information LLM would need to know about the component like the properties, the subcomponents, the default values, and so on and then asked it to do the same thing. However, here also LLM did not do a good job. It could transform simple files or simple usages but it could not transform more complex usages. One reason was that LLM was missing a lot of context here. To give one example, in the source library the button component had three different sizes, small, medium, large, whereas in the target library it only had two sizes, small and large. So, if LLM encounters a button with a medium size, what should it do? Where should it put it, small or large? These kinds of questions LLM was not able to answer and that is why there were issues.

Then we started the third iteration where we put the same prompt as before but we also gave a step-by-step migration instructions which told LLM how to handle this case and also cleared any ambiguity that LLM had. The results were definitely better this time. However, there was still some inconsistency. Some of the more complex business logic was not migrated. Some of the things like cleaning up the old imports or unused variables were also not correctly done. At this stage, we started researching like what can we do to improve the output. For example, we looked around and looked at different research papers and so on. What we figured out is that if we provide patterns that could actually increase the accuracy of the LLM and that's exactly what we did. Along with the previous context, we also passed a set of transformation examples which were just before and after examples and we passed that. With this thing, the accuracy of LLM was pretty good. It had 90% accuracy and we were quite satisfied with it and then decided to proceed further. A small note on accuracy, like we used a metric to calculate how well LLM was behaving and this accuracy metric was just number of lines that were correctly migrated by LLM divided by the total lines of code migrated. By correctly, I mean like if there was import not migrated correctly or some unused variables or some bugs, those were excluded and then we calculated this metrics. Now, once the prompt structure or the was defined, this is how the final prompt looked like. It had four sections.

The first three sections like I talked before is just interface, mapping instructions and examples that were generated and the final section was the file which was needed, which was to be transformed. One thing to note here is that we kept the interface, instructions and examples at the top because these did not differ from file to file. For one component, they were always the same and this way, we could this way, this could be cached and we could save some cost on the LLM side due to prompt caching.

Once the prompt was there, we decided to build a small tool, which was a CLI tool. This tool, what it did was at the first step, it scanned through the whole repository to find out all the files that contained the components that we wanted to migrate. Then, it built the prompt with the components like I showed in the previous slide and the file and then submitted it to the LLM for transformation. LLM will take the file, transform it and send it as a response, which was then extracted by the tool and it applied it to the existing file and committed it. And then, a review was created for an engineer to verify, test it and then merge and deploy.

Now, talking a little bit about the technical details, we used a lot of different tools. These tools were open source, but they actually helped us reduce a lot of effort and also a lot of time. We used a tool called continue.dev. Continue.dev made it very easy to create the prompts. All we had to do was give the source files to the tool and then ask it to generate the instructions, interface and examples and it was easily able to do it. Then, we also used LLM SDK by Dataset. This had a very nice conversation API that allowed us to interact with LLM and also continue responses if it got cut off and so on.

Finally, we also used a tool, AIDAR, which was also AI-based. This tool helped us in automating the git commits and it also helped in creating a nice commit message and description so that the user who is going to review the pull request, they know what has been done and it makes the request reviews easier for them. For the tool, we also had a regression testing framework in place. What we had was a golden set of migration examples before and after and whenever we made changes to the prompts, we tried to run the tool again on the before examples and then compare it with the after examples. We compared it manually or either through LLM and this was used to see if there was any regression or decrease in accuracy of the tool. It was also very helpful if we wanted to test out a new model and see how that model was behaving.

Another thing we had to do to maintain the accuracy of the LLM output was to manage the context size of the prompts that we sent to the tool. First, we wanted to try to run the tool with as many components as possible. However, what we noticed is that if the context size becomes very large, the accuracy of the output started degrading quite a bit. So what we, after a bit of trial and error, figured out is that around 40,000 token size was a good size for the prompt and that's what we used for our tool. So we migrated a small set of components, ensuring that the token size was under 40,000 token.

Now, let's talk a little bit about what was the outcome. We were able to migrate 11 plus complex B2B applications. These applications were quite complex applications. Each of these applications migration was expected to take between one to four months and LLM assisted in this migration by performing the most repetitive and the common transformation or migration tasks, setting up our design product and engineering team to focus on more challenging task, polishing the UI, refining bugs, testing, and then deploying and merging. LLM also carried out the transformations or migration with a very high accuracy of 90 percent and above. And many of these, and there were no specific issues here. It was few bugs here and there, but most of the things were just unused input. So this ensured that our engineers did not need to spend a lot of time fixing issues caused by LLMs. The cost per repository for us was 40 USD. If you look at how much effort the LLM saved us during this migration, this was a very good return on investment. On the other hand, we also faced one incident here. This incident was due to the fact that when LLM migrated the data expected by a component, the LLM made a mistake in that migration. However, this issue was very hard to catch and it also got missed during the review, resulting in the code being deployed to production and the users saw a broken page. Two learnings from this incident for us were, first, LLMs can generate a bit plausible code, which looks plausible, but it can be incorrect. And that's why it's very important that whenever you have something like this, some code generated by LLM, you should review it very carefully. Another issue was that when LLMs migrate the code, it can create a knowledge gap because the users have not migrated it themselves. So they may lack a little bit of familiarity with the library. So, and this actually may cause an incident or it may cause debugging difficult and so on.

So, and this actually may cause an incident or it may cause debugging difficult and so on. And what we recommend here is that a team should basically be as close to the code as possible, try to onboard and try to understand what is happening so these kinds of issues don't happen. Giving an example of how the migration would look like. As you can see, this is an example of a PR, which was basically made by the LLM. You can see, it's a very big PR. So if you can imagine the amount of effort it saved us when by the automated migration carried out by the LLM.

It also did a very fine job of migrating the different components. One thing to interesting is that if you look at the select component, the select component had a bit of logic inside it. For example, it had a map, which actually iterated over select options. And there was also a text component inside the select option. The LLM was able to migrate all of this successfully. It extracted the options list and converted into an object. It also extracted the function and put it in the label field. So as you can see LLM could very successfully carry out multi-step complex transformations without any extra or additional information.

Then there are a few more examples I would like to share where LLMs did a quite good job. This was a notification popup and in the source library, the notification popup had a timeout of 4.5 seconds. But in the target library, there was no notification popup timeout. It had to be explicitly specified. And while we didn't instructed LLM to do it, it automatically took the default value of the source and added it here. So it can be seen like how smart LLMs are in some cases. Also, it fixed an issue in one of the function names. This issue was, I don't know, for very long, but it still was able to fix these kind of issues. Talking about the challenges, hallucination and ignoring instructions are some of the most common concerns when we use LLM.

And we also saw the same issue. One example is this exclusive field, which was not there in our instructions, for example, but LLM still added it. Another issue is that the output is not always deterministic. Something which was working today may start or stop working tomorrow. For example, if you look at the image on the bottom, you can see that it changed the name of the function from a correct one to an incorrect one. Something like this does not happen every time, but it may happen, let's say, once every 10, 15 runs. We had the temperature set to zero, but it still happened. And these are the things we ask our engineers to keep an eye on during the reviews.

Another issue was that we had to manage the prompts very carefully. If the input was very big, then the accuracy of the prompt started decreasing and LLMs could not transform it very correctly. And hence, we had to be careful about it. Another issue was that the responses were quite slow, especially when compared to code mods. Each response from LLM took around 30 to 200 seconds per file. And while this was not an issue during the migration, since the migration tool could run in the background, it however slowed down our iterative and experimental phases because of this time taken. There were also some project related constraints due to which we could not use the tool to its full capacity. For example, there were time constraints due to which we could not migrate the test cases. There were some differences in the components where each of them had needed to be handled on a case by case basis. And there were a lot of variations in the code basis, which made the tool usage more challenging.

Now, talking about lessons learned, LLMs are quite effective. Through this project, we could see that LLMs were quite effective for our migration and now we have been using LLMs in all our future migrations also. The iterative and experimental approach that we applied during this project proved to be very useful. And this is the template we use whenever we have some open questions on clarity about usage of LLMs in our project. LLMs can migrate, but they don't transfer the knowledge. And this can, like I showed before, can cause issues in debugging or can cause even incidents in the worst case. So, our recommendation to teams is that they stay on the top of the code, know what's happening and if there is something new, try to onboard themselves to that code base. As shown in the previous slide, there were a lot of unexpected issues caused by LLMs and teams need to ensure that there is some engineer who is reviewing each and every change and reviewing it carefully because LLMs still can get a lot of things wrong. We used a lot of tools and these tools were very helpful to reduce our time and also reduce our effort. These tools also are like a very fast moving landscape. We always, we get new tools all the time in this space. Hence, whenever we begin a project, we first evaluate what tools we can use for this thing.

Finally, PROMPT best practices and PROMPT engineering is not just a buzzword, it's something which is very useful to improve the accuracy of your output. We used it a lot. For example, we use patterns, we use it to break down a complex set of PROMPTS into smaller ones so that LLMs can handle this better and so on. So, this applying PROMPT engineering definitely pays off.

Finally, this project was something we did some time back and now the question is that does this what we talked about applicable today also? The landscape has evolved now. There are better tools, the models are much better now, the concepts are also newer like MCP, Cloud Code and all those things. However, what I would say is that the core approach still holds. The core discussion what we talked about is still accurate, but we now have better tools to play around with and use them.

We would still say that the code mods are very useful for predictable transformations as they are fast, deterministic and would have no surprises. We would still use LLMs and the library for simple migrations where we have to migrate a file with a lot of business logic like the component library migrations. However, one shortcoming of the current tool was that it did not work well with cases where there were changes across multiple files, which the tool first needed to understand and then make the changes. We encountered such a use case when we were migrating one in-house framework to another in our department.