Lessons from Building Enterprise Remote MCP Server?

I'm Hemant. I'm a senior machine learning manager at PayPal. I also happen to be a GD in web and payments domain. I'm a DuckDuckDuck community member. I've been lucky enough to be part of TC39, contribute a bit here to Node.js. And you can find me at h3man.com or at Gnument, and that's the QR code.

So if you were a chat GPT or any of those users back then, if you were and said, hey, what's the time or what's the date like? It would have responded you back saying that I don't have access to real-time data, or it would say, hey, I or probably it would have made up a time or date, right? But if I open up a terminal and just say date command, I would see the date and time. It works fine.

Same thing. It might sometimes ask for two different responses and say which is better. And both of them are equally bad in this case where it's saying, hey, probably should ask Siri or Google or use your local time. I really don't have access to time. And these are like actual screenshots. So why does that happen? It happened because all of these models kind of are trained on a particular data set and they have a cutoff knowledge. As you can see it on the graph, each of these models have a cutoff knowledge for a particular month beyond which they don't really know. And within that date also, sometimes they hallucinate.

So how do we enable these models to kind of get access to all of this information? That's where tools came into picture or function calling or tool calling where you can enable the model to make a tool call and figure out what the time or what the weather is in this case. Right. I'm saying what's the weather in Fremont? You can see here the LLM is making a call to weather.com and fetching the weather. The forecast seems to be 67 of high and 53 of low. And that seems to be the high here which is interesting. And same thing with Gemini or anything else. If you say what the time, it would probably say the proper time and the date because now it's able to make a tool call and figure out what the time of the date is.

This is how the function called dance works, right? The user would come in and pass a prompt to the application and the application would then say, hey, here are the set of functions that I have. And it sends that as a context to the model. Based on the prompt, the model figures out, hey, I need to call this particular function with this identifier. This is the required parameters. It makes the function call. The function call internally would probably call an API, come back with a response. The model would take the response, massage it further and send it back to the application and the application shows this to the user.

This is how the function called dance works. Yeah, it's fine. We saw that the weather call is happening. You can also do the time or the date call, whatever call you want to make. You can write a function. Everything works fine.

Here's a classic example on how the tool or the function definition would look for a tool call. You'd say it's a type of function and then you would say in this case it's a get weather function, which expects some parameters, maybe location and unit in this case. And we say location is required because if the user is asking for a weather of a particular location, I need to know the location and that's a required field.

Here's another example. You could find such experiments that I do at my GitHub link there on Notebooks. This is a pretty old example here where I'm using small agents and using Dr. Go search as a tool and using a hugging face API for the model and saying what is today's date. Because I'm using a code agent here.

You could see the model kind of wrote a code snippet and then executed that code snippet to get the answer back. And it said, OK, today's date is XYZ. And another example, a similar thing of an agent. If you remember, there was a model called PAM long back before Gemini and others came. So in using PAM models here from Google, I'm using Dr. Go again and then creating an agent and doing agent.run and say, hey, who is Haman.HM? It could do a Dr. Go search for you and come back with the result. Works fine.

But what happened is there were a lot of LLMs and a lot of APIs. I gave a couple of examples of say weather or time. But imagine there are like a plethora of applications that you need to talk and you have to write functions and each of them behave in a separate way. And you need to write functions for each of them. And there are n cross m problems where n LLMs and n number of APIs that you need to handle as a developer. That's where it gets complex. And people were like, hey, this is too much to handle.

And they were discussing, yeah, we need to define a standard. And this is a classic XKCD joke that I love, which says that there are these two figures which are discussing and saying, you know, there are like 14 standards already. What do we do? And they said, hey, we need to define another one, the 15th one. And that's when MCP happened. We all know that's November 25th, 2024 is when Antropic launched MCP and they gave this beautiful analogy of an USB 3. Maybe it kind of clicks for most of them, but it was a bit tricky for me to say, how is USB 3 kind of related to this?

MCP is an open protocol known as the model context protocol, designed to govern model behavior based on context. It operates on JSON RPC, simplifying communication between clients and servers through predefined protocols. By standardizing tool calling, MCP addresses the complexity faced by developers, offering a specific application for adherence.

A variety of companies like Slack, Gmail, and Google Calendar have adopted MCP servers, enabling seamless interaction between clients and servers. Developers benefit from using MCP as an endpoint, allowing their client to communicate effortlessly with the MCP server. MCP hosts, whether cloud-based like OpenAI or local installations via PIP or NPX, facilitate communication and access to resources with ease.

MCP streamlines communication by providing a standardized approach without the need for mixing tools and parameters. The protocol offers a range of attributes including resources, prompts, tools, sampling, transport, routes, and elicitation. Resources expose data to LLMs, prompts serve as templates, tools facilitate tool calling, and elicitation allows interactive user engagement, enhancing the overall communication process.

I just need to use that as an endpoint and my MCP client talks to the MCP server. So what do you see on the top right where there's cloud or OpenAI or Cursor? All of them are MCP hosts which have clients within them, which can talk to MCPs. So MCP can be local and can be remote. Local is something that is running on your device. You probably pull it from PIP or an NPX and run it on your device. And remote is, of course, it has a URL, it has an endpoint. It sits somewhere in the cloud and you just connect to it. And it can also access to your local data. It can make API calls.

Now the model is enriched with a lot of data and context and it's very easy to communicate because it is a standard. There is no mix and match of tool and parameters that needs to be passed and there is no guessing game that's happening. So MCP doesn't just come with tools, right? It comes of a plethora of attributes here, maybe resources, prompts, tools, sampling and transport and routes. And there's also elicitation today. So quickly on these resources are kind of entities that expose data to your LLMs, right? Resources are being used to even expose, say you want to expose a DB or file, or it could be an HTML which needs to be rendered on the client.

All of that comes through resources. And prompts are basically templates. If you have a slash command, you can do it with prompts and tools. You saw what tools for where you could make the tool calling and the LLM can figure out what tools do you have. It makes a tool call and gets a list of all the tools with the context description. It understands what this tool does from the description and such. And transport and routes are also included as part of the spec. Elicitation is interesting where you can have a back and forth between the user, right? If there is a flow, say an onboarding flow for a merchant, where the LLM needs to ask a certain set of questions and understand the context, that can also be done through MCP.

There are different types of transports. Sub-aside events were initially started for remote MCPs, but they are kind of deprecated and streamable HTTP is what is advised. Standard STO, for anything that you're doing locally and doing a quick test, you would probably use a standard STO. And streamable HTTP can also have a fallback to having SSE, where the user does a POST. Basically, the client does a POST to the server. And from the server to client, it would be a GET call. So you would have an HTTP streaming, which makes life easier, both for the client as well as the server. So it's always advised to have a streamable HTTP as your default and not really worry about SSE. But if you're early adopters and you already had SSE running, you should kind of have a fallback. So people are still trying to connect to SSE should be able to with the same endpoint with streaming, which can fall back to SSE if needed. The spec also mentions this on why they are opting to HTTP plus SSE as transport and why SSE is kind of being deprecated. So I went and read the entire spec as of that date and created a POST on MCP repo where you could go ahead and see how all of these works, how tools, resources, prompts. I had a different repo for elicitation too, which talks about how all of these works within MCP, which is a good repo to part.

Talking with PayPal's MCP architecture, I was lucky enough to be part of the Tiger team at PayPal, where we were one of the first enterprises to launch remote MCPs. And we did it in a lightning speed of five days, which is super fun and crazy for an enterprise. And I'll be sharing some of my experience being part of the team. So quickly looking into the architecture of how today the remote MCP of PayPal is, we have few layers here. We have the user layer, of course, where the users are trying to communicate with PayPal's MCP server. We have the application layer. We have the model and the API layer. And then we have the data layer. It looks like a lot of boxes, but think of it as an easy flow, right? As an user, I open up, say, cloud or cloud CLI or chat GPT or wherever and connect my MCP server. And I would just go and type in a prompt. I could say, what are my latest transactions? Or I would say, hey, go ahead and create me an invoice for $50 and send it to Sam at gmail.com, right? So it would be a natural language prompt which breaks down. And then the LLM figures out I need to make a call to PayPal. And there's a set of tools that I have. Oh, this seems to be an invoice I'll call the invoice tool and pass in the parameters and invoice creation happens, right? So in the flow, the user flow is at the very top where it could be any of your MCP clients made by Cloud or GPT, or it could be an MCP inspector where you're trying to test the remote MCP locally. And then you make a request. It goes to the application layer where we have an MCP agent which uses PayPal's tool kit. And it knows there are a list of tools that I have, which also triggers OAuth if needed.

It looks into the header. If there is a bearer token, it will use that. And then all of this is hosted today on CloudFlare, which is wonderful. So we had the CloudFlare team also working with us when we did the initial launch. And then we have the KB store for whatever we want to store, be it access token and such. So that's how the flow works.

Let's quickly look into the use case that I was mentioning in terms of creating an invoice. So at the very left, we have the user, then we have the MCP client, then we have the agent, then we have the server. We have the PayPal toolkit and the PayPal API. The dance is pretty regular. The user comes in, puts the prompts, it goes to the client, client talks to the agent, agent talks to the server, then talks to the PayPal toolkit, which you can still run locally, which has all the set of tools, which talks to PayPal's API.

Here's a quick example of using Antropix SDK to connect to PayPal's MCP server. We can now connect to PayPal's remote MCP server using Antropix SDK. Pass in PayPal's remote URL and the tools you want to use. Pass in the authorization token fetched from your client ID and secret, and voila, you'll be able to connect to the remote MCP server. The MCP spec added a section for security due to industry incidents. It evolved to add more details on security measures like avoiding static client IDs to prevent exploitation.

Here's the invoice that's been created for the request that we passed. So that is the very quick overview on how we could connect. You just need to pass in the endpoint, pass in your token, and you should be able to talk to the tools. And that's where the MCP horror story starts. If you're into this space, you'll probably have read a plethora of cases where a personal access token got leaked or people had access to their private repos and such.

So the MCP spec today also added a section for security, which was eventually not there in the spec. Because of all of this that happened in the industry, the spec evolved to add more details on what security is and how what should be handled. So breaking that down, you could say one of them is confused deputy. So what happens here is if you're using a static client ID, of course, if static, it is fixed to a client. It can be leaked. It can be shared and things can go wrong. So don't do that. That is a path for exploitation. So get user consent per client. So whenever an MCP client is trying to make a connect to a remote MCP server, get the consent, maybe through OAuth or access token that's been passed. Have the user consent.

Then the next one is the token pass through. Make sure that the token that is issued for your server is what's been used. This kind of prevents any losses of data or maybe bypasses that the token is passed around and someone else is using, which is not meant for this user from the client that should be using it. But it went to a different surface. So MCP kind of opens up a huge surface for attacks. Right. So you should be super careful on the request that is coming in. The third part is session hijacking, where you can do prompt injection or impersonalization. There's another wonderful talk up next on how you can use LLMs to attack and also how we can protect from getting these attacks. But the core rule, the fundamental is very simple. Right. It's like validate all of your tokens. Whatever tokens you're getting, please validate it and verify the request. Make sure it's an authorized, authenticated request and maintain the trust boundaries.

By trust boundaries, I say, hey, if this this auth scope is meant to access XYZ tools, they will be only able to access XYZ tools for that, until the auth expires. It's not that they can use all the tools that are in the tool set. So you can also maintain those trust boundaries. Thereby, you can still kind of get better and not get attacked on the surface.

That's when the lesson one starts is on security lessons of multi-layer security architecture. The idea of multi-layer security architecture is to validate, as I said, to validate all of your inputs on the boundary and then kind of implement a principle of least privilege, saying that, hey, this token is meant to use XYZ and it'll use only XYZ tools and not more. Also, have your secure token storage, right? You can have it in a secure store, see if it's valid. Should I re-auth, should I create a new access token?

And there's a whole auth dance of auth 2.0 on dynamic client registration and such. It's always good to also have some kind of a header where you are making sure that this is a request that my server kind of recognizes and is authorized to use. So that's where you kind of make sure having auth is crucial. The very initial launch that we did, we had auth as the part of the flow, right, which was a mandate for us. And again, there are two different flows here. It can be client flows or it can be like auth flows.

By client flows, I mean, the user could come in and when they try to use the tool, they can log into PayPal or log into any surface, do the regular auth dance on the client, give a consent and then start using the tools. Or it could be that they have a client-side secret. They are a developer on the merchant side. They have a client-side secret out of which they could go ahead and create a token and use that token in the flow. So that's where the dual authentication strategy applies. That is whether it's client-side only or whether it's first party or third party. Right.

So in the code snippet you see there, we make sure that if the user has logged in and went through the regular PayPal identity flows where they're doing the auth, then we respect that. We make sure the consent is given and we get the token back and use that. Or we see if it is a third party where the login happened or if it's a first party where the merchant is passing the access token. So both of these dual strategies applied, which kind of helps for you to have a narrow scope and not blow up the entire set of MCP tools to everybody.

Feature flag is important. Suppose you have a set of MCP tools that you want to enable to a set of developers or they are signed up for an experimental feature or they are eligible as a partner to use set of tools, then you can have feature flagging, where you could also have short TTLs on them and make sure it's discoverable. And in this feature flag you could say, hey, OK, I know this client is eligible for this and have a feature flag set for this client. Now I can serve them extra tools if needed or not. So that's the benefit of having a feature flag. And again, the organizing tools and scoping in the right corner where you see the tool category, we have our own 42 tools for first party. We have two tools for third party and for commerce we have two.

And this is evolving and building more such tools based on the use cases. In this tier of architecture, we could scope down and say, hey, you know, this is first party. So I'll surface them with X, Y, Z tools. This is third party. I can do only two tools if needed. Right. This kind of helps you to organize the tools by access level and use dynamic imports in the toolkit rather than loading the entire toolkit at one shot. We can do dynamic import and load what is required. Right. And that kind of also helps you to maintain the scope.

Production deployment strategy. When we started, we were having a CloudFlare worker to which we were quickly deploying and testing whether our flows work. Can I connect to this remote MCP and such? It is good to have this strategy where you could say, hey, you know, just build and deploy to Sandbox. And you should deploy and give a give back a Sandbox URL for you, which is easy for you or for the developers. It's more of a DX lesson where you also have a deployment strategy where it says, should it go to stage? Should it go to Sandbox? Should it go to production? And all of this should be a very easy conflict, which is a standard way of doing. But it is it's good to have it as part of your MCP development lifecycle.

Type, safety and developer experience. Of course, you have to make sure that all of your defined types are for your APIs are well-defined. And, you know, there is a good amount of test code that's written and there is no leaks in terms of type. If you're writing a tool definition, then, you know, OK, that the type is not matching. I have to make sure that everything is good. It's not that during the runtime, but rather the the tool definition time when you're coding itself, you should be able to figure out. And of course, with all the LLM assistance, it's easy if you could give a type definition and ask it to write a tool for you. It would instantly do that for you as well. Right?

There are a few in the room. 402 is like a payment request, a payment required method within HTTP. It's a standard. But X402 is basically an extension of that from Coinbase, which says, hey, you know, if I want to send money to the first person here in the audience and they happen to have a crypto wallet, I would say, hey, you know what? Send 10 PYUSD. PYUSD basically is a PayPal USD, which is a standard fiat coin, which you can use on blockchain. I could say send $10 or 10 PYUSD to Daniel. Then I should be able to do that without a middleman. That's the whole point, right? Because there is blockchain and there is a layer which can facilitate me and say whether whether Daniel's address is right, whether the person who is sending, do they have 10 PYUSD? Then I could directly send it over. So there is less overhead on the merchant and on the clients. And it's pretty instant. It can happen on the Web, too. So if you look at the very right corner, we have a server tool where you could go ahead and have an X402 implemented in that. So whenever there's a call which is being made, you can use X402 and say, hey, you know, for this tool call, you would probably need to pay like 5 PYUSD. Or it could also be like send money to Daniel, which I can use this layer and send it over. This is something that we have not implemented yet, but this is interesting, right? As I'm sharing some lessons from talking to some of the other experts in the industry, X402 is a very interesting protocol to kind of status code to kind of implement. That would kind of enable a lot of agentic friendly behavior, right? If there is some kind of an agentic checkout that needs to happen or we are seeing a lot of shopping experience that's kind of surfacing and all that GPT or cloud and such areas like Gemini shop, Google shop and such. It helps that agentic experience where sending money or checking out and basically exchange of currency gets easier with this protocol. And it's an HTTP standard. So it's wonderful to see where blockchain is being leveraged in a very positive way. This can be part of an MCP tool as well. But the lesson here is, say, if you are a company and you have around 40 APIs to start with, it is very easy for you to write an MCP server today. You could just send in a prompt to an LLM and say, hey, you know, these are my like open API schema, go write an MCP server for it. LLM can easily do it. But should we do it? It's a greater question. So one of the most capable LLM studio also today can't handle more than 40 tools. Right. It will go into a soup and it will cry. It's like what's happening? Too much context. So it's very important that you kind of take a step back and look into what are you trying to solve with your MCP? What is that the user is trying to do? What is their journey and how can I probably bake four or five, six APIs into one single tool rather than opening up all of your APIs as one independent tools in itself? Look into a broader picture and see what is the user's journey here and what are the set of APIs that they're using and how can I combine that and kind of expose it as a tool? So that that kind of makes it less of an overhead for the for the model and also less number of tools and faster in terms of for you to orchestrate. There's no lot of back and forth.

There's no lot of back and forth. So do not just wrap all of your APIs, which is easy to do. But but don't. Here's another example from Rick Roll. I just want to say roll me. Looking for available tools, it made the call to Rick Roller and there was the widget. Auto play true. Confirm. And this should load the recall video. Happy Friday. Yeah, so it's really easy to kind of have this MCP resources to kind of render UI's. If you're following the open a released app SDK, I've made a pip module called Create App, Create Open AI app, which creates a GPT app, which kind of creates an app that can run on GP surfaces. All it is doing internally is it's making a call to an MCP and returns an UI resource. If you look at the right side corner, which says UI, widget, Rick Roller, HTML, that HTML has this video link and the client just renders it. So think of it more like widgets from back in 2000. But now it's getting more baked into the surfaces using MCP. And this is really interesting because this is going to unlock a lot of use cases. May it be classic send money to log into odds to everything that can happen in the in the app itself. Right now, the user is not like switching between multiple apps rather having all of it in the same surface. So it's important to leverage MCP resources for UI.

So it happens that all that we heard so far, we are building MCP in the wrong way. If you have been following, Antropic came up with code execution and Cloudflare also talked about code mode and such. Right. So what what what's what what we have missed so far in the entire journey of MCP is, as I mentioned, is the context overload. So if you have 40 tools, it registers and gives all the tool definition and everything to the model, which is filling up the context quickly. Not only that, in terms of back and forth multi-turn conversation, this context keeps growing and it keeps gets passing to the model. Right. That's where the context window overload happens, where there is you have a lot of tools and it brings in their context, which blows up the context for the model. And it goes crazy or it is the intermediate data explosion that happens that, as I mentioned, if you are doing too much of back and forth and you are sending the same context again, and it keeps keeps building and the model goes crazy. Right.

So how can this be solved? The other part is also on the privacy and the composition. But I'll come to that part. So how can this be solved? Right. If you go back to the very first initial examples where I showed you about asking the date, we did see a code agent which kind of built a code snippet and ran that code. And it did a beautiful job in getting the date. Right. So it happens that LLMs are super smart in writing the code. So why not we leverage that and say, hey, LLM, instead of making a tool call, here are a set of APIs. Now, go write a code which has this workflow, beautiful workflow on how probably say check out should happen. These are the APIs that you need to orchestrate. Now, go write a code and then execute that code and use that result as a tool call. Rather than the other than me passing the entire API and the tool definition and blowing up the context, I would say here are a set of APIs. Now write the code and execute that code for this workflow. Cloud skills is that classic case. Right. But cloud skills are doing the same thing on cloud. Where they said, hey, cloud skills is a folder. It's kind of a directory which has all of this code, snippets, naples and how it works. So whenever the LLM wants, it will execute a piece of code rather than taking the entire context. Right. So the same strategy is kind of expanded to code execution on MCP. And this is this is definitely be be like MCP 2.0, if I could call it, because this is going to change the direction on because we hit a bottleneck. We kind of hit a roof on handling multiple tools. Right. If you have been using MCPs and connecting the tools, you'd have you would have faced it as well. So this will definitely help in terms of making MCP more efficient and not really worry about maybe context window overloading or intermediate data explosion. Right. And privacy and composition, of course, when you have personal information flows or co-ordinated execution can make it more secure because now you're not worried about like passing a credit card and a CVV onto the chat. And who knows the LLM is using it for its training and someone says, hey, give me Dan's card. And it gives me and like, right.

So I'm just saying, but all of this guardrails are there in LLMs. But it's a possibility of an attack. So now there's there's a code which is sitting on a very secure surface, which can do multiple orchestrations, run, call whatever APS it wants and come back with a result, which makes life easier for LLMs as well as the entire experience for the user.

The final and the last lesson, the lesson number 10 is something that is apart from tech. We all get super focused on solving a tech challenge. But when you're launching something at an enterprise scale, it's not just tech, but it also involves some strategy and planning. It involves go to market and involves communication, customer facing challenges, complaints and risk. We have infra and ops, operation, finance, security, name it, we have all of it.

So in that five days journey that I was talking about when we launched the remote MCP server, we had to get legal folks in, we had to get security reviews done. We had we had content writers, and one of the content writers said in my 25 years of experience, I never was able to never did such a fast paced content writing and get it all reviewed and approved. So we were all at a very fast pace, and it requires a lot of coordination and getting all of them into one room. So it's always good to kind of if you're planning to launch something at an enterprise scale, it's always good to kind of build a story ahead and say, you know, I have dependency on these folks. Probably I should give them a head start and say, you know, security.

I'm trying to do this. Can you start reviewing it? Yes. The PR is not much. We are still we have a quick demo to show you. So try to engage folks earlier on. So you kind of have the pathway set and all of them know what what exactly is being done. It's not that you build the entire thing and then asking for a review or asking for a legal consent or asking for tech writers to go and write a blog. So it's kind of a very interesting lesson. Apart from tech, all the periphery is very crucial when you're trying to build and launch things.

So MCP is not just a protocol, but a paradigm shift. So we kind of have to have a different mindset when trying to adopt newer technologies and also be happy to experiment. See when things fail and things go wrong and how do we kind of address this. It's always good to be on bleeding edge, but also being cautious about what you're trying to solve. Make sure your security guardrails and all of that is in access. Make sure the fundamentals and foundations are right. So whatever we are doing today, today's bleeding edge is tomorrow's legacy.

Maybe 20 years down the lane, someone would be laughing at this talk and saying, oh, they were doing MCPs back then. Whatever we're doing today is ephemeral but fun to be part of this journey. It's a great time to be alive. Today's AI, bleeding edge, is tomorrow's legacy. Thanks for listening to Hemant's talk.

So I think we only have time, unfortunately, for one, maybe two questions if we're quick. The first one is about the biggest use cases for financial MCPs in consumer banking and the products being built using these tools or envisioned. As the word shifts towards more agency, the spectrum ranges from user interaction like PayPal to potential autonomous checkout processes. We are currently navigating the middle ground, exploring applications that enhance shopping, checkout experiences, and financial data management.

So I think we only have time, unfortunately, for one, maybe two questions if we're quick. But the first one is going to be, what are the biggest use cases you've seen for financial MCPs specifically as it relates to consumer banking? And what products are people building using these tools and or what would you like to see people build? Yeah, that's a great question. Who asked? I'm not sure if they're in the room or remote, but yeah, that's a great question. Right. So as I was mentioning, I feel the word is heading more of more into an agentic way. Right. Look at the very left of the spectrum with PayPal as a use case where you have an yellow button on a merchant website and user goes and clicks on that yellow button. They log into PayPal. They do the regular over dance and then they buy stuff. They check out. Probably they select their bank or card and do a checkout. That's at the very left. Right.

Think of it at the very right. I'm probably thinking of buying something and some agent is able to listen to what I'm thinking. And it goes and does a checkout with my credentials or PayPal. That's at the very right where it completely thinks are autonomous. But I feel we are somewhere at the middle here where we are not completely to the right and the left is kind of fading away. And we're coming to a surface where may it be clots or GPTs or Gemini are opening up surfaces where people can do e-commerce and banking also. Right.

So I would I would see applications that would make life easier in terms of shopping, checkout and maybe financial data. So I would say, hey, how how much did I spend on shopping last month or how much did I spend on travelling or give me an insight on how I can save money? So it could it could use my open banking data or I or a link my bank account to an agent. It can go ahead and figure out all of my transactions, give me analytics and insights. That would be from the just from the banking perspective. But I feel commerce itself will become more agentic in nature and maybe towards the very right where everything is autonomous and probably not human on the loop, but like human in the loop. It would say, hey, you know, I want to buy this. Looks like you're running out of.