Google Cloud NEXT '17 - News and Updates

Securing the enterprise by securing APIs (Google Cloud Next ’17)

NEXT '17
Rate this post
(Video Transcript)
[MUSIC PLAYING] CHRIS LATIMER: Good afternoon, everybody. My name is Chris Latimer. Thanks for joining us here. I know that we're the only thing standing between you and drinks at the cocktail reception, so we definitely appreciate your attendance for our session here, today, on securing the enterprise with secure APIs. I just now realized that rhymed. I never actually said it out loud before. My name is Chris Latimer. I'm one of the field engineers from Apigee, now part of Google, and I'll let my cohorts here introduce themselves later on, when they get to their part of the presentation as well. Really excited to be here and talk to you guys today about this topic. Being Apigee, we absolutely love APIs. And so what do we mean when we say APIs, when we talk about APIs and API security today? Are we just talking about the technical aspects of API, what protocols we're using, system to system integration– that type of thing? The answer is no. We really think about APIs more broadly.

We think about them from a technical perspective, but we also think about them from a business perspective. And later on, we'll get to hear from Ticketmaster and some of the interesting things that they're doing from an API perspective and how they're securing their APIs. So if you look at what all we're doing with APIs today, it's hard to go even more than five minutes without using an API of some form. If you check social media, you're using an API. When you checked into your flight, checked the flight status, that's another API call. When you make a purchase online, that's also going to be another API call, so of course, we need to secure these APIs. Very, very important data and functionality are running through APIs, so we often want to make sure that we're going to secure those, and our first reaction is to think about something like this. We're thinking about, well, that's a lot of valuable data that somebody could get access to, that maybe they shouldn't get access to, and so what we're going to do is we're going to lock this down as tightly as possible.

Put everything in the vault. Don't let anybody get access to this stuff. Put in place all the monitoring, all the tools that we can to really lock this important capability down. And the problem with that is that, if all of your API security starts to look like this, you're missing out on a lot of the value that APIs can bring. So the hard part of API security isn't just locking everything down. If we just wanted to have a secure API, we could just not build any APIs. That's the most secure API of all. But putting together a secure API while still being able to foster innovation inside of your organization, avoid putting in place unnecessary delays, red tape, bureaucracy– doing all that while keeping your API secure is a much, much harder problem to solve. And so how do we do that? And in order to think about this problem, I like to put it into the context of imagine that you are just creating a new account online– you see a new site; you want to register for an account– and the way that we would think about this process.

If the site wants your email, sure, why not? I give this out all the time to any site. Its pretty usual that they want that. They want my address. Not a big deal to give that information out. It's usually pretty easy to find. You start to get to something like a phone number. Well, now, I'm a little skeptical. Am I going to get robocalls? Are you going to text me? Is this for two-factor auth? Is there a good reason for it? And if they say something like, oh, what's your social? Well, now, you're going to slam the brakes on. Almost nothing is going to get access to that information. You might think about it a little bit. Is there a legitimate need for me to give this information out? There might be in certain cases. You'll look and see, is this actually the site that it claims to be, or is this a broken SSL cert? And if you're comfortable with the risk reward, you're comfortable with handing out that information. An API security requires a similar approach. We're going to expose different types of APIs, and they're going to have different types of security requirements that we're going to want to impose on those APIs.

So let's look through a few examples here. Let's imagine that you're working for a health care company, maybe a hospital, and you want to create an API that's going to expose hospital locations and hours, public information. There's nothing sensitive about this, so we can just open it up and just let everybody go use this thing like crazy. Well, in a sense, yes, but there are still things we need to be concerned about. For example, this may be part of a mission critical application for us. This may be what– you know when you call up the hospital, and you ask where the nearest location is, somebody might be using the same API. When you go to the website, you may be using the same API. So if all of a sudden, you open this up and somebody starts sending a flood of traffic your way, that can still cause pretty major problems for you. So we need to think about that. The traditional approach to this is let's put in place heavyweight governance. Let's lock this down.

Let's not give anybody access unless we go through a rigorous process to understand how you're going to be using this API, how many calls are you going to send to this. But a better approach to that is traffic management, so we can put in place things like quotas and spike arrest and so forth to limit our exposure while still maintaining that culture of fostering innovation. Another thing that we'll usually do is try to put into place a service registry. Let me figure out exactly who's using which API and make sure that we're tracking this information very, very closely. A better approach of that is to use something like an API key, where we can do this automatically. So rather than have a registry that's probably going to get out of date and out of sync and give us unreliable information, what we can do is we can use an API key to automatically make sure that each application that's making an API call is authorized to do so. We also have to worry about content based attacks.

Even though this API may not expose sensitive information on purpose, if somebody starts sending in SQL injections or cross-site scripting or those type of things, we have to still be cognizant of the payload and make sure we're validating that payload. And a common way of approaching this problem is with an API gateway, and because I came from Apigee, I'm going to happen to mention, we just have a really good API gateway that solves a lot of these problems that you have that we're talking about today. Now, I promise this is not going to be a commercial for Apigee, but I did just want to mention this. So if you want to talk more about Apigee and its capabilities, we are happy to do so. So let's think about another API. Let's think about one that, all of a sudden, is opening up scheduling or doctor availability. So now, we've got a few more things that we have to think about. We have to start thinking about we don't want anybody just registering for access to this API and starting to create appointments for doctors.

We want to have a process in place that lets us vet the partners who are going to be using this API, and the traditional approach to this is to have a manual process, the same type of thing. We're going to have a form that somebody's going to fill out. We're going to take this information. We're going to decide whether or not you get access to this API. A better approach is self-service, so rather than you answer a bunch of questions about how your API works, instead of you looking through and having conversations and meetings and slowing things down and coordinating schedules, we can have a developer portal that's going to give you the ability to give your partner self-service aspects– self-service capabilities to register for access to an API, see the documentation, and start to use it right away. We are also going to need things authentications, so we're going to need identity. We're going to need to make sure that the person that is registering for a– or that is scheduling an appointment is the person that's the patient in our system.

So we need some way of identifying folks, and the traditional approach is going to be direct user authentication. If I have partners who are going to be authenticating my patients, I don't necessarily want them to have access to that patient's credentials. Instead, a better approach is to use delegated identity and user authorization, so we would have things in here like OpenID Connect, social logins, other logins where our partners can redirect that user back to our identity management solutions so that they can sign in, and then we know that it is actually them without the partner ever actually seeing those credentials. Now, what if we start to get into some really tricky areas? What if we're going to expose patient data? We're going to give information about their medical history. What do we have to think about then? Well, now, we have to think about making sure we're not end up in jail because we're breaking a bunch of laws. So there is a time and a place that we need to have very, very locked down, tight security in our APIs.

The point of this presentation isn't to say that we absolutely don't need to think about tight API security. We do, and there are times when that's absolutely appropriate. So we want to make sure that the person who's looking at this data is authorized. Is it your primary care physician that should have access to this information, or is it somebody from marketing purposes that wants to sell you a drug? They probably shouldn't have access to that information. We want to be able to have an audit trail. We want to be able to look back and see who has looked at this over time. At one of the hospitals that I've worked with before have had issues where celebrities have come in, and people have wanted to kind of see what's going on there. Obviously not something that you want to have happen in a HIPAA compliant environment. And then there are all the basic things, too. Of course, we need to still think about TLS. We need to use HTTPS. We're going to want to look at payload encryption in certain cases to make sure that our data is very, very tightly locked down.

And so from a general perspective, these are some of the things that we need to think about from an API security– as we're designing our API security. And with that, I'm going to hand it over to my colleague Greg Brail, who is going to talk about a few specific use cases and case studies that we've seen in the wild. GREG BRAIL: Excellent. Thanks Chris. So I'm going to talk now a little bit– Chris has talked about the context. Why do we want to select different types of API security? When is it important to have everything locked down to the max versus making things flexible developers? But what I really want to also talk about is get into a little more detail about API security and ways to secure APIs. And the reason why I think it's important to talk about this– if you're a web developer or have even come close to web development, and you work for any sort of moderately large company, you probably had to go through some sort of mandatory training where they taught you about all the things that can go wrong with web app development.

And you learned about CSS attacks and SQL injection and all kinds of things like that. We don't have that for APIs, and APIs are a little different than web apps. There are some things that can happen to web apps that don't happen to APIs. There are some things that happen to APIs that you don't think about as much when you're building a web app. So I wanted to make sure we talk a little bit about API security, and one of the ways I like to do it is by talking a little bit about some of the things that can go badly when you don't think about all the aspects of API security that are important. So for instance, if you remember nothing else, everything that has a URL– or if I'm really fancy, I'll say URI– has an API. So Chris said earlier, one way to make an API secure is not to have one. That's true, but if you built a mobile app, and the mobile app talks to some service on the internet, you have an API. You might not have advertised it, developers. You might not have a Swagger spec for it.

You might not have GitHub repo with sample code. But you have an API, and a reasonably smart person can, at home, jiggy up their Wi-Fi router to see what's going on between your app and your API and figure out what's happening. And a bunch of smart people have done this to embarrassing effect. For instance, the highly critical enterprise API run by Kylie Jenner was broken to by a 19-year-old who looked at it and discovered that his Kylie Jenner mobile app, or actually her website, had no API security, and he could get some interesting information. Similarly, here's a great article I found on Programmable Web that came out about a year ago titled "How to Use Undocumented APIs to Hack Your Tesla." That's something that, if you own a Tesla, it's kind of cool. Hey, I can hack my Tesla. But the truth is that these are all cases of when people actually were able to look in and see what was happening between the client and the server and figure out the API. So when we ignore the fundamentals of API security because we don't have an API, you have to really ask yourself, well, do we have an API?

Maybe if everything is inside one room or one LAN, OK, we don't have an API. But otherwise, we do, and as enterprise networks get larger, as companies get larger, as things like– as companies have global networks, everything is effectively an open network nowadays, so you have to assume that when you design your apps. So why are APIs important, and why is API security important? Well, a really great example I can talk about here is the world of banking. Especially in the US, a lot of people use services like Mint or Yodlee, which will do aggregation of their bank account information. I use Quicken a lot. It has a similar feature. And if you use any of those services, the first thing they do was say great, give me your bank account username and password. I should have used one of Chris's disturbed person images for that one because, once someone says give me your bank account username and password, it means what they're going to do is they're going to store your bank account password.

Yeah, they're going to encrypt it, but they've got to decrypt it in order to use it, and then they're going to screen scrape your bank's website. They're going to make HTTP requests as if they were a web browser. They're going to parse the HTML, and they're going to figure out what your bank is telling them. So basically, some of the banks have started to realize that this is not great for security because when their end users are giving their passwords to these third parties, you don't know what the third parties are doing with them. You don't know if the third parties are storing that data in a safe way. Even if they're storing that data in the safest way possible, they still have access to their end users' unencrypted passwords, and those passwords give you access to do everything you can do from your bank's website, including transferring large sums of money, making big stock trades, whatever you can do. If we had took an API approach, on the other hand, where banks were to offer APIs that used a security mechanism such as OAuth, that would mean that, as an end user, you would not give these third parties your password.

You would instead go through an authentication process with the bank that gives the third party token, and that token can now be set up by the bank to only offer access to these specific things that the bank wants them to have access to. So for instance, JP Morgan Chase realized this and wrote something in their annual report, and they got into a little bit of trouble with the "New York Times" that said [MOCKING] Jamie Dimon wants to protect you from innovation. [NORMAL TONE] And he has a point, and they have a point, too. But the idea here is by offering open APIs with well-known API security techniques such as OAuth, we can have a much, much better handle on security because security is not about eliminating all risk. Security is about managing risk. And by giving third parties a way to have access to important information like financial information in a controlled way, where you're only allowed to do what the bank allows you to do, we're in a much better position than we are today, where you give someone your password, and they can do anything.

So I'm now going to talk about five or six specific API security things and kind of use some examples. So one big API security mistake we can make is to not use TLS, or SSL as it used to be called, and now the protocol we're supposed to use today is actually called TLS. So for instance, a security researcher did a study last year, and he discovered that, around the world, especially in South Asia, at the time, something like 30% used non encrypted links. Two years ago, that was 90%. Now, maybe that's not your password, but we're to the point now on the internet where everything that is even slightly important, and in fact, even everything that isn't, should be encrypted with TLS. That's why, when you do a Google search, is encrypted with TLS. The cost is just not there. But it's more than just, hey remember to use TLS, don't put an unencrypted API on the internet. It's how you use TLS. So for instance, in some frameworks for the client, it's very easy to turn off TLS verification because you're doing development, and it's kind of a pain to get the right certificate, and the name on the DNS record doesn't match the certificate, and we're in a hurry.

So we'll just check that box in Node.JS or PHP or Java or whatever that says don't verify the TLS certificate. That opens it up for your client app to be able to have its supposedly secure web traffic be intercepted by anyone in Starbucks who has control over the Wi-Fi or over the DNS router. TLS verification is tremendously important. Not only that, TLS has a bazillion options. What version are you using– 1.0, 1.1, 1.2? What cipher suite are you using? Are you using Elliptic Curve? What encryption algorithm are you using? And these things change all the time. So as anybody who's offering an API or any web app and is using TLS, you have to remember to keep up with what's happening in the security world, follow the best practices from your security people and your company and elsewhere, and make sure that you're continually evaluating how you're using this technology and keeping up with the latest attacks. For instance, I still see developers sometimes occasionally want to write code, and they want to hash something, and they use MD5.

MD5 has been a no-no in the security world for years. In fact, at Google, it's not even in the code base. They took it out. But people still remember this from a few years ago. So this is why it's very important, anytime encryption is used, to really think about this. So now, we move on to some of the other basics of APIs. Chris talked about APIs that require an end user to be authenticated. There are all kinds of things that can go wrong with this. So for instance, the Nissan LEAF a little while ago came with a mobile app, and the mobile app let you do things to your car like turn on and off the climate control or start your car remotely. And to make it easy for people to use the app, the authentication mechanism for the app required you to put in the car's VIN number, which is a unique ID that's on every car. It's actually on the dashboard of your car, so it's not a secret, and it is structured in such a way that every type of car, most of the digits are the same, and only the last five digits vary.

So you can imagine that if you knew that, hey, I can turn on the climate control on one Nissan LEAF using a VIN number, what if I were to cycle through 10 or 15,000 VIN numbers and see if I can do that other Nissan LEAFs in the world? And a security researcher named Troy Hunt figured out that the answer was yes. Now, I'm not saying this to make Nissan look bad, because they fixed this. I'm saying this to give you an example of what happens when you don't think through API authentication. There are a lot of best practices out there for basic things, like signing users in, checking passwords, resetting passwords. It's actually pretty hard to get right. Companies like Google do a good job of it. Smaller companies that are specialized in the security world do a good job of it. Large companies that don't, don't always do a great job of it. I've seen some horrible password reset practices from various websites I use in my life. These kinds of things can show up as massive API security vulnerabilities if you're not careful.

But it's not just about authenticating the end user. In the world of APIs, one thing we do that we did not do in the world of web apps is we also authenticate the application. If you were to ask me what does API management do– because I am an API management guy. That's what we do at Apigee. API management is about managing the relationship between the developers who build applications and the APIs that they use, and a big part of that is managing those security credentials. OAuth, which is the de facto standard for API security now– and in fact, from a lot of security now– has built into it the concept of application credentials. In order to build an application that gets an OAuth token– say they use a Google Cloud API– you not only have to supply some user credentials, but you've got a supply application credentials, which the application developer gets as part of the development process. Now, the application of credentials are not 100% proof against attacks. I mean, anybody who has access to the application can use those application credentials, But what they do is they give the security team a lot of extra layers of defense, and that's what security is really about.

So for instance, we have had cases of Apigee Edge customers who had– we had one who accidentally pushed a version of an app to the App Store that had a bug in it. And it was actually hitting an authentication API in an infinite loop, hundreds of times a second, and it was soon overwhelming their whole technology stack. Now, if they had a different set of application credentials for every version and platform of that app, they actually would have been able to cut that app off instantly in their API management tier, or redirect traffic from that app to some sort of other server, which is able to mediate it. And as a matter of fact, they didn't quite did that, but they did have the application authenticated, so they were able to take some of those steps. But that's an example of why it's important to not only authenticate the end user but the application. This also gives you an opportunity to do some things like analytics, and we'll talk a little bit later about bot detection.

But knowing which application is making an API call is one of the pieces of information you can use in that world. Now, I mentioned OAuth. OAuth has been around for a couple of years. It's become the de facto standard for API security. And basically, one reason why it works is because what it allows you to do is it allows a client who makes an API call to exchange some credentials for a token, and that token gives you access to the API. And with OAuth, it's possible for the API provider, using software like Apigee or many other bits of software, to control what a token is allowed to do. Furthermore, unlike a password, a token uniquely identifies a single application on a single device. It's not just a generic bit of security information, and OAuth tokens can be refreshed. But OAuth is a fairly complex family of specs. They are IETF standards. There are a lot of smart people, including some of our security folks at Google, working on it, but there are lots of ways to use OAuth.

So for instance, one way to use OAuth is to make an API call where you send in your password and get back the token. Another way to do it is to have the user's device open up a web browser window where they can get a token by being redirected to a website, and that website can do additional things like check cookies, ask for a second factor, do all kinds of things like that. Now, it turns out that the OAuth folks have actually been working on a spec that describes, hey, these are the best practices for how to use OAuth in native app. We recommend using this web browser redirect method. Here's how to do it safely on different mobile platforms. Furthermore, there have been a couple of things that have been found as vulnerabilities in the OAuth specs that have been fixed in some of the newer IETF specs. So it's really important, if you're going to use OAuth, to pay attention to some of the requirements that you see in the community and some of the discussion that goes on, especially in the IETF, because there are a lot of smart people who have written a lot of really good stuff about how to use OAuth correctly.

I should also mention authorization, and in this form of authorization, I mean once you've authenticated a user, and you know who they are, what are they allowed to use. It's easy to make some dumb mistakes. This one security researcher discovered that his bank actually– I don't think it was a very big bank, but they actually were not verifying the account numbers on the back end. So he figured out the API that his mobile app was using, and he was able to transfer money from any account to any other account in the bank. Now, that's a huge mistake. That's like people getting fired mistake, but these kinds of things can happen. Developers don't always think of these things. One of the things I like about APIs is that an API is a contract, and it's exactly clear what is supposed to be sent between the client and the server. So unlike a web app, where you're downloading JavaScript and executing who knows what, a provider of an API can specify exactly what's sent, exactly what's received, what the valid responses are.

That means you can write test plans and regression tests and penetration tests, and you can verify that what you think your API is, is what you actually implemented. I think that's actually an advantage of APIs, and it's an advantage of API security. And by having that kind of spec, and by thoroughly testing that spec, you should be able to prevent these kinds of dumb authorization mistakes that– they're easy to laugh at now, but there but for the grace of God go all of us. Right? Another one, this is really important, is rate limiting. So the best example of API rate limiting I could find came from a researcher that did some work a couple of years ago on Snapchat when they were a much smaller and less mature company. He discovered that, given a phone number, it was actually possible to get information about someone's Snapchat profile. Now, that doesn't sound like a big problem, but it turned out that there was no rate limit on this API, and he was able to send in huge batches of phone numbers.

Now, if you could imagine say Beverly Hills, California– I guess people use cell phones now, but even then, it doesn't have that many phone numbers in it. It wouldn't take very long to get a lot of interesting information that way. So this was, to me, the best example of, in the real world, an API that should have had some sort of rate limit to help with security. And I mention this because pretty much any API that could be subject to a brute force attack is something that will be subject to a brute force attack. And it may be subject to a brute force attack by someone who is just hammering it for fun, but it could be somebody with legitimate credentials. Lots of people can sign up for these services, so things like product IDs, user names– if you can put in someone's phone number and get information. Another great example is that OAuth password grant type, where you send in your password in an API call, and you get back to a security token. Well, if there's no rate limit on that, now you have a distributed password cracking API that people can now use to crack your password database.

So it's great to monitor, and this stuff is important. But by having basic rate limits on APIs– like no one should call this API more than 500 times a second, or a given application is limited to a certain number of APIs calls per day or hour or month, or even possibly introducing artificial delays to some APIs that may be called very frequently. Those make a very, very big difference in the kind of security posture you have and prevent these kinds of attacks. And also we're going to finally mention probably the biggest security list of things that developers need to know about, comes from OWASP. It's called the Top 10. I couldn't find anything since 2013, but basically, they list the top threats to applications on the internet. Interestingly enough, they don't all apply to native apps that use APIs. If you're building APIs, some of the JavaScript things and cross-site scripting things don't really apply to you, but the rest of it does. Injection attacks, misconfiguration of security, bad authorization decisions– a lot of the things that I've talked about are in here, but there are some things I didn't talk about as well.

For instance, the number one attack is an injection attack, so classic SQL injection stuff, like can I put semi-colon delete star from customers in my user name. It applies to APIs just as much as it applies to web apps. Yes, there are things you can do with regular expressions and stuff to protect from this, but it's really, really important that, as application developers, we remain just as cognizant of this in the world of APIs as we do in the world of web apps. So to sum up this part, and then I'm going to talk for two or three more minutes and turn it over to Ismail. I think everyone needs to know some basic things about API security, and this is not necessarily the same things you need to know for web app security. And a big part of it, in my opinion, is understanding the importance of things like a rate limiting, traffic management, content attacks, and basically, the idea that anything that can be called with a URI, someone is going to figure out what your app is doing and try to break it.

And as app developers, we have to be very cognizant of that. At the same time, by designing APIs that follow the right security principles, by making smart choices about how much security we need on an API. And by making sure that our API definitions are clear and well understood, we've now designed something that we can test that has a much smaller surface area than a web app. And I would argue, if you follow all the rules, and you do everything right, its possible to make things with APIs that have fewer security risks than what we see today with web apps. Now, I'm going to talk for just a short time about Apigee Edge. This is the product that has been acquired by Google as part of Google Cloud, and basically, what we do in Apigee is we provide a product that lets us insert ourselves into the world of APIs. So many companies, when they're looking to offer an API, they're asking themselves how do I implement a lot of these things? How do I implement OAuth and connect it to my users– to my existing user authentication system, my existing single signon system?

How do I implement a rate limit? How do I do checks, basic threat protection checks like malformed JSON and XML. These are the kinds of things that we've always been built into the Apigee Edge product. And the idea of Edge is that, as an API provider, you can put the API management product– and there are a lot of products in this category– in front of your API, and it can handle a lot of these aspects, so that your back end, the stuff your developers are building– you don't have to build this into every service. The larger the company you are, the more it makes sense to have one place where you apply many of these security policies as opposed to building them into every single application. Many large companies– Google has an API management for its own APIs. All the big companies that have APIs have API management as part of their software stack at the edge of the network, somewhere between the load balancers and firewalls and the stuff that actually runs the APIs. What Apigee Edge is– it's a product that allows you to configure it to be that API management layer for the API and configure it however you need it to work.

Now, I'm going to talk for two minutes here about Apigee Sense, and this is the last thing I'm going to talk about, which is a thing that we've used to extend Apigee. And this is going to segue nicely to what Ismail's going to talk about, is we're going to talk a little bit about bots. So there are all kinds of bad things that can happen to APIs. People who are not legitimate users may try to hammer your API, to crack it, to brute force it just to mess with you. But what can also happen, which is a little bit more subtle and harder to deal with, is when legitimate users who have got legitimate application and end user credentials use your APIs in an illegitimate way. This happens a ton, for instance, in the travel industry, where, gee, every hotel has the same prices. Well, it's because somebody somewhere is screen scraping or botting the APIs of all the other hotel websites and trying to book a room in every single price category to try and find out what the prices are and what the availability is.

There's a lot of bots that happen in retail. Ismail comes from Ticketmaster. Some of you may have used Ticketmaster. They have some experience with bots, as much as they're able to tell us. So Apigee Sense is a product that uses the fact that we collect a lot of data in the Apigee platform about the APIs that we handle, and basically, we can use that information– it's information about IP addresses and things– to figure out and to score incoming API calls on whether or not we think they might be a bot. Some of this is heuristic, like for instance, American retail companies don't have a lot of customers in Eastern Europe. Or some of it is heuristic like iPads tend not to be located in Amazon data centers. Some of it is machine learning, and some of it is standard big data stuff. Was one OAuth token used to make more APIs calls than a human user could have made using an app? Was one OAuth tokin used to scan through APIs, like to scan through all the phone numbers in Beverly Hills, from 0000 to 99999?

These are the sorts of things that you can see, and by looking for patterns like this, we're able to then score incoming API calls as definitely a bot, like we know this IP address is a bot because everybody is getting bot traffic from it, to probably not a bot, to something in between, and then take action. Sometimes, take action is block the API call, but that lets the bot people know that you're blocking them. So sometimes a better action is to let the back end know so it can provide a different result, a different price, a different availability, even slow down the API traffic. So that's kind of what we're looking at doing. So with that, that's really all that I have to say about Apigee. I'm now going to turn it on to Ismail, who is an SVP at Ticketmaster and is going to talk to us about his experiences with APIs and API security. Thank you very much. ISMAIL ELSHAREEF: Thank you, guys. Oh, whoa, it's loud. Hi, everyone. Thank you, Apigee and Google, for having me here.

This is really exciting to be here to talk to you guys about our experience with APIs from a business perspective. I'm going to talk a little bit about the tech involved as well. So at Ticketmaster, we're all about delighting global fans in a distributed world. A little bit about myself before I jump into that. I've been in this battlefield of tech and product for a while now. I started as a software engineer, back end engineer and front end engineer. I've been doing APIs for the last seven years or eight years. Joined Ticketmaster 18 months ago to open up the platform and open up their APIs, and six months ago, I started running the entire product organization– consumer product organization, so very, very exciting. So show of hands– who's been on Ticketmaster the last six months? Tried to buy tickets? I can't see. Not a lot, actually, which is surprising. So keep your hands up. Keep it up if you were not able to buy tickets when you visited Ticketmaster. I don't need this.

I don't need to talk. I don't need to talk. It's all good. It's happening. Oh, we have one person. OK, I'm here for you, man. I'm here for you. So we have problems. Not every fan that comes to could buy tickets. So that's why, when we started, what really got me excited about the opportunity at Ticketmaster is the leadership there– Jody Mulkey, the CTO, and the entire team– is that we're really reinventing Ticketmaster. We're changing the way we're selling tickets, and that's by having a fan first thinking. From our infrastructure, and completely changing and upgrading our infrastructure– we have an incredible partnership with AWS and Amazon where we're moving our private cloud into AWS, and it's a process that we're working on right now– into opening our APIs, and that's a partnership that we have with Apigee that's been fantastic. So infrastructure opening up the APIs, and then our products had to be built with fans in mind first, which is not traditionally the way it's been done in the past.

And what that means is that really looking at all our fans, different personas, and understanding why we're building the products that we're building. And our promise to our fans is that we are unlocking moments of joy for global fans through unforgettable product experiences. So if you think about that, and thinking about the fan and fan first, you'll see that– and I don't know if you guys follow Ticketmaster– we have been changing the way we offer value to fans. So for example, this happened today. It's a pre-sale for Ed Sheeran, who's coming to North America. It's going to be one of the biggest tours this year. His new album dropped a week ago, and it's killing it. But this way is for us to say, hey fans, preregister, and we will text you a link before the presale where you can buy Ed Sheeran tickets. We did it with 1975, John Mayer, Depeche Mode– you name it. Because what we find is that, when fans come to Ticketmaster for Depeche Mode, for example, who– they haven't toured in seven years, and if you can get tickets, you're going to freak out because I will freak out, too, because I've been wanting to go.

And I signed up, and I got– yes, we have a fan in the house. I love it. But it's real. No one's going to drop $4,000 to go see Beyonce at the Rose Bowl if they're thinking logically. So it's an emotional business that we're in. We just want to make sure that fans are not disappointed, are not turned down, and are actually getting the tickets that they want. We see 72 million unique visitors a month. Not all of them are fans. So we have a real, real need here, and a real problem to solve with some of the challenges we run into. So tickets are the only commodity I can think of that go up in price– up in value once you buy them. I don't know if you guys bought tickets to Hamilton recently. I have not seen it. I can't afford it. But we deal with a lot of issues and a lot of problems. Bots are one of them. Limit violations– people are trying to circumvent the limits we put on an event. So as a fan you cannot buy– for certain events, you cannot buy more than eight tickets.

For some events, you cannot buy more than four tickets, just to make it fair and give access to a lot of other fans that want to go see the act. And some people and some bots and some bad actors trying to circumvent that by they want to get, I don't know, 100 tickets. I'm talking about concerts now, and theater, not sporting events per se. So there's that level of complication and challenge that we have, and then the biggest challenge we have is reserve and do not buy. So during an on sale, when people bad actors come and select a pair of tickets or a bunch of tickets, and they don't actually end up buying them, what happens is that when they actually select them, those tickets are put into a hold, and no other fans can get to them, which is bad. We have ways of going around that, obviously. We've been doing this for a while now, so we have our own algorithms and our own systems that can kind of determine or try to glean whether this traffic is coming from a good actor, a real fan, a bad fan, a robot, a non robot, and all of that good stuff.

But that's just, Ticketmaster apps. That's just our closed system. But in order for us to deliver on our promise to our fans, our global fans, we determined that we have to open up the platform. We have to make our APIs available to other developers, to other partners, clients, where they can build applications or can build experiences for their own fans, so everyone gets access to those tickets. So when I started 18 months ago, what we did was we consolidated all of these different platforms that we have into one simplified, very easy to use platform partnered with Apigee to expose those APIs. And they would start using those APIs across the board in all different channels starting with our own apps and their own properties, but also for pre-sales on fan clubs, for example, last minute channels, you name it. So we have different outlets for those APIs. But the reason we wanted to go this route, obviously to open up the platform, but also for any measurement of security that we apply to one channel now, we apply it to the APIs, and that's it.

We don't have to think about it anymore. Because when you apply it to the APIs, it doesn't matter who's using the API. The same level of security is going to be applied. In our world, what Greg said, that security is not eliminating risk. It's mitigating risk, and for us, it's all about mitigating risk. It's all about mitigating risk. We'll never know, in real time, who's a true fan and who's a bot. So if you're like me, if you go to a show once a week– and I pay out of pocket, by the way. I don't get free tickets. If you know how to get free tickets, let me know. My behavior and my pattern of activity can look like a bot to our system because I'm coming from my home wifi trying to buy tickets to John Mayer, to 1975, this and that. And it's like, oh, my god, this guy or that entity is buying a lot of tickets, so maybe it's suspect. We're trying to make it smarter and better, but true fans and super fans can come across as bots, and that's a problem that we're trying to solve.

But we opened up the platform. We have a portal, We created an ecosystem where you can go to the portal and understand what APIs we have, who's using them, and how you can get access to them. We really followed all the best practices and everything that Apigee advocates and puts out there. And from my previous experience as well, all the best practices that I learned the hard way sometimes, that we should be doing in order for us to have a successful API program. So we've done all of this stuff. We integrated with some incredible partners to get access and to reach other fans that might not necessarily come to And as a result of that opening up the platform and really thinking broadly, we've seen, just last year, that 10% of all tickets sold by us came off of platform, came through partners, came through our APIs, outside of our products. Fascinating, right? But guess what– the challenges still remain. The challenges that we've seen in our own products still remained true in the open platform world.

They just got amplified. Actually, almost all the errors we see in the Apigee portal coming through our APIs are of these two kinds of errors. Too many requests, rate limiting– so people trying to make too many API requests, trying to get access to those tickets, reserve them. And because we have rate limiting, it's preventing our system from going down and overwhelming our system, but also really controlling the API access for that particular user or application. But also look at the unauthorized calls. A lot of people are trying to make calls without being authorized, without the right OAuth token. It's a real problem. And because building an API– a lot of people think, oh, it's a simple interface, and I have the SDK and documentation. It's pretty simple. Right? It actually takes a lot of effort and planning, especially when it comes to security, identity management, who has access to what, error messaging, all of that– it takes a village for the API to actually be useful and to be simple.

That's the challenge we run into, so when we focus on the fan and opening up everything and trying to think how do we make that access of this API that we just put out there favorable to fans, and for developers to build applications that actually let fans through. So the way we did it was really follow the basic best practices. I cannot emphasize that. A lot of times, the biggest problems happen because you don't follow the guidelines and the best practices, like authorization, authentication, OAuth, rate limiting, using bot blocking that Apigee provide if you don't do this for a living like we do, SSL, just the basic stuff. And if you do the basic stuff, you're actually 50% there, like Greg and Chris were talking about. In our case, we really had to think about the reserve API call. That's where actually people put real tickets, real value, on hold, so we need to rethink how that API call actually works. So the way we did it was we're introducing friction. We're trying to introduce friction.

It's like you cannot do that anonymously. You authenticated your application. We have the OAuth token. Great. But now, we want to know who's the actual user making that call, because then I'll have extra information about that user that will help me determine is that really a legitimate fan, or is that a bot that's trying to– that hacked the app, and now using it in a bad way. So introducing friction by forcing registration or signing in, using whatever you want to sign in for. We use Ticketmaster Accounts for that. And using the fan score, which is our algorithm to determine whether you're a good actor or a bad actor– basically, a fan or a bot. And the last thing, obviously, which is really, really important– and we just did that with Apigee– is identity management, entitlements. So when somebody's making an API call with their OAuth token– so if that checks out, that means Apigee says, OK, this OAuth token is good. Now, taking their API key and their identity and pinging our system, saying what role does this player have in our system?

Are they a supervisor? Are they a fan with a bunch of tickets that they can transfer, and all of that stuff? And once you get that back, you add a bunch of headers to the call, and then when we get it in the back end, we understand if this action is permitted by this user or not. So if you want to transfer some tickets– now just because you're a fan doesn't mean anything. You have to be a fan and also have the rights to transfer those tickets or to sell those tickets or to cancel an order. So these are the three learnings that we found. Just opening up an API ecosystem and a platform actually works really well. Is it perfect? It's not. It's never going to be perfect, and we're always learning every single day. But the fact that we sold 10% of our tickets off of platform using our APIs that we opened up just 18 months ago is a huge testament to the power of APIs to business, but also the fact that you have to stay vigilant in order for you to really expand. I mean, we haven't scratched the surface with APIs just yet.

And with that, I thank you for being here, and we're hiring. We want people to help us with this transformation that we're on. And I guess I'll open up to questions. [MUSIC PLAYING]


Read the video

In a world of mobile apps, cloud offerings, and microservices, where enterprise data is accessed with APIs in a zero-trust environment, deep API security is critical. In this video, Chris Latimer, Greg Brail, Prithpal Bhogill, and Ismail Elshareef discuss how organizations can protect against all kinds of external threats by enforcing a set of consistent security policies at the API proxy layer, creating proxies in front of their APIs with Edge, and using Edge/Sense to identify and mitigate bots by using API access behavior patterns. You’ll learn how Google’s machine learning capabilities further advance comprehensive API security.

Missed the conference? Watch all the talks here:
Watch more talks about Connected Business Platform here:

Leave a Comment

Your email address will not be published. Required fields are marked *

1Code.Blog - Your #1 Code Blog