Understanding Web APIs: What They Are and How They Work

A common question we get is, “Do API teams need to do discovery?” My short answer is yes.

But I realize to effect change in the industry, I need a longer answer. Over the next several weeks, I’m going to do my best to provide a more complete answer. Here’s the plan:

• In today’s article, I’ll explain what APIs are and how they work. My intent with this article is to help folks who don’t work on API teams to understand the context for the next several articles in this series.
• In next week’s article, I’ll get into the most common usability challenges developers face when starting with a new API.
• In the following week, I’ll share my personal experience using a variety of APIs, how it got me interested in this topic, and why it convinced me this is a much-needed area for more discovery.
• And in the subsequent weeks, we’ll share stories from real API teams who are already putting the discovery habits into practice.

Now if you don’t work on an API product, but you are a fan of discovery, I want to encourage you to keep reading and to follow along with the series. I’m going to need your help.

Here’s the challenge. We have two groups of folks: 1) API teams who may not be familiar with discovery and 2) discovery practitioners who may not be familiar with APIs. I want to bring these groups closer together.

I know there is already some overlap between these groups—particularly the folks who focus on developer relations. But the more overlap we can create, the more likely we’ll get better designed API products.

One of my goals with this article is to arm you, the discovery practitioner, with what you need to know to help encourage your API teams to do more discovery.

If you are already familiar with the technical underpinnings of APIs, you might not need this particular article. But I do encourage you to follow along with the rest of the series. Next week, we’ll get into the most common usability challenges engineers face when working with a new web API and then we’ll dive into several real-world stories about API challenges and how the discovery habits can help.

What’s an API?

API stands for application programming interface. It’s a way for code to interact with other code.

An API is a pre-defined language or instruction set that tells a programmer how their code can communicate with another service. For example, Google Calendar has an API that allows you to create events, add guests to an existing event, update an existing event, and so on.

Google Calendar’s graphical interface allows you to create calendar events in your web browser using your mouse and keyboard. Google Calendar’s API allows engineers to write code to programmatically create events.

Why would someone want to write code to programmatically create a calendar event? Let’s consider an example.

Suppose you are creating a website for a hair salon. Whenever a customer books an appointment, you want to add the appointment to the stylist’s calendar. When the customer selects the date and time for their appointment, the details get sent to your web server. Your web server code can then send the event details to the Google Calendar API and request that a calendar event get created on the hair stylist’s calendar.

Before APIs, a staff member might have had to manually enter appointments into the calendar. With the API, this step can be automated.

Most of the products and services that you use integrate with other services. These integrations are supported by APIs. For example, when a merchant uses Stripe to accept payments, the merchant website uses the Stripe API to process the payment. Ads get delivered via APIs. When you visit a webpage that is serving third-party ads, the server that is hosting the website requests the ads that it displays from the ad provider through an API.

Most of the modern web wouldn’t work without APIs.

Understanding HTTP: Most Modern Web APIs Build on Top of HTTP

On the internet, one of the most common API standards is REST. REST APIs allow programmers to send and request data to and from servers using HTTP, the same protocol we use to request web pages.

I’m going to briefly describe how HTTP works. Having a rough understanding of the basics will help you better understand why more discovery is needed on API teams.

Here’s the tl;dr on how HTTP works:

• HTTP defines how a client communicates with a server and how the server responds to the client.
• A request from the client includes an HTTP method, a URL/endpoint, headers, and an optional body.
• There are predefined HTTP methods (request types) that tell a server how to act on the request.
• A response includes a status line (with a code and a text description), headers, and an optional body.
• There are commonly used status line codes that indicate if a request was successfully executed (e.g. 200 OK, 404 Not Found, 500 Internal Server Error)

If you want to know more about the technical details of HTTP, keep reading. If you’ve got the gist of it from the tl;dr, feel free to skip to the “REST APIs: Understanding the Most Common Web API Format” section.

The HTTP protocol is simple. It’s a standard that defines how you request data from a server and then defines rules for how the server responds.

HTTP Requests

Requests consist of 3 parts:

1. The request line which includes a method, a URL/endpoint, and an HTTP version
2. Headers, which allow you to send metadata about the request
3. An optional body, which allows you to send additional data related to the request

For example, when you go to google.com, your browser sends a request that looks like this:

1. The request line: GET google.com HTTP/1.1

GET is the HTTP method. More on methods in a minute. Google.com is the URL/endpoint, and HTTP/1.1 is the version.

We’ll see later that the HTTP method and the URL/endpoint are going to be used extensively in REST APIs.

2. The request might include the following headers (among others):

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

The User-Agent header tells the server what browser and operating system you are using. You can see your browser is also sending along headers that tell the server what types of responses it understands.

In this case, your browser is telling the server that it understands text/html and that it can accept encodings that are gzip, deflate, and br, and that the expected language is English.

Don’t worry about all these technical details. All you need to really understand is that headers allow developers to send additional metadata about the request to help the server better respond to the request.

With REST APIs, headers are often used to send user credentials, so that the server can validate if it should respond to the request.

If an API request is like sending a letter, the headers are like the envelope—it tells the server who’s sending it, what kind of message is inside, and what to do with it.

3. An optional body

GET requests don’t typically include a body. But we’ll see later that other HTTP methods will require a body.

Understanding HTTP Methods (or Request Types)

When we type google.com into our web browser, our browser is making a GET request to the Google server. This is only one of several request types that HTTP supports.

You can think of request types as commands. In the request you are telling the server what you would like it to do. Here are some of the HTTP methods:

GET -> Retrieve resource (e.g., fetch user info)
POST -> Create a new resource (e.g., add a new user)
PUT -> Replace an entire resource (e.g., update all details of a user)
PATCH -> Partially update a resource (e.g., change only the email of a user)
DELETE -> Remove a resource (e.g., delete a user)

For example, when you fill out a form on the web and you hit the submit button, your browser doesn’t send a GET request like it did when you entered a URL. Instead it sends a POST request. It’s telling the browser: Take the data from this form and create something with it. The code that receives the request contains the logic that tells it what to do with the form data.

The most important thing to remember about HTTP methods is that they tell the server how to act on the request. The HTTP standard defines these methods to work in a specific way. As we get into usability issues with APIs, we’ll see that the developers who create APIs don’t always follow these standards and it can create confusion for the developers that are using their APIs.

HTTP Responses

When a server gets an HTTP request, it has to generate a response. Think about it like a call and response. The browser asks for something and the server responds.

Responses also consist of 3 parts:

1. The status line (an HTTP version, status code, status message)
2. Headers
3. An optional body

If you are thinking this looks an awful lot like a request, you are right. Let’s break down each element.

1. The status line consists of three parts: The HTTP version, the status codes, and the status message.

When you enter google.com in your web browser, the status line in the response is typically:

HTTP/1.1 200 OK

HTTP/1.1 is the version, 200 is the status code that means everything worked as expected, and OK is a text description of the status.

If something went wrong, the status line would look different. For example, if Google received your request, but the Google code encountered an error while processing your request, it might send back:

HTTP/1.1 500 Internal Server Error

With this response, the server is telling you it couldn’t process your request.

There’s a server error response that you are likely already familiar with, even if you didn’t know it was an HTTP status code. That’s the dreaded 404 page. When you send a request to a server for a specific resource (like a webpage) and it can’t find that resource, it will respond with:

HTTP/1.1 404 Not Found

As we get into how REST APIs work, you’ll see that developers use this same status code and status message protocol to return API errors. You’ll also see in the upcoming stories that we share that error codes are often a source of frustration for developers.

2. Headers

HTTP responses also include headers. Just like in a request, headers include metadata about the response. For example, when requesting a webpage, the server responses will often include a header that looks like this:

Content-Type: text/html; charset=ISO-8859-1

This is telling your browser that the contents of the body is text/html.

3. The Body

When requesting a webpage, the response body is the html of the page that you requested. It’s what your browser renders in your browser window.

REST APIs: Understanding the Most Common Web API Format

Now that we have a better understanding of HTTP, we can introduce REST APIs. Remember, an API is simply a predefined language that allows code to talk to other code.

REST is a framework that builds on top of HTTP. A REST API is an API that follows the HTTP standards to define how code should talk to other code.

In our web browsing examples, our web browser (the client) made requests and the web server responded. With REST APIs, a developer’s code (the client) makes requests to a web server that responds according to the rules of the API.

When developers write code, they construct requests that include an HTTP method, a URL/endpoint, headers, and sometimes a body. The server that accepts the request responds with a status line, headers, and a body. The developer’s code can then act on this response.

Let’s return to our hair salon example to further illustrate the similarities between requesting web pages over HTTP and making API calls with a REST API:

• A customer enters the hair salon URL into her browser. The browser sends a GET request to the web server and responds with the salon website.
• The customer fills out the appointment form, selecting a date and time, and hits submit.
• On submit, the browser sends a POST request to the hair salon’s website.
• The server receives the POST request with the form details and runs some code.
• That code makes a POST request to the Google Calendar API, asking the Google Calendar server to create a new calendar event on the hair stylist’s server.
• The Google Calendar API creates the calendar event and responds to the hair salon code with a status line (200 OK), headers, and a body. In this case, the response body is a description of the event that was created.

So, how exactly do these code-to-code requests and responses work? Let’s get into the details.

Because we are about to get technical again, let me start with another tl;dr:

• API Endpoints determine which types of resources an engineer can interact with.
• Each endpoint supports some subset of the HTTP methods (e.g. GET, POST, PUT, etc.) dictating what actions are available for each resource.
• In order to create or update a resource, an engineer needs to know how to represent that resource. The API documentation needs to communicate how each resource is represented.

Endpoints Define What Types of Resources a Developer Can Interact With

An API makes a number of URL/endpoints available to developers. In a well-designed API, the endpoints map to resources.

For example, in Google Calendar, you might have the following resources with their corresponding endpoints:

Calendar Lists – a list of calendars for a given user
https://www.googleapis.com/calendar/v3/users/me/calendarList

Calendars – specific calendars
https://www.googleapis.com/calendar/v3/calendars/{calendarId}

Events – specific events
https://www.googleapis.com/calendar/v3/calendars/primary/events/{event_id}

By exposing these endpoints through their API, Google allows developers to interact with each of these resources. For example, a developer can request a list of calendars for a specific user. Or they can request a specific event.

HTTP Methods Determine How Developers Interact with Resources

Each endpoint defines how a developer can interact with its resource by defining which HTTP methods it supports.

For example, here are some of the HTTP methods that the Google Calendar Events endpoint supports:

DELETE – deletes the specified event
GET – fetches the specified event
PATCH – update the specified fields for a specified event
POST – for creating a new event

By sending requests to the Events endpoint, a developer can delete an event, fetch an event, update an event, or create a new event.

When browsing the web, our browser sends GET requests to fetch different web pages. When a developer is using the Google Calendar API, they might use a GET request to fetch specific events.

The other HTTP methods allow developers to do more than just fetch resources. They can use a DELETE request to delete a calendar event, the PATCH request to update an event, and the POST request to create a new event.

Resource Schemas and How They Affect Requests and Responses

If an endpoint maps to a resource and each endpoint has a set of HTTP methods that allow a developer to interact with that resource, we also need to know what that resource looks like.

For example, we can’t create a new calendar event unless we know what’s required to create a new calendar event. API documentation also needs to include all the fields associated with a given resource.

Based on what you see when you look at an event on your Google Calendar in your web browser, you might be able to guess what some of these fields are. They include an event title, a description, an organizer, a start time, a start date, an end date, an end time, and so on.

The Google Calendar event resource includes dozens of fields. We don’t need to look at all the details. Instead, I’ve provided a simplified version. The following is JavaScript Object Notation (JSON). JSON is how developers describe resources in code.

{
 "summary": string,
 "description": string,
 "location": string,
 "creator": {
    "id": string,
    "email": string,
    "displayName": string,
    "self": boolean
   },
 "organizer": {
    "id": string,
    "email": string,
    "displayName": string,
    "self": boolean
   },
 "start": {
    "date": date,
    "dateTime": datetime,
    "timeZone": string
   },
 "end": {
    "date": date,
    "dateTime": datetime,
    "timeZone": string
   }
}

But it’s also fairly human readable. You can see that an event has a summary (the event title), a description, details about who created it, the event organizer, a start date, and an end date.

For each field, the documentation also explains what type of data it expects to receive for that field. Most of these fields are strings, which just means a string of alphanumeric characters. Boolean fields can only have true or false as their value. This tells a developer how to describe a resource when they want to create or update it. It also tells the developer what they will receive when they fetch a resource.

For example, if a developer wants to create a new event, they’ll need to send a POST request to the Events endpoint with all of the required fields defined. They can send the defined fields in the body of their HTTP request.

The server receives the request, acts on it, and then responds with an HTTP response. The status line in the HTTP response tells the developer if the update was successful (e.g. 200 OK) or if there was an error (400 Bad Request). If the update was successful, it’s standard for the API to return the updated resource in the body of the responses, so that the developer can check that the update happened as expected.

So just as your browser sends HTTP requests to fetch websites, developers can send HTTP requests to fetch resources. They can also send requests to create, delete, and edit those resources.

Authentication: Can Anyone Send Requests to an Endpoint?

At this point, you might be wondering, can any developer send any type of request to any endpoint? The short answer is no.

Just like you have to log in with your username and password to access your Google Calendar, developers have to send credentials with their API requests. The credentials determine which resources the developer can access and which ones they can’t. Credentials are typically sent in the HTTP request headers.

When a server receives an API request, it first checks the credentials in the headers. The credentials determine if the requester has permission to fetch, update, create, or delete the resource it’s requesting. If the credentials grant the necessary permission, the server executes the request. If the credentials don’t grant the necessary permissions, then the server returns an error.

Now this is where things can get tricky. Sometimes when a developer accesses an API, they want to use their own credentials. For example, if the hair salon is using Stripe to collect payments, the developer can set up a Stripe account and they can use the credentials for their Stripe account in the API request. When the Stripe server responds to a payment request, it is acting on behalf of the hair salon account.

But sometimes the developer is writing code that needs to take action on behalf of the end user. When a customer books a haircut on the website, maybe the developer wants to add an option to add the calendar event to the customer’s calendar. But the developer doesn’t have credentials that allow it to access the customer’s calendar. The developer needs a way to send the customer’s credentials in the request.

There’s an authentication standard called OAuth that allows the developer to do exactly this.

We don’t need to get into the details of OAuth here. You just need to know that sometimes API requests use the credentials of the developer’s account and sometimes API requests use the credentials of the end user.

When a developer is using an API for the first time, one of the first things they need to do is to determine how they’ll authenticate. If they need to access the API using their own credentials, they need to determine how to obtain those credentials and what permissions they have access to. If they need to authenticate on behalf of an end user, then they need to learn how to obtain those credentials, typically through an OAuth implementation.

Phew, we did it. We covered enough of the basics so that next week, we can get into the common usability issues engineers encounter when using APIs.

We’ll look at issues with:

• Inaccurate, incomplete, or insufficient documentation
• Insufficient endpoint coverage
• Limited access to resources
• Confusing error codes
• Inadequate or confusing authentication options
• Sloppy implementation of REST

If you don’t want to miss next week’s article, be sure to subscribe below.

The post Understanding Web APIs: What They Are and How They Work appeared first on Product Talk.