2.2 Application Programming Interfaces

How Do APIs Work?

Step 1

The process begins when some form of client software (e.g., web browser, application, or just code like python) which may exist on any hardware type (e.g., desktop, mobile device, or website) sends a URL request to an API (a.k.a. ) that resides on the WWW. The request is typically in the form of a GET, POST, PUT, or DELETE which are called "request methods." At a minimum, all requests must include a URL to a valid endpoint. At a maximum, the request may also include a , , and .

  • URL: the location of the intended web service endpoint

  • Data or payload:

    data to be processed by a web service API endpoint

  • Headers: meta-data associated with the API request that directs the endpoint how the data are formatted, authorized, and processing instructions

  • Parameters: key/value pairs included in the URL typically beginning with a "?" symbol after the address and separated by "&" symbol between pairs. For example: https://web.address.com/webservice/endpoint?key1=value1&key2=value2&key3=value3

Step 2

The web service API then processes the request which may include interacting with other services like a database to get any required data. It may simply include some sort of data processing depending on the request type (e.g., GET, POST, PUT, DELETE); for example, as you will learn later in this book, it may involve sending customer data that is used to get a prediction back like whether or not this customer would like to purchase a particular product. It may also include interacting with another external web service endpoint that is completely unseen to the original client.

Step 3

When it has finished its processing, the API will then package up the results in a specific type of format, usually XML or JSON. Extensible Markup Language (XML) used to be very common, but Javascript Object Notation (JSON) is now the most common format because of its compactness. As might the request, the response will typically include headers along with the JSON- or XML-formatted data as well as some repsonse code indicating whether the request was successful or why it was not.

APIs have become very popular in software development because they allow companies to reuse core components of the software that they create. It is common for a company to offer a web version, one or more mobile versions (e.g., iOS, Android), and one or more desktop versions (e.g., Windows, Mac) of their software. Rather than having to duplicate code for different versions of the software, APIs allow companies to have a single endpoint that can be reused by multiple versions of the software. While the tools and approaches that you learned for web scraping weren’t difficult, calling a web page and then scraping the data takes more effort, consumes more resources, and is more fragile than giving individuals direct access to an API.

API Methods

As mentioned above, the four most commonly request methods are GET, POST, PUT, and DELETE. The table summarizes each method.

Table 2.1
Leveraging a Web API
Method Method Example
GET GET requests retrieve data from an endpoint and return the data in a response. A GET request from Twitter could return a tweet as a JSON document including the tweet’s text. Date, user, and other information.
POST The POST method submits information from your request to the endpoint. It returns a response stating whether or not the information you entered was successfully submitted. When you submit an image to a classifier web service, you might get back a list of entities found within the image like {'people':5, 'full faces':4, 'smiling':3, 'neutral':1, 'frowning':0}
PUT Put is similar to POST in that you are sending information, but PUT is used to update already existing information. When you change your address or phone number on an online account, the PUT method is used because your account information already exists.
DELETE DELETE is self-explanatory. An application or service receiving a DELETE request from an endpoint will then try to delete an entry in a database based on the information provided. Like POST and PUT, DELETE will return a response stating whether or not the request was successfully completed.

Data Formats

As mentioned above, JSON and XML are the two most common formats for passing data as a request or receiving data as a result. Both formats are considered “self-describing” as they are human-readable yet have a structure to them. Both of these formats can be easily parsed by a lot of different programming languages, including Python. Similar to HTML, JSON and XML are hierarchical and can have values inside of values.

XML is an earlier format and tends to be a little bit longer when returning the same results, whereas JSON is also quicker to read and write and you can also easily use arrays. As such, JSON has become a preferred format for many APIs.

The following is a JSON example followed by a comparable XML example.

JSON Example

            {"employees":[{"firstName":"Suzy", "lastName":"Smith" },
                         {"firstName":"Bob", "lastName":"Jones" },
                         {"firstName":"Abigail", "lastName":"Adams" }]}

XML Example

            <employees>
              <employee>
                <firstName>Suzy</firstName>
                <lastName>Smith</lastName>
              </employee>
              <employee>
                <firstName>Bob</firstName>
                <lastName>Jones</lastName>
              </employee>
              <employee>
                <firstName>Abigail</firstName>
                <lastName>Adams</lastName>
              </employee>
            </employees>

Authentication Methods

Many APIs require you to be authenticated in order to make requests and receive results. While there are several different approaches that an API might use to authenticate incoming requests, the following four are perhaps the most common: no authentication, basic authentication, API Keys, and OAuth tokens.

  • Open APIs/No Authentication: Not all APIs require you to be an authenticated user; however, these APIs are becoming more scarce by the day. As you might imagine, calls to APIs require compute resources, and someone has to pay for those resources. Open APIs, especially those with valuable data, are often overused and thus are no longer common.

  • Basic Authentication: This is perhaps the most straightforward of any of the authentication methods; however, it is rarely recommended because it is inherently insecure. With this method of authentication, the sender includes a username and password in the request header for each request that is sent. This method is simple as it doesn’t require cookies, session IDs, or other tracking solutions because it uses the HTTP header to pass the data.

  • API Keys: Rather than passing a username and password in the header of each request, API keys provide a way to increase security while locking down who can access a specific API. With this type of authentication, a developer will request a unique key for the application that they are building that needs access to an API. Then, this unique key is passed to the API each time a request is made. The API can track usage and limit the number of calls being made using this key as it uniquely identifies each application the developer has created. Most free APIs will limit how many calls you can make in a day and how frequently you can make those calls. Those that pay a fee to access the API will often have many more calls to the API and often can make those calls as quickly as they would like. This approach has the further benefit that if a key is compromised, the old key can be decommissioned and a new one can be generated and used.

  • OAuth: Under this type of authentication, the software that calls a given API will first retrieve an access token directly from an authorization server. This authorization service will grant access, and when specific resources are requested, it will also handback an access token for that specific resource. With this access token, the client software can then pass that access token to the API similarly to how an API key is used. For cases where you want to see a specific user’s personalized information for a service (e.g., Facebook feed, Instagram stories), then this is the typical approach for doing that.

Public APIs

As described above, APIs can be public or private. Fortunately, there are numerous APIs that are public, although you still might have to register to access the API. The following is a categorized list of publicly accessible APIs that come from the Public APIs GitHub project (https://github.com/public-apis/public-apis):

Take a couple of minutes to explore several APIs in a few of the above categories before seeing how we can use them in the next section.

Previous Next