Trying to figure out how to do API

Python
#1

New at Python. I am trying to grab the data below. I import requests, seem to set things up correctly. When I run the following:

import requests

url = "https://api.fda.gov/drug/ndc.json?search=finished:true&limit=1"
response = requests.get(url)
data = response.json()
data

I get the following:

{‘meta’: {‘disclaimer’: ‘Do not rely on openFDA to make decisions regarding medical care. While we make every effort to ensure that data is accurate, you should assume all results are unvalidated. We may limit or otherwise restrict your access to the API in line with our Terms of Service.’, ‘terms’: ‘https://open.fda.gov/terms/’, ‘license’: ‘https://open.fda.gov/license/’, ‘last_updated’: ‘2022-04-29’, ‘results’: {‘skip’: 0, ‘limit’: 1, ‘total’: 112692}}, ‘results’: [{‘product_ndc’: ‘72572-740’, ‘generic_name’: ‘SODIUM BICARBONATE’, ‘labeler_name’: ‘Civica’, ‘brand_name’: ‘Sodium Bicarbonate’, ‘active_ingredients’: [{‘name’: ‘SODIUM BICARBONATE’, ‘strength’: ‘84 mg/mL’}], ‘finished’: True, ‘packaging’: [{‘package_ndc’: ‘72572-740-20’, ‘description’: ‘20 VIAL in 1 CARTON (72572-740-20) > 50 mL in 1 VIAL’, ‘marketing_start_date’: ‘20200301’, ‘sample’: False}], ‘listing_expiration_date’: ‘20221231’, ‘openfda’: {‘manufacturer_name’: [‘Civica’], ‘rxcui’: [‘1868486’], ‘spl_set_id’: [‘a10fdfbc-94a0-49ab-b26c-de1df042bc89’], ‘is_original_packager’: [True], ‘unii’: [‘8MDF5V39QO’]}, ‘marketing_category’: ‘ANDA’, ‘dosage_form’: ‘INJECTION, SOLUTION’, ‘spl_id’: ‘a10fdfbc-94a0-49ab-b26c-de1df042bc89’, ‘product_type’: ‘HUMAN PRESCRIPTION DRUG’, ‘route’: [‘INTRAVENOUS’], ‘marketing_start_date’: ‘20190801’, ‘product_id’: ‘72572-740_a10fdfbc-94a0-49ab-b26c-de1df042bc89’, ‘application_number’: ‘ANDA211091’, ‘brand_name_base’: ‘Sodium Bicarbonate’}]}

I am then trying to get all the data, I get an error " KeyError : ‘results’ " and I am not understanding why even after spending a couple days working on it. Here is the code:

import requests

ndc = []
offset = 0
limit = 100

while True:
    print("-----")
    url = f"https://api.fda.gov/drug/ndc.json?offset={offset}&limit={limit}"
    print("Requesting", url)
    response = requests.get(url)
    data = response.json()

    # Any NDCs left?
    if len(data['results']) == 0:
        # if not, exit loop
        break
    
    # if we did find NDCs, add them 
    # to our list and then move on to the 
    # next offset
    ndc.extend(data['results'])
    
    offset = offset + 100

I then get the error I mentioned above: KeyError : ‘results’

Any assistance is greatly appreciated. Thank you

#2

When I log out data I get this.

Requesting https://api.fda.gov/drug/ndc.json?offset=0&limit=100
{'error': {'code': 'BAD_REQUEST', 'message': 'Invalid parameter: offset'}}
#3

As lasjorg said “offset” is a bad parameter in the API query.

One other thing you can do is check for the presence of a key in a dictionary with in:

>>> a = {"test": 1}
>>> "test" in a.keys()
True
>>> "hi" in a.keys()
False

This is a way to check for something in a dictionary that will not throw a key error at you. If you use len(dictionary["key"]) and that key is not there, you will get a key error.

Since there is an “error” key when there is an error fetching data, it may be useful and more understandable in your code to test for that condition and break if the condition is true. I.e. Is there an "error" key in this data? Also is the length of keys only one? Then something is wrong, let's quit!.

Be careful about using an endless while loop for fetching data. You will have a flood if it’s successful!

A neat thing with requests is that you can set up a dictionary for parameters and make the request that way (rather than doing string formatting for your url). result = requests.get(url, params=params). I’ll leave that to you to check out.

#4

Apparently, I need to use ‘skip’, instead of ‘offset’. Now, when I run the following code:

import requests

ndc = []
skip = 0
limit = 100


while True:
    print("-----")
    url = f"https://api.fda.gov/drug/ndc.json?skip={offset}&limit={limit}"
    print("Requesting", url)
    response = requests.get(url)
    data = response.json()

    # Any NDCs left?
    if len(data['results']) == 0:
        # if not, exit loop
        break
    
    # if we did find NDCs, add them 
    # to our list and then move on to the 
    # next offset
    ndc.extend(data['results'])
    
    skip = skip + 100

and the results I am getting are showing the skip staying at 0 and not increasing.

Shouldn’t the skip count up by 100? It is staying at 0.

#5

I fixed that. Let me check if everything is working. My goal is to then export the data, which should be in ndc, to a csv.

#6

OK - everything seems to be working fine with the code until it gets to skip = 25100 then it throws an error: KeyError: ‘results’

the code is:

import requests

ndc = []
skip = 0
limit = 100


while True:
    print("-----")
    url = f"https://api.fda.gov/drug/ndc.json?skip={skip}&limit={limit}"
    print("Requesting", url)
    response = requests.get(url)
    data = response.json()

    # Any NDCs left?
    if len(data['results']) == 0:
        # if not, exit loop
        break
    
    # if we did find NDCs, add them 
    # to our list and then move on to the 
    # next offset
    ndc.extend(data['results'])
    
    skip = skip + 100
#7

Read again what I wrote above about checking for something in dictionary keys

#8

Will do - I’ll reach out with any questions.