Working with JSON Data in Python

Python
1

Hi. So I’m currently trying to create a Discord bot with discord.py that allows users to create events, enter a date for the event down to the minute(must be divisible by five), and select a role to ping for the event. Overall that sounds not that hard. But my usual strategy of working with .txt files won’t work, so I’m trying my hand at JSON. And that’s proving to be significantly harder. If I can figure out the baseline function, I should be able to get it connected to the Discord bot with ease.

I need Python to, every five minutes, check the events.json file to see if there is an event soon, or to clean out already executed events. Events will have the variables of “name”, “date”(in a yyyy/mm/dd/hh/mm format), “roles”(one or more roles to ping when executing the event; probably just allow for one role currently, then add support for multiple roles in the future), and “isExecuted”, which will be a boolean. If it is equal to true, then the event will be removed from the array. Here’s my code so far:

events.json

{
    "events": [
        {
            "name": "Pizza Party",
            "date": null,
            "roles": null,
            "isExecuted": false
        },

        {
            "name": "The Big Meeting",
            "date": null,
            "roles": null,
            "isExecuted": false
        },

        {
            "name": "The Small Meeting",
            "date": null,
            "roles": null,
            "isExecuted": true
        },

        {
            "name": "The Medium Meeting",
            "date": null,
            "roles": null,
            "isExecuted": false
        }
    ]
}

main.py

import datetime
import json
from threading import Timer

# Create currentTime object
now = datetime.datetime.now()
currentTime = f"{now.year} {now.month} {now.day} {now.hour} {now.minute}"
# To reference time objects(e.g. year, month, etc.), use now.<time format>, such as now.minute


def check_for_events():
    # To make life easier, the file will only be opened and closed only when the function check_for_events() is called/ran.
    # Open the JSON file
    with open("data.json", "r") as read_file:
        data = json.load(read_file)
        data_length = len(data)

    # Causes task to repeat every five minutes
    Timer(300, check_for_events).start()

    # The actual code.
    print("Checking event list...")

    for events in data['events']:
        x = events # When run, returns TypeError: list indices must be integers or slices, not dict
        if data['events'][x]['isExecuted']:
            print(data['events'][x]['name'] + ' has already been executed and will now be removed.')
            continue
    # Got stuck so haven't done any more code.

check_for_events()

I don’t need the file coded for me in a hand-holding way, but a push in the right direction would definitely help. Manipulating JSON with Python is a lot harder than it looks.

2

Have you looked how data looks after it’s read and when it’s iterated over?

for events in data['events']:
   print(events)

When you iterate over the data['events'] you should be able to use more directly what is assigned to the events variable, instead of using it in a data['events'][events]... way.

3

Output is as follows:

"/home/agent/Python Projects/jsonTest/venv/bin/python" "/home/agent/Python Projects/jsonTest/main.py"
Checking event list...
{'name': 'Pizza Party', 'date': None, 'roles': None, 'isExecuted': False}
{'name': 'The Big Meeting', 'date': None, 'roles': None, 'isExecuted': False}
{'name': 'The Small Meeting', 'date': None, 'roles': None, 'isExecuted': True}
{'name': 'The Medium Meeting', 'date': None, 'roles': None, 'isExecuted': False}

How do I specifically select a variable of an item in an array?

4

How would you do that from a dictionary, for example?
dictionary = {'one': 1, 'two': 2}

5

I never worked with dictionaries before, but sure. How do I access and modify the data in a dictionary?

6

It is similar to how you tried in the code. For example with dictionary, to get 2 one would use dictionary['two']

7

The JSON specification does not specify a format for exchanging dates which is why there are so many different ways to do it. JSON does not know anything about dates. What .NET does is a non-standard hack/extension. The problem with JSON date and really JavaScript in general – is that there’s no equivalent literal representation for dates. In JavaScript following Date constructor straight away converts the milliseconds since 1970 to Date as follows:

var jsonDate = new Date(1297246301973);

Then let’s convert it to js format:

var date = new Date(parseInt(jsonDate.substr(6)));

The substr() function takes out the /Date( part, and the parseInt() function gets the integer and ignores the )/ at the end. The resulting number is passed into the Date constructor .

For ISO-8601 formatted JSON dates, just pass the string into the Date constructor:

var date = new Date(jsonDate);

8

I don’t care about date formatting at all. I simply need to manage certain amounts of data using JSON and Python. All JSON does is store the date; Python does everything else. I simply need to create a script that every five minutes it checks the events listed in a JSON file and the string which specifies the exact date and time they are to occur. If that string is equal to the current time, it will execute a certain bit of code. This is all for a Discord bot made in Discord.py. My problem is that I can’t get the stupid code to read and write to a JSON file. I have no idea why.

9

I’m not sure if you’re still stuck at the same point as in your original post - if not just ignore me!

Anyway, using your data, this code:

import json

with open("data.json") as f:
    data = json.load(f)
    
for event in data['events']:
    if event['isExecuted']:
        print(event['name'] + ' has already been executed and will now be removed.')

prints …

The Small Meeting has already been executed and will now be removed.

I think that’s more or less what you were looking for?

In Python your data is a dictionary with one key , ‘events’.

The ‘events’ key points to a list of dictionaries, with each dictionary corresponding to one event.

So data['events'][0] will get you the first event and data['events'][0]['name] will get you the name of that event.

Does that help?

10

I saw this thread pop up earlier but didn’t reply because it was old. I see you haven’t figured out what to do so I’d like to chime in a little.

Study!

First, please read generally how to use dictionaries in python. My recommendation is before you try to change your code, study first about dictionaries. Otherwise you will end up using code somehow that you don’t understand. If you need to change it later you will quickly have a problem. Dictionaries are very important and used everywhere.

To get you started, read the python docs tutorial section on dictionaries and this part of the python docs have the finer details.

Study about the json library

Here are the docs

What happens when you load json data? What happens when you save a python dictionary using json.dump()?

When you do json.load() the json library will convert JSON into a python dictionary for you. There are a lot of differences between those things, but one thing that is quite similar is how to access the values that are stored. This is why I said, go study about dictionaries.

For your purpose, you need to both read and write json. You want to save the state of an event (if it was executed) which will require at least two things:

  1. Modifying the dictionary
  2. Saving the dictionary (which you want in JSON format)

I can see some potential issues you might run into with using your data like this. Reading and writing one file simultaneously from different programs/scripts could give you issues.

It may be better to write your script “history” (log) to a separate json file and then check the log every time to see if it has already been processed.

I.e. “Look an event to process! Okay, I’ll check if I already did this one in my log. Oh, I did. NEXT”

Other thoughts

x = events # When run, returns TypeError: list indices must be integers or slices, not dict

What is data['events']? If you look carefully at it, you will see it is a list. Specifically, a list of dictionaries. The reason for this error is that you are trying to access a value from the dictionary by supplying a dictionary where it expects an integer (the index of the list).

if data['events'][x]['isExecuted']:

If you are doing for event in data["events"] then you can simply access that part of your dictionary like so:

for event in data["events"]:
  # now event is one dictionary from the list of dictionaries
  # a dictionary has keys, so you can for example:
  print(event["name"])

I think that’s all for now. Let’s see what you can come up with.

11

I simply need to create a script that every five minutes it checks the events listed in a JSON file and the string which specifies the exact date and time they are to occur. If that string is equal to the current time, it will execute a certain bit of code. This is all for a Discord bot made in Discord.py. My problem is that I can’t get the stupid code to read and write to a JSON file. I have no idea why.

So for every event:

  • is the event date and time equal to the current date and time?
  • if it is, then do something with it
  • if it isn’t, skip

In order to compare dates and times you will need to study more about pythons datetime library.

With the below code, do you think a == b will evaluate to True or False?

import datetime
a = datetime.datetime.now()
b = datetime.datetime.now()
a == b
12

Here’s the error I’ve been getting. Sorry, I’m trying to get my bearings. I haven’t touch this project in a long time.

Ignoring exception in command addevent:
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/ext/commands/core.py", line 85, in wrapped
    ret = await coro(*args, **kwargs)
  File "/home/runner/AetheriaDiscordBot/cogs/event_cog.py", line 17, in addevent
    data = json.load(file)
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/ext/commands/bot.py", line 939, in invoke
    await ctx.command.invoke(ctx)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/ext/commands/core.py", line 863, in invoke
    await injected(*ctx.args, **ctx.kwargs)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/discord/ext/commands/core.py", line 94, in wrapped
    raise CommandInvokeError(exc) from exc
discord.ext.commands.errors.CommandInvokeError: Command raised an exception: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

And here’s my current code for the Discord bot cog:

## THIS COG IS NOT YET FUNCTIONING, AND WILL NOT BE INITIALIZED.

import discord # Currently unused. Used in sending Discord messages.
from discord.ext import commands
import json

def split(word):
      return [char for char in word]

class Events(commands.Cog, description='Create and manage events.'):
    def __init__(self, bot):
        self.bot = bot

    @commands.command(name='addevent', help='Add an event to the event list. Usage: `addevent <name of event> <time to execute, format dd:hh:mm>')
    async def addevent(self, ctx, name: str, time: str):
      with open('events.json', "r+") as file:
        data = json.load(file)
      time1 = split(time) 
      day = time1[0] + time1[1]
      hour = time1[3] + time1[4]
      minute = time1[6] + time1[7]
      entry = '{"name": '+ name +', "time": {"day": ' + day + ',"hour": ' + hour + ',"minute": '+ minute +',"executed": false}'
      data['events'].append(entry)
      json.dump(data, file)
      json.close(file)

def setup(bot):
   bot.add_cog(Events(bot))
   print('Events cog successfully loaded.')

I’ve been meaning to add some better commenting, but sadly I haven’t done this yet, so sorry that it’s hard to read.

Also, the datetime example you gave would return False, because Datetime prints the current time down to the millisecond.

13

Okay, I’m not entirely sure how Discord bots work but my basic guess is that someone on Discord could use this bot to add an event to an events.json. If you’re getting user input, you should be careful to check that it is valid input.

  1. The exception you get from json.decoder is because of an empty file.
  • you might think about putting your data related code into separate functions (i.e. get_data(), write_data(), and probably an add_event() which would utilize write_data().)
  • re: json.load() exception based on an empty file. How can you handle that gracefully in your app so that it doesn’t bail?
  1. Python has built-in modules for parsing date and time strings
  • to point you in the right direction: look up how to use strptime() from python’s datetime library
  • You are going to use these dates and times in comparisons for posting the events when they are going to occur? If you don’t validate the input, you will run into problems doing this
  • as it is right now, a user would be allowed to submit an event with any character for the time.
  • Why only the day? Why not the year and month?

  1. data['events'].append(entry) would throw an exception if data contains nothing.

  2. My suspicion is that you will run into a problem with user input if there are a lot of spaces in the event name. I’m just not sure how the discord.ext commands work.

  3. You do not need to make a JSON string manually

  • this entire line starting with entry = '{"name": '+ name +',.......
  • json.load() will take JSON and turn it into a python dictionary
  • json.dump() will take a python dictionary and turn it into a JSON string (and write it to file)
14

Alrighty, so I got the script to actually take input and dump to a JSON file. Here’s the new code for the event cog:

## THIS COG IS NOT YET FUNCTIONING.

import discord # Currently unused. Used in sending Discord messages.
from discord.ext import commands
import json

class Events(commands.Cog, description='Create and manage events.'):
    def __init__(self, bot):
        self.bot = bot

    @commands.command(
      name='addevent',
      help='Add an event to the event list. Usage: `addevent <name of event> <time to execute, format YYYY-MM-DD-HH:MM>'
    )
    async def addevent(self, ctx, name: str, time: str):
      year = time[0:4]
      day = time[5:7]
      hour = time[8:10]
      minute = time[14:]
      
      entry = {
              'name': name,
               'time':
               {
                 'year': year, 'day': day, 'hour': hour, 'minute': minute
               },
               'executed': False
              }
      with open('events.json', "w") as file:
        json.dump(entry, file)
      
      #json.close(file)

def setup(bot):
   bot.add_cog(Events(bot))
   print('Events cog successfully loaded.')

I know all the things you told me about using datetime, but I’m still working on that. I’ll probably use that when I go to make the script actually see if any events need executing. For now I need help on something else. The code, when run, actually completely overwrites the JSON file, deleting any other events in the file. How would I go about doing this using the append function? I tried using append before, but Python doesn’t like doing that with dictionaries.

15

I figured out that I can append to the file instead of overwriting it using the a operator when opening the file. But I need each dictionary to be accessible when the events need to be deleted. For example: if executed is updated to true, eventually the script needs to remove it. How would I remove that entire dictionary? Or another thing: how would I make it so that if an event is executed, the flag on that specific event/dictionary would be updated to true?

16

Okay, what I don’t see in your code is anywhere that you read the data from file.

I’d recommend creating a function:

  1. It will read from your JSON data file and return it as a dictionary.
  • since json.load() will give an error if there is an empty file, wrap this in a try/except block.
  1. If there is no data return an empty dictionary.
  • That empty dictionary could also have “entries” key with an empty list.
  • i.e. return {"entries": []}

I don’t want to write it out for you, but a skeleton…

def get_json_data(filename):
with open(filename, "r") as file:
  try:
    # try to load json from file and return it
    # remember: this will return a python dictionary
  except:
    # if it doesn't work... return an empty dictionary

data = get_json_data()   # this data is now what you need to use, append to, and later write to file.

Adding entries

Now that you have your data dictionary, you can indeed append entries to it.

data["entries"].append(entry)

Write it to file

After you are done, write it to file. For this I suggest creating a separate function. It helps keep code clean and manageable.

def write_json_data(data, filename):
  ...

# somewhere in your process
entry = {'name' .... etc}
data["entries"].append(entry)
write_json_data(data, filename)

Bonus:

We don’t really want to toss a filename around like that. What if the filename needs to change? Then we need to go and change every place where it is written. Modify functions that I wrote so that instead of using filename as an argument, the internals of the functions use a global variable for the filename.

17

Ok, I’ve almost got it working. But here’s a problem I need help with:

In the JSON file, it looks like this:

{"name": "Pizza Party", "time": {"year": "2022", "day": "05", "hour": "21", "minute": "00"}, "executed": false}{"name": "The Big Meeting", "time": {"year": "2022", "day": "05", "hour": "21", "minute": "00"}, "executed": false}

I simply need a way to loop through dictionaries, not values in the dictionaries. Like this:

for x in data:
    if x['executed'] == True:
        del x

I’ve tried this, and it doesn’t work. How do I go about deleting an entire dictionary imported from a JSON file?

P.S:
Here’s a bit of a function I wrote out that you requested. The last bit makes it non-functioning, obviously, but still.

def check_events():
  try:
    file = open("events.json", "r+")
    data = json.load(file)
  except:
    log("warning", "events.json is empty. Assigning data to empty dictionary.")
    data = {"entries": []}

    for x in data:
      if x['executed'] == True:
        del x

(Log is a creation of my own. Here’s the code, if you ever wanna use it.)

def log(type, message):
  now = datetime.datetime.now()
  
  types = ['info', 'warning', 'error']
  if type == 'inf' or type == 'in':
    type = 'info'
  if type == 'warn':
    type = 'warning'
  if type == 'err':
    type = 'error'
  if type not in types:
    type = "unknown"
  
  print(f"[{now.year}/{now.month}/{now.day} {now.hour}:{now.minute}] <{type.upper()}> {message}")
18

be aware that discord.py is not being maintained anymore and the API version used is deprecated, the only discord library being maintained is discord.js

EDIT: ok, it’s not only discord.js, the others are in languages I don’t use

19

When did that happen?

And how can you just “drop support” for not only countless Discord bots but an entire programming language? And there’s the fact that Python is a very popular language. You’d think that Python would be the actively maintained API, not Javascript. Most people hate Javascript.

20

Discord.py is not by Discord itself, it is an opensource wrapper for the API

Here the goodbye from the main maintainer