Developing App for Assisted Communication

I am a software developer of many years experience. My son is autistic and learning to improve his communication through an AAC application. He was nonverbal when he was younger but has made great progress. I have written an open source app that uses Vector Embeddings to match my text to the icons (~5000) in his application. See more details about the app we have worked on here: https://github.com/RonanOD/OpenAAC

When there isn’t a great match with a given word to the set of tiles, I make a call to OpenAI’s DallE-3 to generate an image. Sometimes the image generated isn’t great. I would like to improve the process by training a custom model with the large set of icons and have this model generate a more appropriate icon, either within the OpenAAC app or as a service the app can call. Can anyone on this board recommend any resource/tutorials to help me set this up? I have worked with Python and have a fair understanding of ML concepts.

1 Like

Hello!

That’s a great and interesting project.

Do you already know how to use deep learning models/ jupyter/ matplotlib and all the good things or are you just looking for appropriate data to train your model on?

1 Like

Hi Daniel,
Thanks so much. We have lots of training data, about 5000 icons. The problem is when there isn’t a good match of word to icon and we want to generate a new one that closely matches the dataset. I just lack the hands-on knowledge of what deep learning to use (and how to make it available to a mobile app).

It should be no problem. There are many learning resources for AI image recognition and AI image generation.

If you want to be independent from third party AIs and create your own,
I can share some of the sources that helped me to get started in ML and that I’m still using in my daily work:


The practical side: On this channel you will find videos teaching you how to set up standard tools like Anaconda and Jupyter Lab, create a Python deep learning environment and how to train your first model with TensorFlow. Don’t mind the guys goofy appearance, he’s actually a great tutor!

The theory and for research: Two fantastic resources for getting into ML/DL. As you can imagine too much to learn in a weekend, but the well structured documentation will help you to progress in a reasonable time frame.

You can design the data set out of what you already have available, plus public icon collections like https://thenounproject.com/ to train your image generator.

Step by step from helping the model learn to connect text prompts with prefab images/ icons to generating its own, while improving its accuracy over time.

1 Like

That’s excellent! I have not encountered the noun project before. I will check those labs out to improve my background understanding.
Thanks again

Very nice. Besides helping your son, you might open a new chapter in your growth as a developer like it happened to me.

Once you learn the first ropes of AI/ML/DL, it’s hard to stop. You want to build better and more capable models, it’s like a kid in a candy shop moment.

1 Like

Great point. It is a very exciting field