ChatGPT: Testing its Potential in App Development

An undeniable fact: with its appearance, Chat GPT made lots of developers nervous. Clearly, it is not the first innovative real-life chatbot use case. However, unlike its competitors, OpenAI’s product proposes more than regular texting or already familiar “Question-Answer” or FAQ options. Instead, it would be fair to admit, that it is one of the most elevated and sufficient examples of such software. Previously, we have already shown and analyzed the capabilities of this AI for textual content creation. Today, we would like to pay more attention to one of its most famous features: the ability to write full-fledged code samples.

This feature can be easily considered revolutionary due to the fact, that there are no direct AI rivals in this field. To rephrase it, ChatGPT is probably one of the first well-known artificial intelligence chatbots with natural language processing abilities, capable of coding functionality without human involvement, writing human-like texts, and providing many other valuable tools like language translation.

Actually, OpenAI truly surprised the world with the fact, that numerous proposed features, can be easily performed by the chatbot itself. The most astonishing fact, though, is that in most cases, the results are working as supposed. However, it is also correct to say, that not each output will do so.

To better understand how OpenAI turns written language into code, and whether it will set trends in app development, we decided to test its capabilities and figure out the possible benefit of OpenAI to software developers, will it replace them, or can become a powerful tool in their hands?

The Expectations

Frankly speaking, we tested it before everyone started writing essays and giving online feedback on the OpenAI software. Thus, we had low expectations. The revolutionary nature of the app lies in the fact, that probably no one could even imagine the real outcome. To add some more, its appearance truly provoked debates on the limitations and ethical considerations of using ChatGPT or similar software in various industries.

For instance, we truly believed, that the maximum it can do is to generate decent yet boilerplate code samples. However, it turned out, that ChatGPT is much more interesting and surprising, than it may seem at first glance. As it emerged, it is capable of not only giving the requested code samples but understanding and even leaving related comments, which will be very helpful to better understand the proposed code blocks. Apart from it, this AI also knows how to extend the code if requested, and reflect it if needed.

So, before going deeper into the testing itself, we can define the strengths and weaknesses of ChatGPT, when it comes to coding.

Advantages of ChatGPT

Apart from the previously mentioned stunning capabilities of understanding the code, ChatGPT has other benefits as well. For example, this exact AI gives variable answers, i.e. it is not a “1 input = 1 output”. Instead, it can give multiple solutions for your specific request, knowing multiple ways how to perform the task, or giving you more than one piece of advice on how to deal with some issues that may appear, which is, once again, a very surprising result, compared to the potential rivals.

Also, ChatGPT has an access to vast information, related to various programming languages, libraries, and frameworks. So, in most cases, it can easily find the needed information.

Software Awareness

To give an example, we asked him to write a few functions that can interact with Zoho CRM API in Python. To do so, we explained in detail what we needed: a way to authenticate, insert, or update records and send attachments to them. It was able to search through Zoho CRM API documentation, find the necessary endpoints and then write the python functions which did exactly what we asked. It explained what packages it used, and the functions’ logic as well.

Learning Abilities

Then we started testing its abilities to analyze and extend the code. ChatGPT was able to reorganize those functions into class methods. The AI answered and implemented more complex questions like handling the access tokens expiration and API specific requests limit. However, the answer it gave was not optimal due to the fact, that it would exhaust the limit very quickly. Yet, after we pointed that out, the chatbot quickly understood the problem and proposed a few solutions on how to deal with it. After we chose one of them, OpenAI managed to successfully apply it to the code.

Code Understanding

Afterward, it was still able to extend the code further as much as integrate the API class with other frameworks, Django in particular. It mapped and used model data to send to the API via methods it wrote. It also understood such concepts as code style, coupling/cohesion, and improv/refactor code, based on these methods, which was a pleasant bonus. For example, it suggested adding error handling to process all the API-specific errors. Then ChatGPT refactored and generalized it when we asked. With enough details and indications provided, AI can refactor the code very well. It was also able to generate a couple of simple test cases for the project as well. So, with enough details and indications, OpenAI’s product can refactor code very well.

Drawbacks of OpenAI

First things first, this artificial intelligence often gives incorrect or even wrong answers to various requests. Apart from it, when it comes to coding, ChatGPT is unable to implement complex functionality and generate code samples for it. So, if your request is complicated, most likely, the chatbot will propose templates or basic code structures, which you can extend by yourself. Moreover, even if AI generates code for your request, it can “forget” some of the requirements.

For example, when ChatGPT created our API class, it was generating the access token on every request, although it was already aware of the API refresh and request limits. Chatbot fixed it eventually, but we needed to point it out to him.

Also, sometimes AI proposed solutions, which do not work at all. For instance, we asked if we can disable a particular workflow rule triggers per Zoho API insert/update record requests by its name. It suggested that we should use a non-existent endpoint to find out the rule name and send a parameter in the requests, which also was not in the official docs.

Although, ChatGPT insisted that we should read the docs to find out more about the issue. When we pointed out the error it gave me another solution which was not possible as well. It advised us to read the docs to find out more about the problem, but when we asked for the URLs to the docs, it gave us non-existent URLs again.

The mistakes it makes sometimes are not as crucial if you have some experience in software development. However, in case you are a newcomer to the industry, you may struggle with some issues. For instance, recently, software engineers started to use ChatGPT as an alternative to Google.

The main advantage of this software is that it is much faster. So, a single request will get as fast and as precise as possible. However, on the reverse side, ChatGPT is not connected to the Internet, so can’t give you an up-to-date answer. Additionally, the database, on which it is based, is very limited and is approximately 570GB. In other words, the accessible data it operates has its own boundaries as well.

Both the foregoing facts can be considered reasons for not blindly trusting the responses of the chatbot and spending time checking if they are correct.

The Role of ChatGPT in App Development

Actually, despite all the pros and cons, it is already possible to make predictions on what is the potential role of chatGPT in App Development. Looking ahead, the inability to deal with complex functionality makes it impossible to use it as a replacement for a full-fledged developer. During our testing, we found out that despite the wide range of proposed features and overall fine results it has shown, chatGPT can’t be used for app development without the assistance of the developer.

Human Interaction

Frankly, no matter how difficult and complex the application is, chatGPT still requires direct interactions with the development specialist. However, truth to be told, it can significantly boost the productivity of a software engineer and increase the pace of development. For example, chatGPT is capable of giving working code samples, as well as advising the specialists during the development, proposing various extra solutions, suited for the specific task. This AI understands the overall logic of the code and knows how to work with it. Moreover, it is aware of the basic documentation and knows some possible issues, which may appear during the development of features or functionality implementation.

Nonetheless, these hints should be tested before their implementation due to the fact, that some of the data, it uses, is outdated. Once, we asked it to write some simple scripts to merge PDF files, ChatGPT gave us a deprecated code, which was not working:

import PyPDF2

# Open the first PDF file
with open('file1.pdf', 'rb') as file1:
pdf1 = PyPDF2.PdfFileReader(file1)

# Open the second PDF file
with open('file2.pdf', 'rb') as file2:
pdf2 = PyPDF2.PdfFileReader(file2)

# Create a new PDF file to write the merged content
with open('merged_file.pdf', 'wb') as outfile:
pdf_writer = PyPDF2.PdfFileWriter()

# Add the pages of the first PDF file
for page in range(pdf1.getNumPages()):
pdf_writer.addPage(pdf1.getPage(page))

# Add the pages of the second PDF file
for page in range(pdf2.getNumPages()):
pdf_writer.addPage(pdf2.getPage(page))

# Write the merged content to the new PDF file
pdf_writer.write(outfile)

After we pointed out, that the code sample is not working, the bot proposed another solution, which worked just fine.

URL Issues

Yet, later another mistake appeared. When we were testing its awareness of the Zoho API, at one moment chatbot suggested, that we are using an unexisting endpoint and stated, that it can’t such a case or its solution in the official documentation, so proposed to do it ourselves. When we asked for a URL for the documentation, it gave us an unworking link. After we told, it that the URL is not working, it gave us another invalid link. After we pointed out that the second hyperlink does not work, the bot finally apologized for misleading us.

Conclusion

Eventually, it seems that ChatGPT is a truly powerful instrument, which can be used for countless purposes. However, in any case, it is worth testing the final results, due to the fact, that the chatbot is still very limited. To make it simple, OpenAI is currently setting trends in app development. Yet, some people misunderstood them truly. In fact, to use the full potential o this technology, you have to consider it as an additional tool, not a replacement or a way how to automate the process of app development. Maybe, someday the roles will switch and the development specialists will be the ones, who help the AI to develop software projects. Yet, for now, the situation is the opposite, so the OpenAI to software developers is just a pleasant bonus, not a threat.

Final Thoughts

Using Artificial Intelligence is among the main trends in app development. Even if its impact on the industry is unnoticeable at the moment, we are sure, that in the nearest future, such software solutions as ChatGPT will become a must-have option in every tech stack.

Also, it is hard to predict the real possible potential of this specific technology due to numerous facts. For example, there are no analogs for now. OpenAI’s product itself has conceived built-in limitations as well. Therefore, it is hard to tell for sure what are the odds and prognosis of the true potential of the technology. Nevertheless, it is possible to define how to use its full potential at the moment.

If you are looking for more case studies and analytics on ChatGPT, you are welcome to our Insights section.