AILab Howest

Starting with the tools. In general, there are 2 main categories that could be perceived.

• AI Code Editors
• AI Development platforms

For the sake of comparison I have tried to create, and you wouldn’t guess it, a Todo app in all of them. There are never enough todo apps

There are some things I do want to make clear beforehand. I am a full stack developer with 5 Years of experience, my focus is more on the deployment and backend of POC (proof of concepts), but I have used multiple flavours of JS frameworks in previous projects. I am no authority on Prompting as for programming seeing I work for a research division of a university where most of our projects are until the POC stage.

As for what the created app should contain, I have created some baselines.

• Simple minimal frontend application
  • Showing of date, time and weather
• Only showing task for today
  • Place older tasks in archive and delete after one week
• For code editors a separate FAST API

AI Code editors

The first are the AI Code editors. These are enhanced coding environments where AI Models are helping with certain tasks. They mostly have developers in mind that want to leverage the newfound capabilities of large language models. Between them there are still some differences, some editors have Agentic approaches, others have just and advanced autocomplete.

Over the past months all of these editors have been implementing similar functionalities. This makes comparing them really a more ‘personal’ thing. A Code editor can be a blunt tool but could also be sharpened by looking into all the keyboard shortcuts and advanced functionalities they offer. By spending time and enhancing these to your preference you could surely improve your handling with all of them.

For this blogpost I have chosen the following 3 AI editors. I am sure there are others more niche and newer code editors like Zed or some plugin on Vim or JetBrains suite. I have selected these 3 seeing they are all based on the same User interface.

• Github Copilot
• Cursor
• Windsurf

For people not interested in the whole process here is the TL;DR from my personal experience

My general approach with all of them is to start with the agentic approach/ help with setting up the project following my specifications* next up I am looking through the code, reorganising where needs to be and testing locally. Next up I would like to generate some Tests to see how the custom agents handle this. And as last test I would change a text field to also be able to have a different maximum length, this would also need to be checked in the test. As last step I want to containerise my applications for easy deploy, here I will check if it uses the correct references and how the general codebase context is regarded.

As most do know, the model being used also makes a big difference. The fact of the matter is that this all will probably improve with newer and greater versions. For uniformity all of them were using Claude 3.7 sonnet.

They all have capabilities to have custom MCP servers (If you want to know more about this, read our previous article on it) but these will not be used. https://ailab.howest.be/news/mcp

You could opt for a Low AI workflow or a High AI workflow. For this article I opted for a High AI workflow to really see how this workflow feels and what could be achieved with it. I will go through one process fully and for the others I will stat the notable differences. This to spare you of reading everything all over again.

Github Copilot

Link to tool in database

For transparency, GitHub copilot is the Go to AI Aide that I personally use. Why? Simple, it comes in the github education package, it is directly implemented in my current Vscode environment and I can use it as a ChatGPT window in my editor. It can also do more of course!

Starting by using the agent in copilot to start the project using the project prompt.

By using the agent approach you van follow the steps and see what the AI is executing. This process is not perfect. If you don’t verify It is bound to run into errors. The agent will detect this and try to solve it. Keep in mind that this could also make everything a lot more complex if you are not verifying what it is doing. The goal of using a code editor is to keep this control onto you code.

Nextup you can go over the changes copilot made as a Git Dif, or a changelog of the previous files.

When copilot agent has been iterating a while, it will state so. You could then further specify what you want or just let it continue.

When creating the backend I had spotted it was creating a .venv and a requirements.txt. This is not really needed when using a package manager like UV (this is maybe a bit technical but it shows that the agent is not flawless). Stating this before continuing makes the agent change it’s response.

Here we can see whare extra context from an MCP server could be handy. Looking at the UV documentation it states the following to start a project:

This way of working is not followed by the Agent. Now the question is, do we keep feeding this in to the agent or do we start taking over. For now I Just pasted this knowledge and reviewed what he was stating, now it looked like he was on the right track.

Now there was an extra issue => UV was not yet installed in my base python interpreter. The Agent, seeing the command to create the environment failed. Then proceeded with just creating all the files and stating ‘It appears that the UV command is not available in the current environment. Let's proceed with creating the backend files directly and provide instructions on how to install the necessary dependencies later.’ This behaviour was not set by me. If I would like something else this should be written in the main prompt. This can get really tedious. To solve this I manually installed the package on the main interpreter.

Allready in the code some issues arose. For example using deprecated methods. This I changed manually.

Other issues like the documentation not aligning with how it should be handled also need to be changed. These small errors are quickly adding up, making the creating of the application in a way, more complex.

The venv folder created was also unneeded seeing UV installs it in a .venv folder.

Now the next step was setting up tests for the backend. This was done by selecting the context needed and then stating tests for the backend are needed. It again first created the directories and then started adding tests.

Personally I am not really encouraging the use of AI for writing tests, it could have you fall for a sort of ‘looks good, probably fine’ mentality. The goal of tests is to really test you application on certain logic. Letting this code be created by AI could even defeat the purpose if left unchecked.

Running the tests all of them passed with some warnings due to deprecation messages. I changed these manually.

So now that the backend is set up, we could take a look at the frontend again. From the first Agent request we already had some frontend code, this will probably need to change due to the backend that exists now.

Again selecting the context that I assumed valuable for the question of connecting backend with frontend is given together with the prompt to the agent. It then started changing the frontend. When installing new packages the same issues as with UV popped up. For the frontend with Bun. The ai models really want to use the basic Npm or pip modules. This is due to the vast amount of data they are trained one, in this dataset only a small percentage would be with these more ‘niche’ package managers.It does work when prompted more but it is something to keep an eye out.

Then running the frontend and … Error

To be completely honest. It was an error that I was suspecting. Again asking for something like tailwind makes it harder on the model. It is used to work with normal CSS styling the tailwind setup is a bit more in depth, this in combination with bun makes it also a puzzle.

After making it work manually I prompted the model to remove all the created CSS and make the frontend truly Tailwind.

After waiting for a minute or so the result was ready to be viewed. The website and backend where up en running (after manually running them both)

Now as last step is just taking the whole repository and trying to see if all the context from bun, to uv and routing is taken into account when creating dockerfiles and dockercompose to containerise these apps. This containerisation is a way of bundling applications so that if it works on my machine it also works on everyone else’s. We will probably write a more in depth explanation of what it is and what it can do. If you are interested do let us know!

Doing this step exposed even more errors that happened when setting up the UV environment. This was then fixed manually.

Next to this small error it actually did it pretty well, creating a good network, connecting the services and rewriting the code so that it all works containerised:

This then concludes the creation of a Containerised To Do app using github copilot. It took around 1 to 2 hours including documenting the steps and fixing some problems manually. SO all in all not bad to have something up an running as quickly as possible.

Now for the codebase. It generally is till readable and manageable. This also part due to the simple nature of the application. The backend is nicely compartmentalised in separate functionalities. On the other hand the frontend is not that great, everything is pushed into one App.vue file. This makes it easier and harder at the same time. It maybe comes down to preference but I would have liked to see some more components. This again should then be created manually by having an architecture document or a schema on how to build the application beforehand or by… vibing or adding it into the prompt.

There have also been things that are not looked at, for example the security of it all, adding bearer tokens or JWT. This is something I would suggest handling yourself and making sure that it is correct. If you are using AI to make you work faster, do take the time to make it so that your work is also Better and don’t forget to user your own skills.

Cursor

Link to database

Going onto cursor to create the same application I first started with looking a bit more into the IDE. For example cursor has it’s own .cursor/rules folder to place you cursor rules and other settings into.

Seeing the other tools do not have this functionality I did want to try this out. So what did I do, I just looked for rules file that are of interest for Python fast api, Tailwind and other libraries I planned to use. You can then mention the rules as you mention a file or just have the Agent select the one he needs based on the description.

After setting the rules I gave the Agent the same prompt.

A difference from the get-go was the ‘run command’ being ran in the agent tab, this did make it more contained in the agent process instead of jumping to the main terminal window in. It also started with the backend and not the frontend (this could be the model, but I assume the extra rules also have something to do with it)

There where also again the same issues of not using UV manager for python and defaulting to pip with he requirements.txt file.

Due to the rules the general structure was also much better! The frontend is split in components and is following the vue3 rules.

There were some other functionality issue’s but I will write these of as an LLM problem and not a cursor thing.

When creating the containers, it suffered from the same issues. This even with the existing cursor rules. I probably should need to create a new one specific for docker and the used tools and libraries. This is something that could be interesting for companies working with a consistent codebase, for my use case I just did the changes manually.

Result:

Windsurf

Link to database

Now the last of the tools, Windsurf. It is said that windsurf is sort of the more polished version. Where copilot is the neutral, Cursor is for the thinkerers and Windsurf is more for the vibes.

Right out of the gate it lookls like a lot more ports are forwarded. This presumingly for some more usability. There is also a clear (deploy website button) that is focussed on deploying to Netlify. So the first feel is in line with the vibe: try to make it as easy as possible to start programming.

To start of I selected the ‘Write’ option and not the chat, this seemed like the latest version.

Again it suffered from the problem of not using the package manager specified as did the others. In this case it even did not create a virtual environment. This is not that good

A more general remark is the more ‘simplistic’ approach, not many folders or file nesting just some raw files. This would be more simple for a beginner but would also make it so that the codebase will become pretty cluttered.

An extra feature not seen in the others is the addition of a preview by standard. This brings it already a bit more to the Ai development tools where you really can stay in the tool. It also has the possibility to copy the console errors directly into the chat prompt.

In the building step the same behaviour pops up again. It seems that the context and how it is handled is still something the tools need to figure out. It also performed a bit worse when containerising. This could be a hiccup of the model although the others both handled it in a completely different manner. This makes me assume that it is a rule to simplify it is much as possible that is given as a system prompt.

After some troubleshooting this app also was able to be ran containerised.

AI Development

The Second group are the AI development suites. These are the new kid on the block. Most of them have the goal to ‘democratise’ coding for the masses. Or simply said be an easier to work with site/app development platform than what exists and have also more capabilities and customisations leveraging the code libraries that programmers use for the custom websites.

For this blogpost I have chosen the following 3, given the recent news and funding for some of them. They are placed in no particular order.

• Lovable
• Firebase Studio
• Replit

For people not interested in the whole process here is the TL;DR from my personal experience

My general process with all of them was starting with a base prompt and seeing how far I could go with just ‘Vibing’ my way through the code. This is how I would assume they suspect people to work with it. Seeing most of the tools have the code behind some 2 to 3 click process. Next I handled errors by trying there built in error handling systems (spoiler alert - with some frustration).

All the workflows for these tools are the same :

• Prompt
• Review
• Prompt

Most of the feelings and remarks for the coding tools will be the same. For me it sometimes feels slow, not seeing the code and just waiting for the changes feels unproductive. The fact that it also takes longer further on is not a good forbode for larger projects. As for the quality, most of the tools do make a lot of errors, this is logical seeing in the code editors there are also many changes needed.

I will show some screenshots to give you the general Idea and will then explain the feeling about the specific tool in the little exploring I did with it.

Lovable

Link to database

Coming from a programming background it felt like not much was happening. The chat UI leaves the code changes in the backend. Also the UI was not consistent for me, be it a weird chrome plugin or just the app I had myself refreshing the browser once in a while.

The result itself I found personally the best out of all of them. Possible due to the preferences set by the creates of Lovable. The basic prompt given did not specify which framework to use to give the tools the best chance to show possible optimisations for Non code people.

This screenshot is how the result is displayed next to the ‘chat’ interface

Firebase studio

Link to database

Firebase studio felt a bit more as it was creating something. It shows the code while generating. Also the starting point is nicely done with sort of planning the general project and setting some base colors as well.

Another nice feature I didn’t see in the others was the draw to change. For people that do want to give a try at coding without really having the knowledge it could be a great help being able to draw what they want changed. Although I would suggest learning it if it would become your day job. But for a one of project, perfect!

The general output was the least of all of them, I tried to generate a second time seeing it could be a bad starting point and indeed better results! So keep that in mind. Next to that the number of errors after the first generation was also higher than with the others. Thers wasn’t even a result. Seeing it just went through it’s whole ‘agentic flow’ I would have suspected it would at least show something.

The more Vs code like editing also felt as a good extension. I would say it is more targeted to people interested in coding and open to learning and tweaking it themselves.

Code generation and changes

Draw to change feature

Replit

Link to database

Replit feels a lot more tinkery, a lot of options, side bars and so on. This could make it more interesting but also makes it feel a bit more daunting to get started. There are a lot of extra features which I did not get into. The code changes are visible although directly integrated under the folder tab in the top left.

This makes it more a sort of coding focused approach with the ability to have that AI Agent.

The result in general looked good, error handling worked fine and it could even immediately be deployed on Replit.