A short and biased test of AI copilots

2024-05-12

I am using AI copilots time to time for various tasks. In this post I compare Github Copilot, Jetbrains AI Assistant, and Deepseek AI in answering a specific question related to React. This is a very simple and short biased test.

The case is: I am a developer who is working on a React application and has some issue with how it works. I ask a question about it and expect an answer that would help me to solve the problem.

The original question looks like:

I render a list of categories via calling 'map' on the categories list. Every category then has a onClick event where I call handleEditCategoryClick passing the category into it.
The handler internally calls:
setSelectedCategory(category);
dispatch(editCategoryModalAction(category));
So, my expectation is to open a modal for every specific category so user could edit it.
But when I click on a category on the web page, the modal always opens with the same category. How do I fix that?

I asked the same question in the same form to all three AI copilots and here are the answers I got.

Github Copilot#

I use Github Copilot as an extension in JetBrains IntelliJ IDEA Ultimate. It is a paid subscription, not trial.

The answer#

The issue you're experiencing is indeed related to how React's useState hook initializes its state. When you pass category.name and category.bg_color to useState, it only uses these values for the initial render and does not update them on subsequent renders, even if the category prop changes.
To fix this, you can use the useEffect hook to update the state whenever the category prop changes. Here's how you can do it:
const [categoryName, setCategoryName] = React.useState(category.name);
const [categoryColor, setCategoryColor] = React.useState(category.bg_color);

React.useEffect(() => {
    setCategoryName(category.name);
    setCategoryColor(category.bg_color);
}, [category]);
In this code, useEffect will run whenever the category prop changes. Inside the useEffect, we're updating the state with the new category prop values. This will ensure that your modal always displays the correct category information.

Interpretation#

Github Copilot provided a detailed answer that explains the issue and provides a solution using the useEffect hook. The answer is clear and concise, and it solves the problem. No follow-up questions were needed to clarify the answer.

Jetbrains AI Assistant#

I use Jetbrains AI Assistant as a plugin in JetBrains IntelliJ IDEA Ultimate. It is a paid subscription, not trial.

The answer#

Let's do a step by step debugging.
To verify if the problem is with setSelectedCategory, we need to know the implementation details of it. Particularly if it directly sets the value or performs some calculations which may result in the same category always being set.
If setSelectedCategory works fine, the problem may lie with the state update or the dispatch function editCategoryModalAction. In such a case, we need to investigate how the state update is managed on clicking an item and how redux's dispatch function is implemented.
Another potential cause could be some kind of caching or memorization logic that is causing the same item to be returned.
Let's get started by looking at the implementation of setSelectedCategory. To do it, could you please provide the definition of the setSelectedCategory function or should I find it in your codebase?

Interpretation#

Jetbrains AI Assistant provided some ways on how to debug the issue, but it did not directly address the problem or provide a solution. The answer was more focused on identifying potential causes of the issue rather than providing a solution. A follow-up question was needed to clarify the answer.

Follow-up question#

The implementation of setSelectedCategory looks like this:
    const [categoryName, setCategoryName] = React.useState(category.name);
    const [categoryColor, setCategoryColor] = React.useState(category.bg_color);
And this is how I call it:
{selectedCategory && (
   <EditCategoryModal isOpen={isModalOpen}
        onRequestClose={() => dispatch({type: EDIT_CATEGORY_MODAL_CLOSED})}
        category={selectedCategory}
        presetColors={presetColors}
        onSaveCategory={(category) => {
        // Close the modal after saving
        dispatch({type: CLOSE_BOOKMARK_MODAL});
        }}/>
     )}
So, when user clicks on a category, I call setSelectedCategory. And here then if a category is selected, the modal window is called, and I pass the selectedCategory into it.
But, when the modal appears, it contains the old value.
I have a feeling that before the first call, the modal window's state is empty so it initializes from the arguments passed to the modal. But the next time, since the state is not empty, it does not update it. And so the old value appears in the modal.
How do I fix this?

Answer to the follow-up question#

It seems like when you're opening the modal to edit a category, the internal state of the modal is not updating because it is only initialized once when the EditCategoryModal component is first rendered (with the initial category prop) by the useState hooks. After that, even though the category prop changes, the state inside the EditCategoryModal component does not re-initialize.
We can solve this by using the useEffect hook which will re-run whenever the category prop changes. Then we set our state inside this useEffect. Here is how you can do it:
const {name, bg_color} = props.category;
const [categoryName, setCategoryName] = React.useState(name);
const [categoryColor, setCategoryColor] = React.useState(bg_color);

React.useEffect(() => {
setCategoryName(props.category.name);
setCategoryColor(props.category.bg_color);
}, [props.category]);
In the above code, useEffect will update categoryName and categoryColor whenever the category prop changes. This should solve your issue and the modal would show the correct category values upon each opening.

Interpretation#

Jetbrains AI Assistant provided a working solution to the problem after the follow-up question. The answer was clear and concise, and it solved the problem. No more follow-up questions were needed to clarify the answer.

Deepseek AI#

I use LM Studio as a standalone web application for accessing the Deepseek AI model. For this test I used Llama TheBloke/deepseek-coder-6.7B-instruct-GGUF/deepseek-coder-6.7b-instruct.Q8_0.gguf model.

The answer#

The issue you're experiencing might be due to React's concept of "key". When you map over an array in React and render each item, it is important to provide a unique key for each element. This helps React identify which items have changed, are added or removed from the list.
In your case, if category object has some sort of id property that uniquely identifies each category, you can use this as the key:
categories.map(category => (
  <div onClick={() => handleEditCategoryClick(category)} key={category.id}>
    {category.name}
  </div>
))
If category object does not have an id, you can use the index of each item in the array as a key:
categories.map((category, index) => (
  <div onClick={() => handleEditCategoryClick(category)} key={index}>
    {category.name}
  </div>
))
Remember that using the array index as a key can negatively impact performance and may cause issues with component state, if the items in your list could be reordered or deleted. Using an id is usually more reliable.
Also note that you should not use onClick event on the div but on some specific element inside of it (like a button) to make sense from UX perspective. The div can just serve as a container for your category name and any other related data, while the clickable area is actually on the child elements.

Interpretation#

Both Github Copilot and Jetbrains AI Assistant were running in the IDE, so they both can have access to the codebase. Whereas, Deepseek AI was running as a standalone desktop application and did not have access to the code, so it could rely only on the data and context provided in the question. This is why, I guess, the answer looks like a general advice on how to use keys in React components. The answer is valid and can help improve the performance of the application, but it does not directly address the issue. A follow-up question was needed to clarify the answer.

Follow-up question#

The implementation of setSelectedCategory looks like this:
    const [categoryName, setCategoryName] = React.useState(category.name);
    const [categoryColor, setCategoryColor] = React.useState(category.bg_color);
And this is how I call it:
 {selectedCategory && (
    <EditCategoryModal isOpen={isModalOpen}
     onRequestClose={() => dispatch({type: EDIT_CATEGORY_MODAL_CLOSED})}
     category={selectedCategory}
     presetColors={presetColors}
     onSaveCategory={(category) => {
     // Close the modal after saving
     dispatch({type: CLOSE_BOOKMARK_MODAL});
    }}/>
 )}
So, when user clicks on a category, I call setSelectedCategory. And here then if a category is selected, the modal window is called, and I pass the selectedCategory into it.
But, when the modal appears, it contains the old value.
I have a feeling that before the first call, the modal window's state is empty so it initializes from the arguments passed to the modal. But the next time, since the state is not empty, it does not update it. And so the old value appears in the modal.
How do I fix this?

Answer to the follow-up question#

The issue you are experiencing might be due to React's concept of "state". When a component renders and its state changes, React will re-render that component with the new state. However, if your setSelectedCategory function is not inside a useEffect hook or within another function that gets called on every render (like in an event handler), it won't cause a re-render of the component where you are using selectedCategory as a prop for EditCategoryModal.
This means that when your modal renders, it will not know about the state change and keep showing the old value because React doesn't automatically update child components when their parent's state changes.
To solve this issue, you should make sure to wrap your component in a function and call it inside React.useEffect hook with an empty dependency array ([]). This will ensure that the effect runs only once after initial render:
React.useEffect(() => {
  // Your code here...
}, []);
Inside this function, you can call setSelectedCategory(category) and it should update your component's state correctly. Then when the modal re-renders due to a prop change (i.e., selectedCategory changes), it will use the updated value of selectedCategory from the state instead of the old one passed as a prop.

Interpretation#

Deepseek AI provided a working solution to the problem after the follow-up question. The answer was clear and concise, and it solved the problem. No more follow-up questions were needed to clarify the answer. On the other hand, both Github Copilot and Jetbrains AI Assistant provided the useEffect solution that can be copied and pasted, whereas Deepseek AI provided a general advice on how to use useEffect hook. If you're not proficient in React, you still might need an explanation or a more concrete example.

Conclusion#

All three AI copilots provided a working solution to the problem after the follow-up question.

Github Copilot gave a detailed answer that explains the issue and provides a ready to use solution using the useEffect hook. No follow-up questions were needed to clarify the answer.
Jetbrains AI Assistant brought some ways on how to debug the issue, but it did not directly address the problem or provide a solution. A follow-up question was needed to clarify the answer. After the follow-up question, Jetbrains AI Assistant provided a working solution to the problem.
Deepseek AI started with a general advice on how to use keys in React components, but it did not directly address the issue. A follow-up question was needed to clarify the answer. After the follow-up question, Deepseek AI also were able to provide a working solution to the problem.

So, Github Copilot is my choice. And when I have privacy concern or I am offline, I would go with Deepseek AI.