Teaching a robot to recognize weeds requires thousands of photographs in which every single plant has been manually labeled by an expert who can distinguish between species. It is a time-consuming process with limited reusability: it often has to be repeated for every new field and every new season. This is why artificial intelligence is increasingly being used to generate some of these images, raising the question of how realistic they need to be in order to be useful. A common assumption is that more realistic synthetic images will make it possible to train more accurate detectors. Our results, however, suggest that realism alone is not necessary.
This question was explored within the NeutraWeed project, which focuses on the recognition and selective management of weeds in crops (we discussed it here), during a research period at the Centro de Automática y Robótica of CSIC in Madrid. NeutraWeed is based on the observation that not all weeds are harmful to crops and that some can coexist with them. Before deciding which plants should be removed, however, the robot first needs to be able to recognize them.
Two ways to generate an image
When synthetic images are used to train a detector, the first question seems almost inevitable: how closely do they need to resemble real photographs? But visual realism is only part of the problem. What also matters is how many different situations the images are able to represent.
Generative adversarial networks (GANs) put two networks against each other: a generator, which produces images, and a discriminator, which tries to distinguish generated images from real ones. During training, the generator gradually learns to produce increasingly convincing images. However, this mechanism does not guarantee that the full variety present in the data will be reproduced. The generator may learn to produce certain configurations very well while continuing to neglect others, a phenomenon known as mode collapse.
Diffusion models, including Stable Diffusion, are trained according to a different principle. Starting from images that are progressively corrupted by noise, they learn to reconstruct the information needed to move back toward the original image. There is therefore no discriminator tasked with deciding whether the result looks real: the model learns directly from the examples used during training. Less frequent configurations can therefore also contribute to the learning process.
This does not mean that a diffusion model automatically reproduces the entire data distribution, nor that it is immune to bias. The point is that realism and variety are not the same thing. An image can look perfectly realistic without teaching the detector anything new. Conversely, an image that is less convincing from a photographic point of view may expose it to a situation it has never encountered before.
What the detector needs
This distinction matters because the generator and the detector have different goals: the former needs to produce convincing images, while the latter needs to learn how to distinguish species correctly.
When it comes to plants, during training the detector learns to separate species based on the features present in the images. To recognize them, it primarily needs to see the species under the many conditions in which they may appear in the field: small and large plants, directly lit or in shadow, isolated or partially hidden by other leaves, and photographed on different types of soil. Making an image more realistic may simply mean refining an example the model has already seen. Increasing the variety of scenes, on the other hand, can expose the model to new combinations of shape, position, lighting, occlusion, and context. For the detector, therefore, a more realistic image and an image that adds something new do not necessarily have the same value.
Visual realism and usefulness for the detector do not necessarily coincide. On the right, an example from the more photorealistic dataset; on the left, an example from the less photorealistic dataset, which led to an increase in detector accuracy.
The same applies to the botanical accuracy of synthetic images. A botanist identifies a species through very specific traits, such as leaf venation, growth habit, or the order in which leaves emerge. An automatic detector, however, does not necessarily “look” at the plant in the same way: it relies on combinations of shape, color, and arrangement that allow it to distinguish between species at the resolution of the images used for training.
This does not mean that botanical accuracy is irrelevant. When the generator had not been adequately adapted to the species under consideration, the synthetic images actually reduced the detector’s performance. Once the features needed to distinguish one species from another were represented, however, greater fidelity in botanical details did not provide benefits comparable to those achieved by increasing the variety of scenes.
The results
To investigate the relationship between the realism of synthetic images and their usefulness for training, we compared several variants of the generation pipeline, each designed to modify specific visual properties. The results show that, in our experiment, increasing the variety of scenes was more useful than simply increasing photorealism.
This result also has a practical implication: if increasing photorealism requires more computational resources, it may be more effective to use those resources to generate a more diverse set of scenes. In the case of in-field plant recognition, the techniques we adopted proved particularly useful because species recognition relies heavily on features such as leaf shape and arrangement. The question, then, is not only how realistic a synthetic image looks, but also how much it expands what the model is able to learn.
Stay tuned!
