Bounties attract serious brainpower to the challenge.
The concept: The idea is to combine visual plant identification technology with a large database of ethnobotanical (plant utilization) information so that the user could instantly know which plants from their natural surroundings can be used and how along with the information when/how to collect and prepare them for use. Two main use categories are edible and medicinal plants, but all the other possible uses would be included (e.g. technical, handicraft, veterinary, etc.). The app would be intended for use in the wild or semi-wild plant habitats (not for garden plants).
The novelty of it: The novelty of the idea lies in the app being able to identify many plant species quickly from a single image or footage (with the help of supporting information like geolocation) rather than only specific species from a specific photo. And also in vastly enlarging the ethnobotanical database available to the app so that the majority of wild plants in any area would have some uses attributed to them. The app would therefore be able to quickly present you with the list of plant species present around you and information on how each species can be used. Practical guides, such as videos on plant collection and preparation would also be included.
Technical aspects:
Visual identification part: Using device's camera the person would "scan" a certain place in the wilderness. The most convenient approach would be to simply film or make an extended panoramic photo around yourself. There are already apps for plant identification from photos and they are pretty accurate. With the improved image recognition technology, it would be possible to identify the type of the habitat as well as separate plants from a high enough quality panoramic photo or footage.
Identification of the habitat type (which is relatively easy to do from the views of the surroundings) in combination with the information of the geographic location of the person would already let the app guess what species of plants can be present in the surroundings and make it easier for the algorithm to then recognize separate plants by analyzing specific parts of the image or footage.
The app could then also guide the user to approach more specific areas and take additional photos from closer distances to further drive the process of identifying all the species present in the surrounding habitat. If a person gets interested in a particular plant, they could then take additional photos up-close to help the algorithm identify the species with higher certainty.
Wild plant habitat mapping is a common practice in environmental science and the app could also use already available information (databases of habitat mappings) to supplement the identification process. Plant species are usually repeating over a relatively large land area (which makes up a habitat) so when the species in a certain patch of the habitat are identified the person then can walk around and they will most likely find the same species.
A simple smartphone camera might not be enough for accurate large-scale identification, so a separate, specialized camera (like a 360° camera on a stick) paired with a smartphone or other personal device might be necessary. In a more distant future scenario, the technology could also use material samples to identify the species using methods of genetics and molecular biology.
Utilization part: Some present plant identification apps already provide ethnobotanical information about some of the plant species, but still to a very limited extent. The envisioned app would use the information available in plant-use databases like pfaf, but additionally, many other literature sources (books, scientific articles) to compile a large enough information pool which then would be available to the user.
It's important to prioritize the information which is trustworthy and based on practical knowledge, ideally, it should be backed with either traditional knowledge (ethnobotanical research) or otherwise practically tested (books by plant enthusiasts, etc.). Specific use being mentioned in multiple literature sources is also a criterion for trustworthiness. Reliability rank of an information about specific use of a plant should be shown to the app user in "plant's profile".
To make plant utilization as easy as possible, information about the practical aspect (how and when to collect the plant or its parts and how to prepare and consume or otherwise utilize it) should be presented in a most learning-efficient and user-friendly manner. Therefore extensive use of educational videos seems like the best approach.
Possible disadvantages/risks:
The danger of poisonous plants:The plants that are seriously poisonous are not that common. But they are the ones that should be given exceptional emphasis in the identification algorithm. If a person decides to try using a certain plant they should then additionally take several photos of that plant up-close for the algorithm to completely rule out the similarities with poisonous plants and ensure safety at the same time identifying the species with higher accuracy.
Endangered plants: This kind of technology could make things easier for people who are willing to abuse wild resources. Some possible solutions to this: pop-ups and emphasized information near descriptions of certain plants about their endangered status and warnings about fines associated with their abuse (for starters) and app use only allowed through logging in with personal ID, if things would get out of control. Although it's not likely that a lot of people would use the app extensively, some, I imagine, would use it for playing and others (like nature enthusiasts, hikers, low-budget travelers, or people who live in rural settings and more or less depend on wild nature for survival) would use it more extensively.
Why is it important:
Responsible use of wild resources reduces the consumption of farmed food as well as other manufactured products, which in turn lowers capitalistic consumerism, increases the possibility of replacing agricultural lands with natural or semi-natural ecosystems which then reduces CO2 emission, improves air quality, and as a feedback loop provides more natural products to depend on.
Making utilization of wild plants easier for laypeople increases public interest and appreciation of nature, for they get to know how it can be useful practically and that the possibilities of utilization are surprisingly abundant. All it takes is easily accessible information to simplify the process enough.
Responsibly used natural resources provide a possibility for free off-grid living and enable individuals to be independent and self-reliant.
The idea can be extended to identify the uses of objects other than plants
Samuel BelloSep 03, 2021
Plants are generally harder to identify when compared to most of the objects we see every day. If an app or online platform could provide the same services by generating the uses of the objects just by uploading the pictures of the objects.
The identification of the object's uses can be done by humans at first while a system that automates the process can be put in place later.
Please leave the feedback on this idea
AI identification
Spook LouwApr 15, 2021
I have a friend who has been working on the visual identification part for the company he owns
It's obviously not used for the same purpose as you are proposing and it focuses on fruit. They use it to get an estimated yield and to identify problem areas but I believe the technology might be similar to what you will need.
I know you mentioned that similar technology already exists, but it might be worth taking a look at what they're working on. Here's a video of their AI tech at work.
If this can be linked to a database of the information you mentioned your app will be good to go!
[1]https://www.revolutesystems.com/
Please leave the feedback on this idea
Povilas S2 years ago
Thanks. I checked it out. They could improve the website a bit, to make it easier to read/understand the info, - in the lower part, which is divided into 3 separate vertical sections, the text from each section overlaps and makes it difficult to read, I tried on different browsers, it's the same (maybe feedback will be useful for them). But generally, it seems like a useful technology for farmers.
The technology is, indeed, used differently, it's mostly for counting fruits, as I understand, so to differentiate them from the rest of the plant is the main task of the algorithm. I don't know much about visual identification algorithms, but I think base principles are the same, no matter what you try to differentiate, just that to identify different species requires a much more sophisticated algorithm than to tell, e.g. the fruit from the leaves.
Yes, there are already many apps for plant identification from photos, which were not yet present a few years ago, so I'm happy about this because the basic technology is available, but what should be improved to make an app like this possible is identifying separate species from a panoramic photo or video of natural surroundings (not just by single shots of separate plants). So yes, fruit identification technology does that to an extent, but it differentiates simple, similar objects, the app I envision would instead mark different species of trees, herbs, etc. This is much more difficult but nevertheless seems possible.
That website gave me another idea that satellite images could possibly be useful for the identification of natural habitat types.
Please leave the feedback on this idea
Add your creative contribution
General comments
Shubhankar Kulkarni2 years ago
Great idea and completely doable! I like how the first of the three points towards to end tries to accommodate virtually every big problem :)
Bird enthusiasts like to know where a species was seen and when. https://ebird.org/home helps them immensely. In this regard, I like that the app will store the location of the plant. Birds move and the sighting is by chance. That will not be the case with trees. If someone posts in the spring season that a plant in their neighborhood is flowering, other users may visit and enjoy the beauty.
One suggestion: It would be user-friendlier if you could avoid the 360° camera on a stick and other apparatus. Smartphones are smarter than ever and can accommodate all technical aspects. As an alternative, I would go with this - "... they should then additionally take several photos of that plant up-close ...". Not just for poisonous plants but all the plants. The app could have separate sections to upload/ capture photos of a leaf, maybe the stem, if possible flowers and fruits, etc. The database can be equally equipped so that the app could identify the plant from these photographs.
A question: You mentioned in your reply to Juran - "the app would be intended to be able to identify species from pictures of general greenery made somewhat at a distance". Wouldn't identification from photos of individual trees be easier? Also, if someone wants to identify all the species from a landscape, the image will need to be of a very superior quality (high resolution) and an equally superior algorithm, further increasing the cost of both the user devices and the app. I am not against it but I think this should not be the primary aim. The primary aim can be identifying individual plants (from individual plant photos).
Two different add-ons:
1. I agree with Juran K. 's suggestion of the app having the property of identifying plant diseases. This is not just helpful for the crops but can also be helpful for wild species. If there is a diseased plant, the user should know that it is diseased (some diseases though visual may not be able to be differentiated from the plant's phenotype). For example, the fern spores are atypical and can be mistaken for insects or disease. If the plant has a disease, the user should know whether it is safe for them to consume/ use it.
2. I am assuming ethnobotanical uses will include both medicinal and culinary uses. In this case, the users should be able to post recipes and upload photos. This social aspect of commenting and reacting to photos, recipes, and other preparations will increase the popularity of the app and improve people's knowledge regarding the use of the species. Even when using a plant for non-culinary purposes, it would be beneficial for the user to know how a plant can be used. Recipes and other preparations will help.
Please leave the feedback on this idea
Povilas S2 years ago
Shubhankar Kulkarni Yes, plants being stationary is a very convenient feature in this context. You can spot a certain habitat with a lot of useful plants somewhere and if you mark the coordinates you can come back there after many years and still find it more or less the same if humans didn't interfere. And the same goes for individual plants and their populations. This is especially useful in the context when information is shared amongst people.
When I was traveling low-budget I had an idea to start a wiki "strain" for travelers to share information about edible and otherwise useful wild plants. Similar to hitchwiki or nomadwiki. Once someone spots a population of useful plants in a certain location they can then report it in an article about the closest town (or another geographical object) together with the description about plant's uses, personal experience with it, etc. An interactive map then can be developed to mark all the locations with certain plants spotted in them and update the info once someone else has visited the same location.
This is related to what you suggest about the social part of the app. However, in the context of the envisioned app, I see some disadvantages to this. First, to make an app a high-quality tool, the information about the uses should be trustworthy enough, mixing it with social media-like posts is not a very good idea if you want to keep the information accurate. Even though this has the benefit of supplementing already existing info and adding even more practical experience, but at the same time, you can't filter all the bogus which in ethnobotany is already quite abundant.
Mixing a practical tool with social media might not be a very good idea, it depends on whether the people using it are serious and responsible. It might be better to make a separate, related program for this, or an extension, not directly interlinked.
Please leave the feedback on this idea
Povilas S2 years ago
Shubhankar Kulkarni "The primary aim can be identifying individual plants (from individual plant photos)." But this is already possible with existing apps, so the only improvement then would be the ethnobotanical part, which is also to an extent already possible.
You see the way I envisioned it is for an app to enable you to quickly know which species from your surroundings can be used and how by making a list of useful species available in a certain spot of nature. In the case when the app can only identify one plant at a time, you have to go around photographing plants and play this kind of a blind chance game, even if the app provides ethnobotanical information about each plant. Especially for a person who knows very little about plants, they can hardly see few species, because they don't look for differences, they see "the greenery", so even to find enough different plants to photograph would be a task for them, you have to train the eye for this.
Imagine people are camping and want to try eating some wild plants. If the app can only identify one plant at a time they would have to go around photographing until they found something edible enough - this is not a very appealing approach, although it will work, yes, they will try something with enough perseverance, but imagine if they can know quickly all the edible plants available around them and then can only choose what to eat according to what's most nutritious or otherwise attractive/beneficial, this is game-changing. Then they can go look for the plant they chose, reidentify it with closer pictures, collect and prepare it. They don't have to play the blind searching game. Now imagine you need a herb to stop the bleeding from a wound - even worse.
That's why I think that investing in this is absolutely worth it. Even if it takes a specialized camera and a very sophisticated algorithm. For people who just want to play around and wouldn't have enough motivation to seriously use plants even if they had a list of all useful plants available around them, the first option is good enough, but for people who are more seriously interested, having a separate camera in a backpack is not a problem.
You make a good point about the identification of plant diseases being useful in the case of wild plants. This would be helpful, yes.
Please leave the feedback on this idea
J. Nikola2 years ago
Add-on idea:
- make the database of plant diseases with pictures, morphological and physiological symptoms, causes, and solutions
It could help farmers and all the people who just entered gardening to detect the problem when they encounter one. The app would scan the area as you mentioned and according to the habitat, condition of the soil and other plants, the period of the year, and data on climate, disease break-outs, spraying, and general fertilization habits in the last few years, identify the problem that you encountered and offer you the most efficient and economical solution.
Please leave the feedback on this idea
J. Nikola2 years ago
The idea is awesome! Not only the technology and algorithms behind it, or the three points you highlighted at the end, but also the frame you are trying to fit it in.
The kids nowadays feel adventurous when visiting the nearby forest, which was completely normal and natural for me as a kid. Therefore, this app would offer a nice tool to overcome the distance between nature and technology-raised kids. With the use of their "new body extension" (aka smartphone), they could safely dive into the wilderness of nature. Really nice!
Would you make this app freely available or it would have a subscription or something similar?
If it would be free for everybody (which would fit in your vision), how would it be financed?
How would you solve the problem of different growth phenotypes? Many young plants are almost identical and have differences visible only when dissected.
Also, to detect the species more precisely, maybe it would be nice for the algorithm to consider the time of the year and define the phase the plant is now in. Could solve some issues.
Povilas S Jan 23, 2021
