AI for Jewelry Recognition
Technologies:
About the Client
Our client was an international IT company Yellow Media with which we cooperated under the white-label model. The company provides services to clients from various industries and markets, and develops unique, effective, and innovative software products for leading enterprises. Accordingly, they provided our Data Scientists as outsourced specialists for their client.
Business Context
The Amazinum company has outsourced its specialists to our partners. Thus, the end customer was a jewelry marketplace where users can upload photos of their jewelry for sale. Thus, our data scientists faced two tasks:
- Classify jewelry according to certain characteristics according to the photo. The aim here was to help users with profile filling, so they spend less time describing item features.
- Carrying out the identification of the model of the jewelry by its photograph, with the same aim to improve user experience on the website.
Challenges for Amazinum Data Scientists
Among the challenges our Data Scientists faced was a large amount of unorganized data. From the photo database, Amazinum specialists carried out the gathering and filtering of the data.
Amazinum Data Scientists in Action
Task 1
To perform the first task, namely the classification of jewelry according to certain characteristics, the client provided access to a database with products and images. Specialists needed to develop the following classification models – type of jewelry; type and color of the metal; type and shape of the stone.
Accordingly, first, it was necessary to prepare the data. For each type of model, data scientists had to collect as many diverse image samples as possible, which would be balanced across classes. After collecting the dataset, they started building a pipeline to train the model on the dataset. They trained several image classification architectures for neural networks – ResNet, MobileNet, YOLO, and Vision Transformer, hyper-tuned them, and performed evaluations until they stopped at the baseline architecture, which satisfied both the specialists and the client.
During prototyping, the scope has changed to include a process that would perform jewelry localization on the image. It finds jewelry and removes irrelevant details by cropping the image. Such flow improves the accuracy of further classification processes. Data science specialists decided to replace the jeweler-type classification model with a jeweler detection model that performs both localization and classification using the YOLO detection model.
After that, they deployed a solution as a WebAPI server with an endpoint that accepts images and returns results with a description of each characteristic of the received jewelry product.
Task 2
To implement the second part of the project, the client provided a list of jewelry brands and products that the solution has to detect. Data scientists performed the filtering and collection of the data.
After that, they used the Vision Transformer Model, which collected visual embeddings from images. The model transforms the image into a vector of characteristics that describes an image. Initially, the data scientists used a pre-trained Vision Transformer, later they conducted experiments on tuning a model on the jewelry dataset. Such a tuned model is oriented toward extracting visual characteristics specifically for jewelry. In this way, specialists made a solution using the Image Retrieval System. They formed a database of the brands and products that needed to be recognized and created visual embeddings for each image in the database. And deployed it as an endpoint to existing AI WebAPI, which, upon receiving an image, vectorizes it and searches for the most similar images in the database.
Review
AI Development for IT Company
Their easy solutions to complex problems, pleasant communication, and the team members’ responsibility are impressive.
Amazinum impresses the client with their ability to provide easy solutions to complex issues. Their responsible team manages the project efficiently, ensuring everything is at a high level. They also facilitate pleasant communication through online meetings.
5.0
★ ★ ★ ★ ★
Their easy solutions to complex problems, pleasant communication, and the team members’ responsibility are impressive.
