Using Open Source Embeddings Model

[cursive citrus]

Hai everyone, I'm working on a use case where I need to classify an embedding either to be positive or negative ? I'm getting embedding from open ai, I need to classify this embedding as positive or negative. I have a training data, can I use open source embedding models to convert them to embedding and predict on the embedding given by the open ai ?

Asked by @rugged sapphire

[floral skiff]

hi @rugged sapphire

Azure AI Foundry is a robust platform that can help you achieve your goal of classifying embeddings as positive or negative. Here’s a tailored approach to using Azure AI Foundry for your use case:

Steps to Classify Embeddings Using Azure AI Foundry

1. Generate Embeddings for Training Data

   • Use Azure AI Foundry's embedding models to convert your training data into embeddings.
   • Ensure you have labeled data (positive or negative) to train your model.

2. Create a Training Pipeline

   • Set up a training pipeline in Azure AI Foundry to process your training data.
   • Include steps to generate embeddings using the foundry's embedding model and store them.

3. Train a Classifier

   • Train a machine learning classifier using the embeddings and labels. Azure AI Foundry supports various machine learning algorithms.
   • Use tools like Azure Machine Learning to manage and optimize your training process.

4. Deploy the Model

   • Once your classifier is trained, deploy it as an Azure AI service. This allows you to use it for real-time or batch predictions.
   • Ensure the deployment is scalable based on your needs.

5. Generate Embeddings for Azure OpenAI Data

   • When you receive embeddings from Azure OpenAI, ensure they are compatible with the embeddings generated from your training data.
   • This step is crucial for maintaining consistency in your model's predictions.

6. Classify the Embeddings

   • Use the deployed model to classify the embeddings provided by Azure OpenAI as positive or negative.
   • The deployed classifier will process these embeddings and return the predicted labels.

Here’s a simplified workflow using Python and Azure Machine Learning:

from azureml.core import Workspace, Dataset, Datastore
from azureml.train.automl import AutoMLConfig
from azureml.core.experiment import Experiment

# Initialize your workspace
ws = Workspace.from_config()

# Load your training data
datastore = Datastore.get(ws, 'workspaceblobstore')
dataset = Dataset.Tabular.from_delimited_files(path=(datastore, 'data/training_data.csv'))

# Configure AutoML
automl_config = AutoMLConfig(
    task='classification',
    training_data=dataset,
    label_column_name='label',
    primary_metric='accuracy',
    compute_target='your_compute_target'
)

# Submit the experiment
experiment = Experiment(ws, 'classify_embeddings')
run = experiment.submit(automl_config)
run.wait_for_completion()

# Deploy the best model
best_run, fitted_model = run.get_output()
model = run.register_model(model_name='embedding_classifier', model_path='outputs/')

Key Points:

• Utilize Azure AI Foundry’s capabilities to preprocess data, train models, and deploy them.
• Ensure compatibility between training embeddings and embeddings from Azure OpenAI.
• Use Azure Machine Learning for efficient model training and deployment.

Some links
AI Foundry Docs https://learn.microsoft.com/azure/ai-studio
AI Foundry SDK https://learn.microsoft.com/azure/ai-studio/how-to/develop/sdk-overview
Azure ML Docs https://learn.microsoft.com/azure/machine-learning
Learning Azure ML https://learn.microsoft.com/azure/machine-learning/tutorial-azure-ml-in-a-day

[eager siren]

do you mean setting up training pipeline in Azure Machine Learning resource in the second point? I cant find any option for it in Azure AI foundry