We're in an AI gold rush. Businesses everywhere are scrambling to integrate AI, and for most, this means one thing: paying for API calls to a large, generic, third-party model.
This is the AI Consumer model. It's fast, it's easy, but it's fundamentally a rental. You're renting intelligence. You're subject to their pricing, their downtime, and their "one-size-fits-all" logic. Your data is processed on their servers, and the model you're using is the same one your competitors are using.
The alternative? Become an AI Producer.
An AI Producer doesn't just use AI; they create specialized AI assets. They build models that are custom-trained on their own data, speak their company's unique language, and run securely on their own infrastructure.
This isn't a fantasy. This is what the Flowork platform is built for. Today, we're pulling back the curtain to show you the two core services that make this possible: the DatasetManagerService and the AITrainingService.
- The Foundation: DatasetManagerService You can't build a custom model without custom data. Before you can "produce" AI, you need a factory for your raw materials. This is the job of the DatasetManagerService.
Its role is to ingest, validate, format, and securely store the datasets you'll use for fine-tuning. A model is only as good as its data, and this service ensures your data is clean, secure, and ready for training.
Here is a conceptual look at its (Go-based) service definition. This is the blueprint for managing your most valuable asset: your data.
`
/* * DatasetManagerService
- Manages all user-provided datasets for training and fine-tuning.
- This service is responsible for ingestion, validation, and formatting. */ package services
import (
"context"
"io"
)
// DatasetFormat defines the expected structure of the data.
type DatasetFormat string
const (
FormatJSONL DatasetFormat = "jsonl"
FormatCSV DatasetFormat = "csv"
FormatText DatasetFormat = "text"
)
// Dataset represents the metadata for a stored dataset.
type Dataset struct {
ID string json:"id"
UserID string json:"user_id"
Name string json:"name"
Description string json:"description"
Format DatasetFormat json:"format"
EntryCount int json:"entry_count"
Status string json:"status" // e.g., "uploading", "validating", "ready"
CreatedAt int64 json:"created_at"
}
// DatasetManagerService defines the interface for managing training data.
type DatasetManagerService interface {
// CreateNewDataset initializes a new dataset record and returns an ID
// for uploading the raw data file.
CreateNewDataset(ctx context.Context, userID, name, description string, format DatasetFormat) (*Dataset, error)
// UploadDatasetData streams the raw data file (e.g., a .jsonl file)
// to be associated with a dataset ID.
UploadDatasetData(ctx context.Context, userID, datasetID string, data io.Reader) error
// ValidateDataset triggers an asynchronous job to validate the format
// and quality of the uploaded data.
ValidateDataset(ctx context.Context, userID, datasetID string) error
// GetDataset retrieves the metadata for a specific dataset.
GetDataset(ctx context.Context, userID, datasetID string) (*Dataset, error)
// ListDatasets returns all datasets available for a user.
ListDatasets(ctx context.Context, userID string) ([]*Dataset, error)
// DeleteDataset removes the dataset and its associated data files.
DeleteDataset(ctx context.Context, userID, datasetID string) error
}`
Why this matters: Instead of your data being scattered across folders or buckets, this service provides a single, secure, and structured registry. When you're ready to train, you don't point to a file; you point to a validated DatasetID.
2. The Factory: AITrainingService
With your data prepared by the DatasetManagerService, it's time to manufacture your custom AI. This is the job of the AITrainingService.
This service is the "factory." It takes three things:
- A Base Model (e.g., llama3-8b, mistral-7b, or a domain-specific model).
- A Dataset ID (from the DatasetManagerService).
- Hyperparameters (the "settings" for the training job, like learning rate).
It then provisions the necessary resources (on the user's own server/infrastructure) and runs the fine-tuning job. Critically, this process is asynchronous. Training can take hours or even days. The service manages this entire lifecycle, from "Pending" to "Succeeded," and delivers a new, specialized model endpoint when it's done.
Here's the conceptual service definition:
`
/* * AITrainingService
- Manages the entire lifecycle of a model fine-tuning job.
- It coordinates with the DatasetManager and the underlying AI engine. */ package services
import (
"context"
)
// FineTuneJob represents the state of a single training run.
type FineTuneJob struct {
ID string json:"id"
UserID string json:"user_id"
JobName string json:"job_name"
BaseModelID string json:"base_model_id" // e.g., "llama3-8b"
DatasetID string json:"dataset_id" // From DatasetManagerService
Status string json:"status" // e.g., "pending", "running", "succeeded", "failed"
Hyperparameters map[string]string json:"hyperparameters" // e.g., {"learning_rate": "0.0001"}
CreatedAt int64 json:"created_at"
CompletedAt int64 json:"completed_at,omitempty"
Error string json:"error,omitempty"
ResultModelID string json:"result_model_id,omitempty" // The ID of the new, fine-tuned model
}
// AITrainingService defines the interface for creating and managing fine-tune jobs.
type AITrainingService interface {
// StartFineTuneJob queues a new training job.
// This returns immediately with the job's metadata.
StartFineTuneJob(ctx context.Context, userID, jobName, baseModelID, datasetID string, params map[string]string) (*FineTuneJob, error)
// GetJobStatus checks the current status of a running job.
GetJobStatus(ctx context.Context, userID, jobID string) (*FineTuneJob, error)
// ListJobs returns all training jobs for a user.
ListJobs(ctx context.Context, userID string) ([]*FineTuneJob, error)
// CancelJob attempts to stop a "pending" or "running" job.
CancelJob(ctx context.Context, userID, jobID string) error
// GetTrainedModelEndpoint returns the inference endpoint/details
// for a successfully completed job.
GetTrainedModelEndpoint(ctx context.Context, userID, resultModelID string) (string, error)
}`
The "Producer" Workflow: A Practical Example
So, how does this all come together?
- Goal: You want to create a customer support bot that knows your 50 products and speaks in your company's friendly, casual tone.
- Data Prep: You gather 500 examples of great customer interactions. You format this as a .jsonl file.
- Step 1: Upload (DatasetManager) o You call CreateNewDataset(...) with the name "Support Bot v1". o You call UploadDatasetData(...) to upload your .jsonl file. o You call ValidateDataset(...). The service confirms all 500 entries are valid. Your dataset ds_abc123 is now "Ready".
- Step 2: Train (AITrainingService) o You call StartFineTuneJob(...) with: baseModelID: "mistral-7b" datasetID: "ds_abc123" hyperparameters: {"epochs": "3"} o The service returns job_xyz789 with Status: "Pending".
- Step 3: Monitor o You poll GetJobStatus(...) for job_xyz789. You see it change from "Pending" to "Running". o Two hours later, you check again. The status is "Succeeded", and the ResultModelID is model_sup_v1.
- Step 4: Produce o You call GetTrainedModelEndpoint(...) for model_sup_v1. o You now have an API endpoint for your model.
You are no longer a consumer. You are a producer.
You have just manufactured an AI asset that is 100% yours. It's more accurate for your use case, it's more efficient, and it runs on your hardware, completely securing your customer data.
Stop renting. Start owning.
Check out the architecture for yourself.
https://github.com/flowork-dev/flowork-platform
DOC : https://docs.flowork.cloud/
WEBSITE : https://flowork.cloud/
Top comments (0)