DEV Community

TS
TS

Posted on

cell-04-SBERT TEXT REPRESENTATION 

# ============================================================
# SBERT TEXT REPRESENTATION
# ============================================================

from sentence_transformers import SentenceTransformer

SBERT_MODEL_NAME = "all-MiniLM-L6-v2"

sbert_model = SentenceTransformer(SBERT_MODEL_NAME)

train_texts = train_df["Combined_Text"].fillna("").astype(str).tolist()
test_texts = test_df["Combined_Text"].fillna("").astype(str).tolist()

X_train_sbert = sbert_model.encode(
    train_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)

X_test_sbert = sbert_model.encode(
    test_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)

print("SBERT dimension:", X_train_sbert.shape[1])
print("Training embeddings:", X_train_sbert.shape)
print("Test embeddings:", X_test_sbert.shape)


# ============================================================
# HYBRID FEATURE FUSION
# ============================================================

def fuse_features(sbert_features, structured_features):
    return np.hstack([
        sbert_features,
        structured_features
    ])
Enter fullscreen mode Exit fullscreen mode

SBERT TEXT REPRESENTATION

This cell converts the combined text feature into numerical semantic embeddings using SBERT. These embeddings will later be combined with the structured features to create the hybrid feature representation.

1. Import SBERT 

from sentence_transformers import SentenceTransformer
Enter fullscreen mode Exit fullscreen mode
  • Imports the SentenceTransformer class from the sentence_transformers library.
  • SBERT stands for Sentence-BERT.
  • It converts text into dense numerical vectors called embeddings.
  • These embeddings capture the semantic meaning and relationships between words/text, rather than simply representing individual words.

Viva question: Why did you use SBERT?

Answer:

“I used SBERT to convert the combined textual information into dense semantic embeddings. This allows the ML models to capture relationships between product names, companies, and countries that may not be captured by simple categorical or numerical encoding.”

2. Select the SBERT model 

SBERT_MODEL_NAME = "all-MiniLM-L6-v2"
Enter fullscreen mode Exit fullscreen mode
  • Stores the name of the pretrained SBERT model in a variable.
  • "all-MiniLM-L6-v2" is a lightweight sentence-transformer model.
  • It provides a good balance between semantic representation, computational cost, and embedding size.

The important point is that you are using a pretrained model, rather than training SBERT from scratch on your PCF dataset.

Viva question: Why this particular model?

Answer:

“I selected all-MiniLM-L6-v2 because it is a relatively lightweight pretrained sentence-transformer that provides useful semantic embeddings while keeping computational requirements manageable.”

If examiner asks: Why not a larger model?

“A larger transformer could potentially provide richer representations, but it would increase computational cost. For this project, I prioritised a practical balance between semantic representation and efficiency.”

3. Load the pretrained model 

sbert_model = SentenceTransformer(SBERT_MODEL_NAME)
Enter fullscreen mode Exit fullscreen mode

This creates the SBERT model object.

The process is:

"all-MiniLM-L6-v2"
        ↓
SentenceTransformer()
        ↓
Pretrained SBERT model loaded
Enter fullscreen mode Exit fullscreen mode

The model is now ready to convert text into embeddings.

Important viva point

You are not fitting SBERT on the PCF dataset here.

You are using an already pretrained language model to generate representations.

That is different from:

model.fit(...)
Enter fullscreen mode Exit fullscreen mode

used by conventional ML models.

4. Get training text 

train_texts = train_df["Combined_Text"].fillna("").astype(str).tolist()
Enter fullscreen mode Exit fullscreen mode

Let's break this line into four operations.

train_df["Combined_Text"]

Selects the Combined_Text column from the training dataset.

Earlier, you created something like:

Product: Laptop | Company: Dell | Country: USA
Enter fullscreen mode Exit fullscreen mode

So each row contains a combined textual representation.

.fillna("") 

.fillna("")
Enter fullscreen mode Exit fullscreen mode

Replaces missing text values with an empty string.

For example:

NaN
Enter fullscreen mode Exit fullscreen mode

becomes:

""
Enter fullscreen mode Exit fullscreen mode

This prevents missing values from causing problems during text encoding.

.astype(str) 

.astype(str)
Enter fullscreen mode Exit fullscreen mode

Converts every value into a string.

SBERT expects textual input, so this ensures the input has the appropriate type.

.tolist() 

.tolist()
Enter fullscreen mode Exit fullscreen mode

Converts the pandas Series into a Python list.

For example:

Pandas Series
     ↓
Python list
     ↓
["Product: Laptop | Company: Dell | Country: USA",
 "Product: Phone | Company: Apple | Country: USA",
 ...]
Enter fullscreen mode Exit fullscreen mode

This list is then passed to SBERT.

5. Get test text 

test_texts = test_df["Combined_Text"].fillna("").astype(str).tolist()
Enter fullscreen mode Exit fullscreen mode

This performs exactly the same preparation for the test dataset.

The important difference is:

train_df → train_texts
test_df  → test_texts
Enter fullscreen mode Exit fullscreen mode

The two datasets remain separate.

Viva question: Why do you encode train and test separately?

Answer:

“I encode the training and test observations separately so that each observation receives its embedding without mixing the datasets. The same pretrained SBERT model is used for both, ensuring that the representations are generated in a consistent way.”

6. Generate training embeddings 

X_train_sbert = sbert_model.encode(
    train_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)
Enter fullscreen mode Exit fullscreen mode

This is the main SBERT operation.

The model takes the training text and converts every text record into a numerical vector.

Conceptually:

Combined_Text
      ↓
     SBERT
      ↓
Dense numerical embedding
Enter fullscreen mode Exit fullscreen mode

For example, conceptually:

"Product: Laptop | Company: Dell | Country: USA"
                     ↓
        [0.12, -0.34, 0.56, ..., 0.08]
Enter fullscreen mode Exit fullscreen mode

The actual vector contains many numerical dimensions.

7. show_progress_bar=True 

show_progress_bar=True
Enter fullscreen mode Exit fullscreen mode

Displays the encoding progress while SBERT processes the observations.

For example:

Encoding: 100% |████████████| ...
Enter fullscreen mode Exit fullscreen mode

This does not affect the model's predictions.

It is simply useful for monitoring a potentially time-consuming operation.

Viva question: Does this parameter affect accuracy?

Answer:

“No. It only controls whether a progress bar is displayed during encoding.”

8. convert_to_numpy=True 

convert_to_numpy=True
Enter fullscreen mode Exit fullscreen mode

Requests the embeddings as a NumPy array.

This is important because later you use:

np.hstack(...)
Enter fullscreen mode Exit fullscreen mode

to combine the SBERT features with structured features.

So the pipeline becomes:

SBERT
 ↓
NumPy embedding matrix
 ↓
np.hstack()
 ↓
Hybrid features
Enter fullscreen mode Exit fullscreen mode

9. Generate test embeddings 

X_test_sbert = sbert_model.encode(
    test_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)
Enter fullscreen mode Exit fullscreen mode

The same pretrained SBERT model converts the test texts into embeddings.

Notice that you do not train or refit SBERT using the test data.

You simply apply the already loaded model.

This is important for maintaining a clean evaluation process.

Viva question: Is this data leakage?

Answer:

“No, because the SBERT model is pretrained independently and is not fitted on the PCF test data. The test text is only passed through the pretrained encoder to generate representations.”

A more cautious answer if the examiner is strict:

“The encoder itself is pretrained externally rather than learned from this dataset. I use it only as a fixed representation model and do not fine-tune it using the PCF test set.”

10. Print SBERT dimension 

print("SBERT dimension:", X_train_sbert.shape[1])
Enter fullscreen mode Exit fullscreen mode

X_train_sbert.shape contains:

(number of observations, number of embedding dimensions)
Enter fullscreen mode Exit fullscreen mode

For example:

(800, 384)
Enter fullscreen mode Exit fullscreen mode

Then:

.shape[1]
Enter fullscreen mode Exit fullscreen mode

selects the second dimension:

384
Enter fullscreen mode Exit fullscreen mode

So this prints the number of features generated by SBERT.

For all-MiniLM-L6-v2, the embedding dimension is typically 384.

11. Print training embedding shape 

print("Training embeddings:", X_train_sbert.shape)
Enter fullscreen mode Exit fullscreen mode

This shows:

(number of training observations, embedding dimensions)
Enter fullscreen mode Exit fullscreen mode

For example:

Training embeddings: (800, 384)
Enter fullscreen mode Exit fullscreen mode

This allows you to verify that the number of embeddings corresponds to the number of training observations.

12. Print test embedding shape 

print("Test embeddings:", X_test_sbert.shape)
Enter fullscreen mode Exit fullscreen mode

Similarly, this verifies the dimensions of the test embeddings.

For example:

Test embeddings: (200, 384)
Enter fullscreen mode Exit fullscreen mode

The important requirement is:

number of training embeddings = number of training rows

number of test embeddings = number of test rows
Enter fullscreen mode Exit fullscreen mode

HYBRID FEATURE FUSION

Now you combine the semantic text representation with the structured representation.

def fuse_features(sbert_features, structured_features):
Enter fullscreen mode Exit fullscreen mode

This defines a function called fuse_features.

It accepts two inputs:

sbert_features
structured_features
Enter fullscreen mode Exit fullscreen mode

sbert_features

These are the SBERT embeddings.

For example:

384 dimensions
Enter fullscreen mode Exit fullscreen mode

structured_features

These are the features you previously created from:

  • Year
  • Product weight
  • One-hot encoded categorical variables
  • Country target encoding

So you have two different types of information:

SBERT
→ semantic information

Structured features
→ explicit numerical/categorical information
Enter fullscreen mode Exit fullscreen mode

13. Horizontally concatenate the features 

return np.hstack([
    sbert_features,
    structured_features
])
Enter fullscreen mode Exit fullscreen mode

np.hstack() means horizontal stacking.

Suppose:

SBERT = 384 features
Structured = 10 features
Enter fullscreen mode Exit fullscreen mode

Then:

384 + 10 = 394 features
Enter fullscreen mode Exit fullscreen mode

The result becomes:

[ SBERT features | Structured features ]
Enter fullscreen mode Exit fullscreen mode

For every observation, the two feature vectors are placed side by side.

Conceptually:

SBERT
[0.12, -0.34, 0.56, ..., 0.08]
              +
Structured
[2023, 1.5, 0, 1, 0, ...]
              ↓
Hybrid
[0.12, -0.34, 0.56, ..., 0.08, 2023, 1.5, 0, 1, 0, ...]
Enter fullscreen mode Exit fullscreen mode

Why use hybrid features?

This is one of the most important viva questions.

Your reasoning is:

Text alone
   ↓
captures semantic information

Structured features alone
   ↓
captures explicit numerical/categorical information

Both together
   ↓
hybrid representation
   ↓
potentially richer information for ML
Enter fullscreen mode Exit fullscreen mode

Strong viva answer

“I used hybrid feature fusion because the two representations capture different types of information. SBERT captures semantic relationships within the combined text, while the structured features explicitly represent numerical and categorical information such as reporting year, product weight, industry, protocol, and country. By horizontally concatenating them, the downstream ML models can use both sources of information simultaneously.”

Very important distinction

Do not say:

“SBERT understands the exact carbon footprint.”

That would overclaim.

Instead say:

“SBERT provides a semantic representation of the textual information, which may contain information useful for predicting PCF.”

The PCF prediction itself is performed by your downstream ML model.

Overall pipeline for this section 

Product
Company
Country
   ↓
Combined_Text
   ↓
Pretrained SBERT
   ↓
384-dimensional semantic embeddings
   ↓
              ┌────────────────────┐
              │ Structured features│
              │ Year               │
              │ Weight             │
              │ Industry           │
              │ Protocol           │
              │ Country encoding   │
              └────────────────────┘
                         ↓
SBERT embeddings ──→ Feature Fusion ←── Structured features
                         ↓
                  Hybrid Features
                         ↓
                    ML Model
                         ↓
                  PCF Prediction
Enter fullscreen mode Exit fullscreen mode

Most likely examiner questions

Why SBERT instead of TF-IDF?

“TF-IDF mainly represents lexical importance based on word frequency, whereas SBERT produces dense contextual semantic embeddings. Since product and company descriptions can contain meaningful relationships beyond exact word matching, SBERT provides a richer semantic representation.”

Why combine SBERT with structured features?

“Because SBERT and structured features contain complementary information. SBERT captures semantic text information, while structured features preserve explicit numerical and categorical information.”

Why use the same SBERT model for train and test?

“To maintain a consistent feature space. The same pretrained encoder transforms both datasets, but the test data is not used to fit or fine-tune the encoder.”

What does np.hstack() do?

“It horizontally concatenates the two feature matrices, placing the SBERT and structured features side by side for each observation.”

What is the biggest limitation of this approach?

“SBERT was pretrained for general language representation, not specifically for PCF prediction. Therefore, its embeddings may not capture all domain-specific carbon-footprint relationships. The hybrid approach helps by adding explicit structured information, but it does not guarantee that every relevant PCF relationship is represented.”

Presentation wording

“In this stage, I convert the combined textual representation into semantic embeddings using the pretrained all-MiniLM-L6-v2 SBERT model. First, I prepare the training and test text by handling missing values and converting them into lists. I then pass both datasets through the same pretrained SBERT encoder and obtain numerical embeddings. I verify the dimensions of these embeddings to ensure that the transformation was successful. Finally, I define a feature-fusion function that horizontally combines the SBERT embeddings with the structured features. This creates a hybrid representation containing both semantic textual information and explicit numerical and categorical information, which is then used by the downstream machine-learning models.”

For your SBERT pipeline

  • Explain how to verify row alignment
  • Explain feature scaling after fusion

Top comments (0)