SKH-NLP Learner¶

The Taxonomy Discovery task was modeled as a classification problem and addressed using two approaches. Initial experimentation was done on BERT, fine-tuned with various prompts in a classification setup. Further experimentation was conducted on LLaMA3-70B, with different prompt formulations for optimal results. Evaluation employed substring and Levenshtein distance functions to assess answer correctness.

Note

Hint

The original implementation is available at https://github.com/s-m-hashemi/llms4ol-2024-challenge repository.

Overview¶

Original implementation has used the GeoNames dataset was used, containing 476 (child, parent) pairs with 9 distinct parent classes, making it a 9-class classification problem. To train the BERT classifier, a negative dataset was generated by (1) reversing records (swapping parent and child) and (2) manipulating records (randomly replacing the parent with one of the other 8).

Methodological Summary:

BERT-Based Approach. Modeled as a multi-class problem with 9 classes using a single binary classifier iteratively for each class. The model was fine-tuned to determine whether an is-a relationship exists between a given (parent, child) pair.
LLaMA-Based Approach. Evaluation focused on prompt engineering using two concepts—classification (instance–class) and hierarchy (is-a, parent–child). In some cases, the model partially used class names (e.g., only part of “mountain, hill, rock”). To handle this, substring matching and Levenshtein distance were applied during evaluation to map outputs to the closest class titles.

Taxonomy Discovery (Zero-Shot)¶

Loading Ontological Data¶

We first load the GeoNames ontology and split the taxonomic edges into train and test sets.

from ontolearner import GeoNames, train_test_split

ontology = GeoNames()
ontology.load()
data = ontology.extract()

train_data, test_data = train_test_split(
    data,
    test_size=0.2,
    random_state=42,
)

Initialize Learner¶

Before defining the learner, choose the ontology learning task to perform. Available tasks have been described in LLMs4OL Paradigms. The task IDs are: term-typing, taxonomy-discovery, non-taxonomic-re.

task = "taxonomy-discovery"

Next, we configure the zero-shot taxonomy learner. This learner uses a generative LLM together with string-normalization strategies (e.g., Levenshtein distance) to map model outputs to ontology classes.

from ontolearner import SKHNLPZSLearner

llm_learner = SKHNLPZSLearner(
    model_name="Qwen/Qwen2.5-0.5B-Instruct",
    device="cpu",               # use "cuda" if you have a GPU
    max_new_tokens=16,
    save_path="./outputs/",     # directory or full file path for CSV
    verbose=True,
    normalize_mode="levenshtein",  # "none" | "substring" | "levenshtein" | "auto"
)

# Load the underlying LLM and prepare it for zero-shot taxonomy discovery
llm_learner.load(llm_id=llm_learner.model_name)

Learn and Predict¶

from ontolearner import evaluation_report

# Zero-shot setting: training data may be used only for label inventory or analysis;
# the main predictions are produced directly on the test split.
predicts = llm_learner.predict(test_data, task=task)

# Build gold-standard labels for evaluation
truth = llm_learner.tasks_ground_truth_former(data=test_data, task=task)

# Evaluate zero-shot taxonomy discovery performance
metrics = evaluation_report(y_true=truth, y_pred=predicts, task=task)
print(metrics)

Taxonomy Discovery (Supervised Fine-Tuning)¶

Loading Ontological Data¶

For supervised fine-tuning, we again use GeoNames and split the taxonomic relationships into train and test sets.

from ontolearner import GeoNames, train_test_split

ontology = GeoNames()
ontology.load()
data = ontology.extract()

train_data, test_data = train_test_split(
    data,
    test_size=0.2,
    random_state=42,
)

Initialize Learner¶

task = "taxonomy-discovery"

We then configure the supervised BERT-based learner. This learner fine-tunes a BERT-Large model using sequential prompts over (parent, child) pairs.

from ontolearner import SKHNLPSequentialFTLearner

bert_learner = SKHNLPSequentialFTLearner(
    model_name="bert-large-uncased",
    n_prompts=2,
    random_state=1403,
    device="cpu",  # Note: CPU training for BERT-Large is slow.
    output_dir="./results/",
    num_train_epochs=1,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs/",
    logging_steps=50,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

# Load the base BERT model and prepare it for supervised taxonomy discovery
bert_learner.load(llm_id=bert_learner.model_name)

Learn and Predict¶

from ontolearner import evaluation_report

# Fine-tune BERT on the taxonomic training data
bert_learner.fit(train_data, task=task)

# Predict taxonomic relations on the held-out test set
predicts = bert_learner.predict(test_data, task=task)

# Build gold-standard labels and evaluate
truth = bert_learner.tasks_ground_truth_former(data=test_data, task=task)
metrics = evaluation_report(y_true=truth, y_pred=predicts, task=task)
print(metrics)