SKH-NLP Learner ================ .. sidebar:: SKH-NLP Learner Examples * Taxonomy Discovery (Supervised Fine-Tuning): `llm_learner_skhnlp_sft_taxonomoy_discovery.py `_ * Taxonomy Discovery (Zero-Shot): `llm_learner_skhnlp_zs_taxonomoy_discovery.py `_ The Taxonomy Discovery task was modeled as a classification problem and addressed using two approaches. Initial experimentation was done on BERT, fine-tuned with various prompts in a classification setup. Further experimentation was conducted on LLaMA3-70B, with different prompt formulations for optimal results. Evaluation employed substring and Levenshtein distance functions to assess answer correctness. .. note:: Read more about the model at `SKH-NLP at LLMs4OL 2024 Task B: Taxonomy Discovery in Ontologies Using BERT and LLaMA 3 `_. .. hint:: The original implementation is available at `https://github.com/s-m-hashemi/llms4ol-2024-challenge `_ repository. Overview --------------------------------- Original implementation has used the GeoNames dataset was used, containing 476 (child, parent) pairs with 9 distinct parent classes, making it a 9-class classification problem. To train the BERT classifier, a negative dataset was generated by (1) reversing records (swapping parent and child) and (2) manipulating records (randomly replacing the parent with one of the other 8). Methodological Summary: - **BERT-Based Approach.** Modeled as a multi-class problem with 9 classes using a single binary classifier iteratively for each class. The model was fine-tuned to determine whether an *is-a* relationship exists between a given (parent, child) pair. - **LLaMA-Based Approach.** Evaluation focused on prompt engineering using two concepts—classification (instance–class) and hierarchy (is-a, parent–child). In some cases, the model partially used class names (e.g., only part of “mountain, hill, rock”). To handle this, substring matching and Levenshtein distance were applied during evaluation to map outputs to the closest class titles. Taxonomy Discovery (Zero-Shot) --------------------------------- Loading Ontological Data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We first load the GeoNames ontology and split the taxonomic edges into train and test sets. .. code-block:: python from ontolearner import GeoNames, train_test_split ontology = GeoNames() ontology.load() data = ontology.extract() train_data, test_data = train_test_split( data, test_size=0.2, random_state=42, ) Initialize Learner ~~~~~~~~~~~~~~~~~~~~~ Before defining the learner, choose the ontology learning task to perform. Available tasks have been described in `LLMs4OL Paradigms `_. The task IDs are: ``term-typing``, ``taxonomy-discovery``, ``non-taxonomic-re``. .. code-block:: python task = "taxonomy-discovery" Next, we configure the zero-shot taxonomy learner. This learner uses a generative LLM together with string-normalization strategies (e.g., Levenshtein distance) to map model outputs to ontology classes. .. code-block:: python from ontolearner import SKHNLPZSLearner llm_learner = SKHNLPZSLearner( model_name="Qwen/Qwen2.5-0.5B-Instruct", device="cpu", # use "cuda" if you have a GPU max_new_tokens=16, save_path="./outputs/", # directory or full file path for CSV verbose=True, normalize_mode="levenshtein", # "none" | "substring" | "levenshtein" | "auto" ) # Load the underlying LLM and prepare it for zero-shot taxonomy discovery llm_learner.load(llm_id=llm_learner.model_name) Learn and Predict ~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: python from ontolearner import evaluation_report # Zero-shot setting: training data may be used only for label inventory or analysis; # the main predictions are produced directly on the test split. predicts = llm_learner.predict(test_data, task=task) # Build gold-standard labels for evaluation truth = llm_learner.tasks_ground_truth_former(data=test_data, task=task) # Evaluate zero-shot taxonomy discovery performance metrics = evaluation_report(y_true=truth, y_pred=predicts, task=task) print(metrics) Taxonomy Discovery (Supervised Fine-Tuning) ------------------------------------------- Loading Ontological Data ~~~~~~~~~~~~~~~~~~~~~~~~~~~ For supervised fine-tuning, we again use GeoNames and split the taxonomic relationships into train and test sets. .. code-block:: python from ontolearner import GeoNames, train_test_split ontology = GeoNames() ontology.load() data = ontology.extract() train_data, test_data = train_test_split( data, test_size=0.2, random_state=42, ) Initialize Learner ~~~~~~~~~~~~~~~~~~~~~ Before defining the learner, choose the ontology learning task to perform. Available tasks have been described in `LLMs4OL Paradigms `_. The task IDs are: ``term-typing``, ``taxonomy-discovery``, ``non-taxonomic-re``. .. code-block:: python task = "taxonomy-discovery" We then configure the supervised BERT-based learner. This learner fine-tunes a BERT-Large model using sequential prompts over (parent, child) pairs. .. code-block:: python from ontolearner import SKHNLPSequentialFTLearner bert_learner = SKHNLPSequentialFTLearner( model_name="bert-large-uncased", n_prompts=2, random_state=1403, device="cpu", # Note: CPU training for BERT-Large is slow. output_dir="./results/", num_train_epochs=1, per_device_train_batch_size=8, per_device_eval_batch_size=8, warmup_steps=500, weight_decay=0.01, logging_dir="./logs/", logging_steps=50, eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, ) # Load the base BERT model and prepare it for supervised taxonomy discovery bert_learner.load(llm_id=bert_learner.model_name) Learn and Predict ~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: python from ontolearner import evaluation_report # Fine-tune BERT on the taxonomic training data bert_learner.fit(train_data, task=task) # Predict taxonomic relations on the held-out test set predicts = bert_learner.predict(test_data, task=task) # Build gold-standard labels and evaluate truth = bert_learner.tasks_ground_truth_former(data=test_data, task=task) metrics = evaluation_report(y_true=truth, y_pred=predicts, task=task) print(metrics)