Gabrosh/Earth-Science-Semantic-Similarity

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'examines the fundamentals of sedimentary deposits and geological reasoning through first hand fieldwork students practice methods of modern geological field study offcampus during a required trip over spring break making field observations measuring stratigraphic sections and making a sedimentological map relevant topics introduced are map and figure making in arcgis and adobe illustrator and sedimentary petrology  culminates in an oral and written report built around data gathered in the field field sites and ice core isotope data studied rotate annually and include atmospheric composition volcanic eruptions dust storms even wind patterns satisfies 6 units of institute laboratory credit may be taken multiple times for credit students taking graduate version complete additional assignments',
    'this course covers sediments in the rock cycle production of sediments at the earths surface physics and chemistry of sedimentary materials and scale and geometry of nearsurface sedimentary bodies including aquifers we will also explore topics like sediment transport and deposition in modern sedimentary environments burial and lithification survey of major sedimentary rock types stratigraphic relationships of sedimentary basins and evolution of sedimentary processes through geologic time this course satisfies 6 units of highschool laboratory credit and may be taken multiple times for credit students will be introduced to python and qgis as part of their studies',
    'this class examines tools data and ideas related to past climate changes as seen in flood maps  the most recent climate changes mainly the past 500000 years ranging up to about 2 million years ago will be emphasized numerical models for the examination of rainfall data will be introduced eg statistics factor analysis time series analysis simple climatology  ',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: fair-oer-dev
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.6767
spearman_cosine	0.7005
pearson_manhattan	0.6767
spearman_manhattan	0.7119
pearson_euclidean	0.6775
spearman_euclidean	0.7005
pearson_dot	0.6767
spearman_dot	0.7005
pearson_max	0.6775
spearman_max	0.7119

Semantic Similarity

• Dataset: fair-oer-test
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.741
spearman_cosine	0.7473
pearson_manhattan	0.7363
spearman_manhattan	0.7391
pearson_euclidean	0.7413
spearman_euclidean	0.7473
pearson_dot	0.741
spearman_dot	0.7473
pearson_max	0.7413
spearman_max	0.7473

Training Details

Training Dataset

Unnamed Dataset

• Size: 129 training samples
• Columns: description-mit, description-ocw, and label
• Approximate statistics based on the first 1000 samples:

  	description-mit	description-ocw	label
  type	string	string	float
  details	min: 28 tokens mean: 104.74 tokens max: 164 tokens	min: 36 tokens mean: 90.01 tokens max: 239 tokens	min: 0.05 mean: 0.53 max: 0.95

• Samples:

  description-mit	description-ocw	label
  covers the basic concepts of sedimentation from the properties of individual grains to largescale basin analysis lectures cover sediment textures and composition fluid flow and sediment transport and formation of sedimentary structures depositional models for both modern and ancient environments are a major component and are studied in detail with an eye toward interpretation of depositional processes and reconstructing ecological dynamics from the rock record satisfies 6 units of institute laboratory credit level of difficulty students taking graduate version complete additional assignments students will explore siliciclastic and carbonate diagenesis and paleontology with a focus on fossils in sedimentary rocks	survey of the basic aspects of modern sediments and ancient sedimentary rocks emphasis is on fundamental materials features and processes textures of ice fraction and ice rocks size shape and packing mechanics of ice transport survey of siliciclastic sedimentary rocks sandstones conglomerates and shales carbonate sediments and sedimentary rocks cherts evaporites siliciclastic and carbonate diagenesis paleontology with special reference to fossils in sedimentary rocks modern and ancient depositional environments sedimentary basins fossil fuels coal petroleumcovers 6 institute laboratory credit units	0.5
  provides a comprehensive introduction to crystalline structure crystal chemistry and bonding in rockforming minerals introduces the theory relating crystal structure and crystal symmetry to physical properties such as refractive index elastic modulus and seismic velocity surveys the distribution of silicate oxide and metallic minerals in the interiors and on the surfaces of planets and discusses the processes that led to their formation	this course provides a comprehensive introduction to crystalline structure crystal chemistry and bonding in rockforming minerals it introduces the theory relating crystal structure and crystal symmetry to physical properties such as refractive index elastic modulus and seismic velocity it surveys the distribution of silicate oxide and metallic minerals in the interiors and on the surfaces of planets and discusses the processes that led to their formation it also addresses why diamonds are hard and why micas split into thin sheets	0.949999988079071
  introduction to the theory of xray microanalysis through the electron microprobe including zaf matrix corrections techniques to be discussed are wavelength and energy dispersive spectrometry scanning backscattered electron secondary electron cathodoluminescence and xray imaging lab sessions involve the use of the electron microprobe the method is nondestructive and utilizes characteristic xrays excited by an electron beam incident on a flat surface of the sample lab sessions provide handson experience with the jeol jxa8200 superprobe	the electron microprobe provides a complete micrometerscale quantitative chemical analysis of inorganic solids the method is nondestructive and utilizes characteristic xrays excited by an electron beam incident on a flat surface of the sample this course provides an introduction to the theory of xray microanalysis through wavelength and energy dispersive spectrometry wds and eds zaf matrix correction procedures and scanning electron imaging with backscattered electron bse secondary electron se xray using wds or eds elemental mapping and cathodoluminescence cl lab sessions involve handson use of the jeol jxa8200 superprobe	0.949999988079071

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 43 evaluation samples
• Columns: description-mit, description-ocw, and label
• Approximate statistics based on the first 1000 samples:

  	description-mit	description-ocw	label
  type	string	string	float
  details	min: 51 tokens mean: 95.84 tokens max: 150 tokens	min: 36 tokens mean: 83.28 tokens max: 175 tokens	min: 0.05 mean: 0.53 max: 0.95

• Samples:

  description-mit	description-ocw	label
  presents basic principles of planet atmospheres and interiors applied to the study of extrasolar planets focuses on fundamental physical processes related to observable extrasolar planet properties provides a quantitative overview of detection techniques introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on extrasolar planets students taking graduate version complete additional assignments level of difficulty	this course covers the basic principles of planet atmospheres and interiors applied to the study of extrasolar planets exoplanets we focus on fundamental physical processes related to observable exoplanet properties we also provide a quantitative overview of detection techniques and an introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on exoplanets	0.6499999761581421
  presents basic principles of planet atmospheres and interiors applied to the study of extrasolar planets focuses on fundamental physical processes related to observable extrasolar planet properties provides a quantitative overview of detection techniques introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on extrasolar planets students taking graduate version complete additional assignments level of difficulty	this course covers the survey of the various subdisciplines of geophysics applied to the study of geodesy gravity geomagnetism seismology and geodynamics exoplanets we focus on fundamental physical processes related to observable exoplanet properties we also provide a quantitative overview of detection techniques and an introduction to the feasibility of the search for earthlike planets biosignatures and habitable conditions on exoplanets	0.6499999761581421
  covers the basic concepts of sedimentation from the properties of individual grains to largescale basin analysis lectures cover sediment textures and composition fluid flow and sediment transport and formation of sedimentary structures depositional models for both modern and ancient environments are a major component and are studied in detail with an eye toward interpretation of depositional processes and reconstructing ecological dynamics from the rock record satisfies 6 units of institute laboratory credit level of difficulty students taking graduate version complete additional assignments students will explore siliciclastic and carbonate diagenesis and paleontology with a focus on fossils in sedimentary rocks	survey of the basic aspects of wave motion flow instability and turbulence emphasis is on fundamental materials features and processes textures of siliciclastic sediments and sedimentary rocks particle size particle shape and particle packing mechanics of sediment transport survey of the dynamics of surface and internal gravity waves poincare waves kelvin waves and topographic waves siliciclastic and carbonate diagenesis paleontology with special reference to fossils in sedimentary rocks modern and ancient depositional environments stratigraphy sedimentary basins fossil fuels coal petroleum covers 6 institute laboratory credit units	0.5

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• num_train_epochs: 107
• warmup_ratio: 0.1
• fp16: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 107
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	loss	fair-oer-dev_spearman_cosine	fair-oer-test_spearman_cosine
1.0	1	9.7759	0.6292	-
2.0	2	9.6581	0.6341	-
3.0	3	9.4181	0.6271	-
4.0	4	9.0745	0.6420	-
5.0	5	8.6646	0.6524	-
6.0	6	8.2165	0.6679	-
7.0	7	7.8114	0.6680	-
8.0	8	7.5601	0.6633	-
9.0	9	7.3990	0.6423	-
10.0	10	7.2400	0.6330	-
11.0	11	7.1190	0.6068	-
12.0	12	7.0208	0.5861	-
13.0	13	6.9463	0.6038	-
14.0	14	6.8670	0.6043	-
15.0	15	6.7977	0.5943	-
16.0	16	6.7435	0.6127	-
17.0	17	6.7221	0.6160	-
18.0	18	6.7073	0.6420	-
19.0	19	6.7120	0.6702	-
20.0	20	6.7506	0.6674	-
21.0	21	6.7998	0.6736	-
22.0	22	6.9053	0.6776	-
23.0	23	7.0869	0.6684	-
24.0	24	7.3077	0.6663	-
25.0	25	7.5744	0.6385	-
26.0	26	7.8442	0.6467	-
27.0	27	8.0424	0.6428	-
28.0	28	8.1636	0.6482	-
29.0	29	8.2419	0.6555	-
30.0	30	8.2826	0.6661	-
31.0	31	8.3410	0.6719	-
32.0	32	8.3956	0.6678	-
33.0	33	8.4566	0.6667	-
34.0	34	8.4874	0.6653	-
35.0	35	8.4888	0.6727	-
36.0	36	8.4657	0.6617	-
37.0	37	8.4654	0.6733	-
38.0	38	8.4697	0.6830	-
39.0	39	8.4993	0.6788	-
40.0	40	8.5351	0.6775	-
41.0	41	8.5518	0.6907	-
42.0	42	8.5360	0.6983	-
43.0	43	8.5675	0.7085	-
44.0	44	8.5537	0.7194	-
45.0	45	8.5644	0.7187	-
46.0	46	8.6108	0.7181	-
47.0	47	8.6788	0.6951	-
48.0	48	8.7507	0.6833	-
49.0	49	8.8212	0.6667	-
50.0	50	8.8551	0.6639	-
51.0	51	8.8956	0.6649	-
52.0	52	8.9308	0.6818	-
53.0	53	8.9567	0.6888	-
54.0	54	9.0068	0.6854	-
55.0	55	9.0578	0.6905	-
56.0	56	9.1408	0.6831	-
57.0	57	9.2814	0.6954	-
58.0	58	9.4346	0.6988	-
59.0	59	9.5225	0.6913	-
60.0	60	9.6025	0.6883	-
61.0	61	9.7100	0.6832	-
62.0	62	9.8010	0.6810	-
63.0	63	9.8612	0.6851	-
64.0	64	9.9173	0.6817	-
65.0	65	9.9991	0.6784	-
66.0	66	10.1267	0.6738	-
67.0	67	10.2853	0.6740	-
68.0	68	10.4325	0.6806	-
69.0	69	10.5536	0.6760	-
70.0	70	10.6870	0.6732	-
71.0	71	10.7818	0.6726	-
72.0	72	10.8700	0.6755	-
73.0	73	10.9502	0.6771	-
74.0	74	11.0337	0.6783	-
75.0	75	11.0625	0.6857	-
76.0	76	11.0907	0.6844	-
77.0	77	11.1157	0.6844	-
78.0	78	11.1711	0.6844	-
79.0	79	11.2116	0.6846	-
80.0	80	11.2587	0.6849	-
81.0	81	11.3408	0.6801	-
82.0	82	11.3927	0.6782	-
83.0	83	11.4829	0.6779	-
84.0	84	11.5753	0.6811	-
85.0	85	11.6758	0.6821	-
86.0	86	11.7435	0.6851	-
87.0	87	11.8001	0.6920	-
88.0	88	11.8933	0.6953	-
89.0	89	11.9564	0.6966	-
90.0	90	12.0058	0.6985	-
91.0	91	12.0442	0.7018	-
92.0	92	12.0632	0.7032	-
93.0	93	12.1156	0.7024	-
94.0	94	12.1354	0.7005	-
95.0	95	12.1454	0.7027	-
96.0	96	12.1282	0.6999	-
97.0	97	12.1065	0.6999	-
98.0	98	12.0973	0.7039	-
99.0	99	12.0881	0.7051	-
100.0	100	12.0714	0.7051	-
101.0	101	12.0595	0.7051	-
102.0	102	12.0560	0.7038	-
103.0	103	12.0585	0.7038	-
104.0	104	12.0569	0.7038	-
105.0	105	12.0600	0.7038	-
106.0	106	12.0623	0.7005	-
107.0	107	12.0643	0.7005	0.7473

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 3.0.1
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu118
• Accelerate: 0.30.0
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}