docs.md

- 2024_main_document_lvl
- 2024_main_paragraph_lvl

- 2023_main_document_lvl
- 2023_main_paragraph_lvl

- Embeddings convert pdfs
  - Para
  - Docs
- HNSW - Kmeans fast searcddh
- K means graphs based on the topics
- Check for similarity between our own db
  - Para
  - Docs
- Get The most important Ones
- Get the Unquine sentances like title & other content ?? - LLM think karun karel
- Search Google using the unquine searches --> get the top 3 and do the same check again --> result

### 1. Data Input:

- **Input Data:** Collect a diverse dataset of academic papers, articles, or textual content from various sources.
- **Format:** Ensure the data is in a consistent and machine-readable format, such as plain text or a format compatible with your chosen NLP library.

### 2. Data Cleaning:

- **Text Cleaning:**
  - Remove metadata, formatting, and irrelevant details.
  - Handle special characters, punctuation, and stopwords.

- **Normalization:**
  - Convert text to lowercase to ensure uniformity.

- **Tokenization:**
  - Tokenize the text into words or subword tokens.
  - **Libraries:**
    - For Python, you can use NLTK or spaCy for tokenization.

### 3. Embedding Generation:

- **Word Level Embeddings:**
  - Utilize pre-trained word embeddings like Word2Vec or GloVe.
  - **Libraries:**
    - For Word2Vec: Gensim library.
    - For GloVe: spaCy or gensim.

- **Paragraph Level Embeddings:**
  - Aggregate word embeddings using techniques like averaging or using Doc2Vec.
  - **Libraries:**
    - Gensim for Doc2Vec.

- **Document Level Embeddings:**
  - Consider using the average of paragraph embeddings or more advanced models.
  - **Libraries:**
    - spaCy or transformers library for more advanced models.

### 4. Pairwise Comparison:

- **Similarity Measures:**
  - Calculate cosine similarity, Jaccard similarity, or other relevant measures.
  - **Libraries:**
    - scikit-learn for cosine similarity.

### 5. Clustering:

- **K-Means Clustering:**
  - Partition documents into K clusters.
  - **Libraries:**
    - scikit-learn for K-Means.

- **Hierarchical Clustering:**
  - Build a hierarchy of clusters.
  - **Libraries:**
    - scipy.cluster.hierarchy for hierarchical clustering.

- **DBSCAN:**
  - Density-based clustering.
  - **Libraries:**
    - scikit-learn for DBSCAN.

### 6. Scoring System:

- **Threshold Setting:**
  - Establish a threshold for similarity scores to classify documents.
  - Determine the threshold through experimentation.

- **Scoring Logic:**
  - Develop a scoring system based on the results of pairwise comparison and clustering.
  - Decide on the scoring weights for each component.

### 7. Hybrid Approach:

- **Traditional Models:**
  - Use traditional similarity measures for efficiency.
  - Implement efficient algorithms for quick pairwise comparisons.

- **Large Language Models:**
  - Fine-tune or use pre-trained models for enhanced context understanding.
  - Hugging Face Transformers library for accessing pre-trained models.

- Fingerprinting Concept