debug: Add comprehensive logging to diagnose 4-token issue

PROBLEM:
Model generates only 4 tokens and produces no valid NDJSON patches.
Need to see what's actually being generated before filtering.

DIAGNOSTIC LOGGING ADDED:

1. Prompt Logging:
- Full prompt length
- First 500 chars (shows chat template opening)
- Last 200 chars (shows generation prompt start)
- Verifies apply_chat_template() output is correct

2. Generation Config Logging:
- max_new_tokens value
- EOS token ID
- Pad token ID
- Confirms generation settings

3. Raw Token Logging:
- Every token chunk as it arrives (debug level)
- Shows exact model output before any processing

4. Raw Line Logging:
- Every complete line before heuristic filter
- Shows what gets filtered and why

5. Buffer State Logging:
- Unparsed buffer contents after generation
- Detects partial/incomplete lines

This will reveal:
- Is the chat template correct?
- Is the model hitting EOS immediately?
- Is the model generating prose instead of JSON?
- Are we filtering out valid output by mistake?

app/services/structured_summarizer.py

@@ -444,6 +444,14 @@ Rules:
             # Build prompt
             full_prompt = self._build_prompt(text, style)
 
+            # DEBUG: Log the actual prompt being sent to model
+            logger.info("=" * 80)
+            logger.info("🔍 DEBUG: Full prompt being sent to model:")
+            logger.info(f"Prompt length: {len(full_prompt)} chars")
+            logger.info(f"First 500 chars:\n{full_prompt[:500]}")
+            logger.info(f"Last 200 chars:\n{full_prompt[-200:]}")
+            logger.info("=" * 80)
+
             # Tokenize
             inputs = self.tokenizer(full_prompt, return_tensors="pt")
             inputs = {k: v.to(self.model.device) for k, v in inputs.items()}
@@ -467,6 +475,13 @@ Rules:
                 "eos_token_id": self.tokenizer.eos_token_id,
             }
 
+            # DEBUG: Log generation config
+            logger.info(f"🎛️ Generation config:")
+            logger.info(f"  max_new_tokens: {max_new_tokens}")
+            logger.info(f"  do_sample: False (deterministic)")
+            logger.info(f"  eos_token_id: {self.tokenizer.eos_token_id}")
+            logger.info(f"  pad_token_id: {gen_kwargs['pad_token_id']}")
+
             # Start generation in background thread
             generation_thread = threading.Thread(
                 target=self.model.generate, kwargs=gen_kwargs, daemon=True
@@ -485,6 +500,9 @@ Rules:
                     token_count += 1
                     buffer += text_chunk
 
+                    # DEBUG: Log every raw token chunk
+                    logger.debug(f"🔤 Token #{token_count}: {repr(text_chunk)}")
+
                     # Process complete lines
                     while "\n" in buffer:
                         line, buffer = buffer.split("\n", 1)
@@ -493,6 +511,9 @@ Rules:
                         if not line:
                             continue
 
+                        # DEBUG: Log every line BEFORE filtering
+                        logger.info(f"📄 Raw line (at token #{token_count}): {line[:100]}...")
+
                         # Heuristic: skip anything that clearly isn't a JSON patch object
                         # This filters out lines like "#include <bits/stdc++.h>" or random prose.
                         if not line.startswith("{") or "op" not in line:
@@ -546,6 +567,12 @@ Rules:
             # Wait for generation to complete
             generation_thread.join()
 
+            # DEBUG: Log what's left in the buffer (partial line)
+            if buffer.strip():
+                logger.warning(f"🗑️ Unparsed buffer remaining: {repr(buffer[:200])}")
+            else:
+                logger.info("✅ Buffer was fully consumed (no partial lines)")
+
             logger.info(
                 f"🏁 Model generation completed: {token_count} tokens, "
                 f"done_received={done_received}"