Compare commits
4 commits
6968d3bf12
...
fd5afdd94f
| Author | SHA1 | Date | |
|---|---|---|---|
| fd5afdd94f | |||
| 37e4ed15cc | |||
| fcf7eb45d6 | |||
| bd2e21ea9c |
9 changed files with 163 additions and 16 deletions
3
.claude/knowledge-nominations.md
Normal file
3
.claude/knowledge-nominations.md
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
- [102326] /end-day: Terminal-Befehle für Markus immer mit `cd "<Pfad>" &&` liefern | Evidence: deploy.sh scheiterte aus ~ (No such file)
|
||||
- [102326] /end-day: Prod-DB hat kein alembic_version → Schemaänderungen per idempotentem ALTER TABLE beim Startup | Evidence: psql alembic_version fehlt, create_all ergänzt keine Spalten
|
||||
- [102326] /end-day: Sentinel-Hinweis bestätigt: Header-Dekodierung muss alle Segmente + Fehler abfangen, mehrere Aufrufer teilen das Muster → ein Helper | Evidence: bd2e21e
|
||||
|
|
@ -80,3 +80,16 @@ Rechnungen/
|
|||
- **Stand:** Phase 0 beginnt. RAG-Query am 102126 fehlgeschlagen (`qwen3-coder:30b` 404 auf NucBox-Ollama) — separat fixen.
|
||||
- **Phase 0 erledigt (102126 abends):** Live auf NucBox `~/invoice-check/` (Compose, `.env` aus Vault via `~/cstone-secrets/bin/with-secrets.sh invoice-check scripts/deploy.sh`). Postfach Strato `info@c-stone-dev.com`. Erstscan `POST /emails/scan-all/start` über 5.240 Mails, nach 90 s bereits 93 Treffer/34 PDFs abgelegt. Zugriff nur per SSH-Tunnel (`-L 5178:localhost:5178 -L 8005:localhost:8005`). **Befunde (offen, Scope Phase 1+):** kein Scheduler (Scan manuell), `/emails/*` ohne Auth (nur localhost), Aussteller-Parser liefert Wortfetzen, Steuer-Erkennung schlägt überall auf „ohne MwSt" an, Dockerfile kopiert `.env.example` als `.env`.
|
||||
- **Erstscan-Ergebnis (102126, 23:15):** Abbruch bei 4.198/5.240 Mails — `LookupError: unknown encoding: unknown-8bit` in `backend/routers/emails.py:327` (eine Mail killt den ganzen Scan). Bis dahin 321 Treffer, 123 PDFs verarbeitet, 110 Dateien abgelegt (nur `2026/ohne MwSt`). **Fix offen (Freigabe ausstehend):** try/except um die GESAMTE Header-Dekodierung pro Mail (Subject + From, alle Segmente von `decode_header`, nicht nur `[0]`), `errors="replace"` — Sentinel-Hinweis: nur `.decode()` zu kapseln reicht nicht. Danach Redeploy + Scan neu (`EmailLog.email_id unique` überspringt Verarbeitetes).
|
||||
|
||||
## 102326 — Anforderungen Markus (Ergebnis des Erstscans)
|
||||
- Scan mit Header-Fix (`bd2e21e`, `decode_hdr`) läuft seit 102226 23:57, Task `dccba3a8…`, Postfach Strato `info@c-stone-dev.com` INBOX.
|
||||
- **Gewünschtes Ergebnis:** Liste aller Rechnungsmails in a) Anhang, b) Download-Link, c) Portal-Hinweis. Entspricht Klassifikator Phase 1 (Anhang/Link/Portal/Ignorieren).
|
||||
- **Ablage Anhänge:** `Jahr/Monat/` statt `Jahr/mit|ohne MwSt`.
|
||||
- **Postfach auswählbar machen** (Account via `/settings` vorhanden, Ordner hartcodiert INBOX). Noch kein Plan/keine Freigabe.
|
||||
|
||||
## 102326 — Stand Tagesende
|
||||
- **Live auf NucBox:** Fix `bd2e21e` + Klassifikator `37e4ed1` deployt. Scan komplett: 391 Mail-Logs → attachment 170 / link 92 / portal 26 / ignore 103. `GET /emails/classified[?format=xlsx]` liefert die Liste a)/b)/c).
|
||||
- **Befunde Stichprobe:** AWS-/Canva-"Links" sind Login-Seiten (→ portal); Instagram/Meta „Zahlungsbeleg" fälschlich ignore; `email_from` teils MIME-codiert in DB.
|
||||
- **Deploy-Hinweis:** `deploy.sh` nur mit `cd <Projektpfad> &&` + `with-secrets.sh invoice-check scripts/deploy.sh`; Auto-Mode-Classifier blockt Deploy durch Claude → Markus führt aus.
|
||||
- **Schema:** Prod-DB ohne `alembic_version` (create_all); neue Spalten per `ALTER TABLE … IF NOT EXISTS` in `main.py` startup.
|
||||
- **Nächstes:** Klassifikator-Nachbesserung, dann Schritt 3 (Monatsablage) + 4 (Postfachwahl). Offen: Gmail-Creds, Jahr-Herkunft, 199 vs. 134.
|
||||
|
|
|
|||
13
Daily Notes/102226.md
Normal file
13
Daily Notes/102226.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# 102226 - Daily Work Log
|
||||
|
||||
## Decisions
|
||||
-
|
||||
|
||||
## Meetings & Conversations
|
||||
-
|
||||
|
||||
## Notes
|
||||
-
|
||||
|
||||
## End of Day Summary
|
||||
-
|
||||
17
Daily Notes/102326.md
Normal file
17
Daily Notes/102326.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# 102326 - Daily Work Log
|
||||
|
||||
## Decisions
|
||||
- Fix + Klassifikator Schritte 1+2 freigegeben; Schritte 3+4 (Monatsablage, Postfachwahl) noch nicht
|
||||
- Regel: Terminal-Befehle für Markus immer mit `cd <Pfad> &&` (global + Memory)
|
||||
|
||||
## Meetings & Conversations
|
||||
-
|
||||
|
||||
## Notes
|
||||
- Scan-Ergebnis: 170 Anhang / 92 Link / 26 Portal / 103 ignore
|
||||
- Gmail-Auth schlägt fehl (keine Creds)
|
||||
|
||||
## End of Day Summary
|
||||
- Erledigt: Header-Fix `bd2e21e`, Klassifikator + `/emails/classified` `37e4ed1`, beides deployt, Scan komplett
|
||||
- Offen: Klassifikator-Nachbesserung, Monatsablage, Postfachwahl, Gmail-Creds, 199 vs. 134
|
||||
- Morgen: Nachbesserung → Redeploy → Nachscan; dann Schritte 3+4 planen
|
||||
|
|
@ -1,10 +1,17 @@
|
|||
# Task Board — C-Stone Invoice Check → Invoice Picker
|
||||
|
||||
## 🔥 Aktuell (Phase 1 — Klassifikator + Direktlink)
|
||||
- [ ] Mail-Klassifikator (Anhang / Link / Portal / Ignorieren)
|
||||
- [ ] **Morgen zuerst:** Klassifikator nachbessern (Login-/Konsolen-Links → portal; „Zahlungsbeleg" → portal; `email_from` mit `decode_hdr` dekodieren), Redeploy (Markus), Nachklassifizier-Scan; 103 `ignore`-Mails stichprobenartig prüfen
|
||||
- [ ] Schritt 3 (Monatsablage `Jahr/Monat`) + Schritt 4 (Postfach/Ordner wählbar) — Freigabe ausstehend; vorher klären: Jahr aus Mail- oder Scan-Datum? 199 Invoice-Einträge vs. 134 PDFs (Duplikate?)
|
||||
- [ ] Gmail-Credentials fehlen (`AUTHENTICATIONFAILED` im Scan)
|
||||
- [x] Mail-Klassifikator (Anhang / Link / Portal / Ignorieren) — `37e4ed1`, live; Ergebnis 170/92/26/103
|
||||
- [ ] Direktlink-Fetcher (Stripe/Paddle) → OpenAI automatisch
|
||||
- [ ] Qualität prüfen: Aussteller-Erkennung liefert Wortfetzen („ichtet", „kundennr"), alles „ohne MwSt" → Parser-Review
|
||||
|
||||
- [x] (Basis 102326, Nachbesserung offen) **Ergebnisliste:** alle Rechnungsmails in 3 Klassen: a) Anhang, b) Download-Link, c) Hinweis „im Portal herunterladen" → Export/Ansicht (Excel + Frontend)
|
||||
- [ ] **Ablage Anhänge:** `Jahr/Monat/` (statt `Jahr/mit|ohne MwSt`); Steuer-Info bleibt in DB/Excel. Bestehende 110+ PDFs umziehen
|
||||
- [ ] **Postfach auswählbar:** IMAP-Account via `/settings` (DB) schon konfigurierbar; Ordner ist hartcodiert `INBOX` (`emails.py:299`) → Account + Ordner(n) wählbar machen
|
||||
|
||||
## 📋 Backlog (Invoice Picker)
|
||||
- [ ] Phase 2: Playwright-Basis, persistentes Profil, Re-Login-Flow, Manuell-Liste + Benachrichtigung
|
||||
- [ ] Phase 2: Adapter Amazon
|
||||
|
|
@ -14,6 +21,7 @@
|
|||
- [ ] pytest conftest.py
|
||||
|
||||
## ✅ Erledigt
|
||||
- [x] Erstscan-Fix `bd2e21e` (decode_hdr), Scan komplett durch (102226)
|
||||
- [x] Projektordner, Base Project, SaaS-Struktur
|
||||
- [x] Backend: FastAPI, Router (emails/invoices/auth/settings), Models, Alembic
|
||||
- [x] PDF-Extraktion PyMuPDF, Steuer-Erkennung, Excel-Export
|
||||
|
|
|
|||
|
|
@ -65,6 +65,10 @@ async def startup_event():
|
|||
"""Datenbank erstellen bei Start"""
|
||||
Base.metadata.create_all(bind=engine)
|
||||
ModelsBase.metadata.create_all(bind=engine)
|
||||
# create_all ergänzt keine Spalten an bestehenden Tabellen (kein alembic_version in Prod)
|
||||
with engine.begin() as c:
|
||||
c.execute(text("ALTER TABLE email_logs ADD COLUMN IF NOT EXISTS mail_class VARCHAR"))
|
||||
c.execute(text("ALTER TABLE email_logs ADD COLUMN IF NOT EXISTS link_url TEXT"))
|
||||
|
||||
|
||||
# Router einbinden
|
||||
|
|
|
|||
|
|
@ -76,6 +76,8 @@ class EmailLog(Base):
|
|||
processed = Column(Boolean, default=False)
|
||||
processing_result = Column(String, nullable=True) # "invoice_found", "not_an_invoice", "error"
|
||||
invoice_id = Column(Integer, nullable=True) # Verweis auf Invoice-Tabelle
|
||||
mail_class = Column(String, nullable=True, index=True) # "attachment" | "link" | "portal" | "ignore"
|
||||
link_url = Column(Text, nullable=True) # erster Rechnungs-Link (nur bei mail_class="link")
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ from datetime import datetime
|
|||
from typing import List, Optional, Dict, Any
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
from email.header import decode_header
|
||||
from email.header import decode_header, make_header
|
||||
|
||||
from backend.database import get_db, SessionLocal
|
||||
from backend.models import EmailLog, Invoice
|
||||
|
|
@ -21,6 +21,16 @@ from backend.routers.settings import get_imap_config
|
|||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def decode_hdr(value) -> str:
|
||||
"""Header robust dekodieren (alle Segmente); kaputte Encodings (unknown-8bit) -> replace statt Abbruch."""
|
||||
if not value:
|
||||
return ""
|
||||
try:
|
||||
return str(make_header(decode_header(str(value))))
|
||||
except Exception:
|
||||
return str(value)
|
||||
|
||||
# ponytail: in-memory scan state, fine for single process
|
||||
_scan_tasks: dict = {}
|
||||
_active_scan_id: Optional[str] = None
|
||||
|
|
@ -47,6 +57,34 @@ INVOICE_KEYWORDS = [
|
|||
INVOICE_STORAGE_PATH = os.getenv("INVOICE_STORAGE_PATH", "/tmp/invoice-storage")
|
||||
|
||||
|
||||
LINK_RE = re.compile(r"https?://[^\s\"'<>)\]]+", re.I)
|
||||
LINK_HINT = re.compile(r"invoice|rechnung|receipt|beleg|billing|download|\.pdf|stripe\.com|paddle\.com", re.I)
|
||||
PORTAL_HINT = re.compile(
|
||||
r"in ihrem (kunden)?(konto|portal)|im (kunden)?portal|your account|sign in|log ?in|anmelden|einloggen|"
|
||||
r"view (your )?(invoice|receipt|bill)|rechnung (ist )?(verfügbar|bereit)|invoice is available",
|
||||
re.I,
|
||||
)
|
||||
|
||||
|
||||
def classify_mail(msg) -> tuple:
|
||||
"""-> (mail_class, link_url). Reihenfolge: Anhang > Link > Portal-Hinweis > ignore."""
|
||||
text = ""
|
||||
for part in msg.walk():
|
||||
fn = part.get_filename()
|
||||
if fn and decode_hdr(fn).lower().endswith(".pdf"):
|
||||
return "attachment", None
|
||||
if part.get_content_type() in ("text/plain", "text/html"):
|
||||
payload = part.get_payload(decode=True) or b""
|
||||
text += payload.decode(part.get_content_charset() or "utf-8", "replace") + "\n"
|
||||
# ponytail: Regex statt HTML-Parser; Footer-Links ("invoice" im Pfad) können False-Positives liefern
|
||||
url = next((u for u in LINK_RE.findall(text) if LINK_HINT.search(u)), None)
|
||||
if url:
|
||||
return "link", url
|
||||
if PORTAL_HINT.search(re.sub(r"<[^>]+>", " ", text)):
|
||||
return "portal", None
|
||||
return "ignore", None
|
||||
|
||||
|
||||
def connect_imap(host: str, port: int, user: str, password: str) -> imaplib.IMAP4_SSL:
|
||||
"""Verbindet mit IMAP-Server"""
|
||||
mail = imaplib.IMAP4_SSL(host, port)
|
||||
|
|
@ -74,9 +112,7 @@ def search_invoices(mail: imaplib.IMAP4_SSL, folder: str = "INBOX") -> List[str]
|
|||
from_addr = msg.get("From", "")
|
||||
|
||||
# Subject decodieren
|
||||
decoded_subject, encoding = decode_header(subject)[0]
|
||||
if isinstance(decoded_subject, bytes):
|
||||
subject = decoded_subject.decode(encoding or "utf-8")
|
||||
subject = decode_hdr(subject)
|
||||
|
||||
# Prüfen auf Keywords
|
||||
subject_lower = subject.lower()
|
||||
|
|
@ -97,9 +133,7 @@ def extract_attachments(msg: email.message.Message) -> List[Dict[str, Any]]:
|
|||
|
||||
filename = part.get_filename()
|
||||
if filename:
|
||||
decoded_filename, encoding = decode_header(filename)[0]
|
||||
if isinstance(decoded_filename, bytes):
|
||||
filename = decoded_filename.decode(encoding or "utf-8")
|
||||
filename = decode_hdr(filename)
|
||||
|
||||
content_type = part.get_content_type()
|
||||
payload = part.get_payload(decode=True)
|
||||
|
|
@ -200,9 +234,7 @@ def extract_and_process_attachments(msg: email.message.Message, email_id: str, d
|
|||
if not filename:
|
||||
continue
|
||||
|
||||
decoded_filename, encoding = decode_header(filename)[0]
|
||||
if isinstance(decoded_filename, bytes):
|
||||
filename = decoded_filename.decode(encoding or "utf-8")
|
||||
filename = decode_hdr(filename)
|
||||
|
||||
content_type = part.get_content_type()
|
||||
|
||||
|
|
@ -312,31 +344,36 @@ def _scan_imap(mail_conn, db: Session, state: dict, since_date: Optional[str] =
|
|||
for email_id in email_ids:
|
||||
email_id_str = email_id.decode()
|
||||
log_entry = db.query(EmailLog).filter(EmailLog.email_id == email_id_str).first()
|
||||
if log_entry and log_entry.processed:
|
||||
if log_entry and log_entry.processed and log_entry.mail_class:
|
||||
continue
|
||||
result2, msg_data = mail_conn.fetch(email_id, '(RFC822)')
|
||||
if result2 != "OK":
|
||||
continue
|
||||
msg = email.message_from_bytes(msg_data[0][1])
|
||||
if log_entry: # Altbestand ohne Klasse: nur nachklassifizieren, Anhänge nicht erneut ablegen
|
||||
log_entry.mail_class, log_entry.link_url = classify_mail(msg)
|
||||
db.commit()
|
||||
continue
|
||||
subject = msg.get("Subject", "")
|
||||
from_addr = msg.get("From", "")
|
||||
date_str = msg.get("Date", "")
|
||||
email_date = parse_email_date(date_str)
|
||||
decoded_subject, enc = decode_header(subject)[0]
|
||||
if isinstance(decoded_subject, bytes):
|
||||
subject = decoded_subject.decode(enc or "utf-8")
|
||||
subject = decode_hdr(subject)
|
||||
subject_lower = subject.lower()
|
||||
matched_keywords = [kw for kw in INVOICE_KEYWORDS if kw in subject_lower]
|
||||
log_entry = None
|
||||
if matched_keywords:
|
||||
mail_class, link_url = classify_mail(msg)
|
||||
log_entry = EmailLog(
|
||||
mail_class=mail_class,
|
||||
link_url=link_url,
|
||||
email_id=email_id_str,
|
||||
email_subject=subject,
|
||||
email_from=from_addr,
|
||||
email_date=email_date,
|
||||
email_folder="INBOX",
|
||||
matched_keywords=matched_keywords,
|
||||
has_attachment=True,
|
||||
has_attachment=mail_class == "attachment",
|
||||
processed=True,
|
||||
)
|
||||
db.add(log_entry)
|
||||
|
|
@ -430,6 +467,40 @@ async def scan_status(task_id: str):
|
|||
return state
|
||||
|
||||
|
||||
@router.get("/classified")
|
||||
async def classified(format: str = "json", db: Session = Depends(get_db)):
|
||||
"""Rechnungsmails nach Klasse: attachment | link | portal (ignore ausgeblendet)."""
|
||||
rows = (
|
||||
db.query(EmailLog)
|
||||
.filter(EmailLog.mail_class.in_(["attachment", "link", "portal"]))
|
||||
.order_by(EmailLog.mail_class, EmailLog.email_date.desc())
|
||||
.all()
|
||||
)
|
||||
data = [
|
||||
{"klasse": r.mail_class, "datum": r.email_date.strftime("%Y-%m-%d") if r.email_date else "",
|
||||
"absender": r.email_from, "betreff": r.email_subject, "link": r.link_url or ""}
|
||||
for r in rows
|
||||
]
|
||||
if format != "xlsx":
|
||||
return data
|
||||
import io
|
||||
from openpyxl import Workbook
|
||||
from fastapi.responses import StreamingResponse
|
||||
wb = Workbook()
|
||||
wb.remove(wb.active)
|
||||
for k, title in (("attachment", "a) Anhang"), ("link", "b) Link"), ("portal", "c) Portal")):
|
||||
ws = wb.create_sheet(title)
|
||||
ws.append(["Datum", "Absender", "Betreff", "Link"])
|
||||
for d in data:
|
||||
if d["klasse"] == k:
|
||||
ws.append([d["datum"], d["absender"], d["betreff"], d["link"]])
|
||||
buf = io.BytesIO()
|
||||
wb.save(buf)
|
||||
buf.seek(0)
|
||||
return StreamingResponse(buf, media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
headers={"Content-Disposition": "attachment; filename=rechnungsmails.xlsx"})
|
||||
|
||||
|
||||
@router.get("/scan-all")
|
||||
async def scan_all_emails(db: Session = Depends(get_db)):
|
||||
"""Synchroner Scan (Legacy) — für direkten Aufruf ohne Polling."""
|
||||
|
|
|
|||
16
backend/tests/test_classify.py
Normal file
16
backend/tests/test_classify.py
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
import email
|
||||
from backend.routers.emails import classify_mail
|
||||
|
||||
|
||||
def _m(raw):
|
||||
return email.message_from_string(raw)
|
||||
|
||||
|
||||
def test_classify():
|
||||
pdf = "Content-Type: multipart/mixed; boundary=b\n\n--b\nContent-Type: text/plain\n\nhi\n--b\nContent-Type: application/pdf\nContent-Disposition: attachment; filename=r.pdf\n\nx\n--b--\n"
|
||||
assert classify_mail(_m(pdf)) == ("attachment", None)
|
||||
link = "Content-Type: text/plain\n\nIhre Rechnung: https://pay.stripe.com/invoice/abc/pdf"
|
||||
assert classify_mail(_m(link))[0] == "link"
|
||||
portal = "Content-Type: text/plain\n\nIhre Rechnung ist verfügbar. Bitte in Ihrem Konto herunterladen."
|
||||
assert classify_mail(_m(portal)) == ("portal", None)
|
||||
assert classify_mail(_m("Content-Type: text/plain\n\nHallo"))[0] == "ignore"
|
||||
Loading…
Add table
Reference in a new issue