?i! dZddlZddlmZmZmZmZddlmZe hdZ de de fd Z d&de d e dee fd Zd ee d ee defdZeej"ej$ej&ej(ej*ej,ej.ej0fZdede fdZdede fdZde dee fdZde de defdZde de defdZ d'deeej"ej$fdedeee e ffdZ d'deeej"ej$fdedeeej"ej$ffdZ d'deej"dedeej"fdZ! d'deej$dedeej$fdZ" d'deej&dedeej&fdZ# d'deej(dedeej(fdZ$ d'deej*dedeej*fdZ% d'deej,dedeej,fd Z& d'deej.dedeej.fd!Z'd"d#d$eededdfd%Z(y)(z.Near-duplicate detection for last30days skill.N)ListSetTupleUnion)schema>.aianatbebydohnifinisitmemynoofonorsotoweallandarebutcanforgethashowitsnotthewasyoufromhavejustshowthatthiswhatwillwithyourabouttextreturnc|j}tjdd|}tjdd|}|jS)zgNormalize text for comparison. - Lowercase - Remove punctuation - Collapse whitespace [^\w\s] z\s+)lowerresubstrip)r7s Q/home/ubuntu/.openclaw/workspace/skills/last30days-official/scripts/lib/dedupe.pynormalize_textrAs? ::t|tjr|jddSt|tjr|jddSt|tj r|jddSt|tj rX|j}|jdr|ddj}|S|jdr|ddj}|St|tjr |jSt|S)zGet text for cross-source comparison. Same as get_item_text() but truncates X posts to 100 chars to level the playing field against short Reddit/HN titles. Strips 'Show HN:' prefix from HN titles for fairer matching. NdzShow HN:zAsk HN:) rQrXItemr7rWrYrTrS startswithr?rZr])rOrSs r@_get_cross_source_textrdHs$ %yy#$))*yy#$,,-yy#$--.    J '!"IOO%E   i (!"IOO%E $--.zz  rBctjdd|jj}|Dchc]}|tvs t |dkDs|c}Scc}w)z8Tokenize text for cross-source token Jaccard comparison.r:r;r)r=r>r<split STOPWORDSrE)r7wordsws r@_tokenize_for_xrefrjasH FF:sDJJL 1 7 7 9E B! 2s1vzA BB Bs AAAtext_atext_bct|}t|}|r|syt||z}t||z}|r||z SdS)z.Token-level Jaccard similarity (word overlap).rK)rjrE)rkrltokens_atokens_brLrMs r@_token_jaccardrpgsN!&)H!&)H 8x(*+L 8# $E#(<% 1c1rBcntt|t|}t||}t||S)zAHybrid similarity: max of char-trigram Jaccard and token Jaccard.)rNrGrpmax)rkrl trigram_sim token_sims r@_hybrid_similarityrurs1$Z%7F9KLKvv.I {I &&rBitems thresholdcg}|Dcgc]}tt|}}tt|D]J}t|dzt|D]-}t ||||}||k\s|j ||f/L|Scc}w)zFind near-duplicate pairs in items. Args: items: List of items to check threshold: Similarity threshold (0-1) Returns: List of (i, j) index pairs where i < j and items are similar r)rGr]rFrErNappend)rvrw duplicatesrOngramsr j similaritys r@find_duplicatesr~ysJ;@ @$jt,- @F @ 3u: *q1uc%j) *A+F1IvayAJY&!!1a&) ** AsBc8t|dkr|St||}t}|D]G\}}||j||jk\r|j |7|j |It |Dcgc] \}}||vs |c}}Scc}}w)zRemove near-duplicates, keeping highest-scored item. Args: items: List of items (should be pre-sorted by score descending) threshold: Similarity threshold Returns: Deduplicated items r)rEr~setscoreadd enumerate)rvrw dup_pairs to_remover r|idxrOs r@ dedupe_itemsrs 5zQ  y1II1 8>>U1X^^ + MM!  MM!  #,E"2 KYS$c6JD KK Ks  BBct||S)zDedupe Reddit items.rrvrws r@ dedupe_redditr y ))rBct||S)zDedupe X items.rrs r@dedupe_xrrrBct||S)zDedupe YouTube items.rrs r@dedupe_youtuberrrBct||S)zDedupe TikTok items.rrs r@ dedupe_tiktokrrrBct||S)zDedupe Instagram items.rrs r@dedupe_instagramrrrBct||S)zDedupe Hacker News items.rrs r@dedupe_hackernewsrrrBct||S)zDedupe Polymarket items.rrs r@dedupe_polymarketrrrBg?)rw source_listscg}|D]}|j|t|dkry|Dcgc] }t|}}tt|D]}t|dzt|D]}t ||t ||ur t ||||}||k\s8||j ||jvr+||jj||j ||j ||jvs||jj||j ycc}w)aAnnotate items with cross-source references. Compares items across different source types using hybrid similarity (max of char-trigram Jaccard and token Jaccard). When similarity exceeds threshold, adds bidirectional cross_refs with the related item's ID. Modifies items in-place. Args: *source_lists: Variable number of per-source item lists threshold: Similarity threshold for cross-linking (default 0.40) rN) extendrErdrFtyperuid cross_refsry) rwr all_items source_listrOtextsr r|r}s r@cross_source_linkrs@I#& %& 9~7@ @d #D ) @E @ 3y> " Dq1uc)n- DAIaL!T)A,%77+E!HeAh?JY&Qrs4 **      ;S;S;S;6SX6SX6%6 !!6<<1C1CVEVEV$$f&;&;V=R=RTZThThi jC(S2CSCSXC 23222's'C'E' f''56 7 %S/:L f''56 7LL %!!6<</ 01LF* !! "** &  **  ** &,,** "" #** &  ** !! "** &  ** $$ %** &  ** %% &** &   ** %% &** &   *%D=%D%D %DrB