?ix%dZddlZddlmZmZmZmZddlmZddl m Z m Z de dee fd Z Gd d eZ dde d eed ededeee eff dZdedee effdZddeededeee effdZddeededee fdZ ddee efdeed ededee eff dZ ddee efde d edee effdZy) zReddit thread enrichment with real engagement metrics. Supports two backends: 1. ScrapeCreators API (preferred) - no rate limits, 1 credit/call 2. reddit.com/.json (fallback) - free but 429-prone N)AnyDictListOptional)urlparse)httpdatesurlreturnc^ t|}d|jvry|jS#YyxYw)zxExtract the path from a Reddit URL. Args: url: Reddit URL Returns: Path component or None z reddit.comN)rnetlocpath)r parseds X/home/ubuntu/.openclaw/workspace/skills/last30days-official/scripts/lib/reddit_enrich.pyextract_reddit_pathrs3# v}} ,{{s ( (,ceZdZdZy)RedditRateLimitErrorz3Raised when Reddit returns HTTP 429 (rate limited).N)__name__ __module__ __qualname____doc__rrr!s=rr mock_datatimeoutretriesc||St|}|sy tj|||}|S#tj$r(}|jdk(rt d||Yd}~yd}~wwxYw)akFetch Reddit thread JSON data. Args: url: Reddit thread URL mock_data: Mock data for testing timeout: HTTP timeout per attempt in seconds retries: Number of retries on failure Returns: Thread data dict or None on failure Raises: RedditRateLimitError: When Reddit returns 429 (caller should bail) Nrriz#Reddit rate limited (429) fetching )rr get_reddit_json HTTPError status_coder)r rrrrdataes rfetch_thread_datar%&sw( s #D ##D'7K >> ==C &)LSE'RSYZ Zs.A)A$$A)r#c dgd}t|trt|dkr|S|d}t|tr|j dij dg}|r|dj di}|j d|j d|j d |j d |j d |j d |j d dddd|d<t|dk\r|d}t|tr|j dij dg}|D]}|j ddk7r|j di}|j ds<|j dd|j d |j dd|j dddd|j d d}|dj ||S)zParse Reddit thread JSON into structured data. Args: data: Raw Reddit JSON response Returns: Dict with submission and comments data N) submissioncommentsrrr#childrenscore num_comments upvote_ratio created_utc permalinktitleselftexti)r*r+r,r-r.r/r0r'kindt1bodyauthor [deleted],)r*r-r6r5r.r() isinstancelistlendictgetappend) r#resultsubmission_listingr)sub_datacomments_listingchildc_datacomments rparse_thread_datarFJsF dD !SY] a$d+%))&"599*bI {vr2H!g. ( ^ < ( ^ <'||M:%\\+6!g.$LLR8#>$F<  4yA~7 & -'++FB7;;JKH! 399V$,62.zz&)$ZZ3#)::m#<$jj;?"JJvr24C8!'K!8 z"))'2 3 Mrr(limitcz|Dcgc]}|jddvs|}}t|dd}|d|Scc}w)zGet top comments sorted by score. Args: comments: List of comment dicts limit: Maximum number to return Returns: Top comments sorted by score r6r7z [removed]c&|jddSNr*rr=cs rz"get_top_comments..s!%%2CrTkeyreverseN)r=sorted)r(rGrNvalidsorted_commentss rget_top_commentsrVsK! V1AEE(O;U$UQ VE VU(CTRO 6E "" Ws88cg}|d|dzD]}|jddjrtdkr4gd}tfd|DrMdd}tdkDr:t |D]\}}|d vs |d kDs|d|d z}n|j d z}|j |t||k\s|S|S) aMExtract key insights from top comments. Uses simple heuristics to identify valuable comments: - Has substantive text - Contains actionable information - Not just agreement/disagreement Args: comments: Top comments limit: Max insights to extract Returns: List of insight strings Nr2r5r1)z@^(this|same|agreed|exactly|yep|nope|yes|no|thanks|thank you)\.?$z^lol|lmao|hahaz ^\[deleted\]z ^\[removed\]c3fK|](}tj|j*yw)N)rematchlower).0pr5s r z+extract_comment_insights..s"@Qrxx4::<(@s.1z.!?2rz...)r=stripr;any enumeraterstripr>) r(rGinsightsrE skip_patternsinsighticharr5s @rextract_comment_insightsrksHJUQY'{{62&,,.s4y2~  @-@ @ t* t9s?$W- 345=QV%dqsmG 3 "..*U2 x=E !  O=< Oritemmock_thread_datac |jdd}t||||}|s|St|}|jd}|jdg}|ra|jd|jd|jdd |d <|jd } | rtj| |d <t |} g|d <| D]} | jdd} | rd| nd} |d j | jddtj| jd | jdd| jdddd| dt| |d<|S)uEnrich a Reddit item with real engagement data. Args: item: Reddit item dict mock_thread_data: Mock data for testing timeout: HTTP timeout per attempt (default 10s for enrichment) retries: Number of retries (default 1 — fail fast for enrichment) Returns: Enriched item dict Raises: RedditRateLimitError: Propagated so caller can bail on remaining items r r1rr'r(r*r+r,)r*r+r, engagementr-date top_commentsr.https://reddit.comrr6r5Nr*rpr6excerptr comment_insights)r=r%rFr timestamp_to_daterVr>rk)rlrmrrr thread_datarr'r(r-rqrNr. comment_urls renrich_reddit_itemrzst( ((5" C$C)97T[\K  { +FL)Jzz*b)H^^G,&NN>:&NN>: \!nn]3  22;?DL$H-LD   EE+r* :C*9+6  ^##UU7A&++AEE-,@AeeHb)uuVR(#. %     8 ED  Krtokenc ,ddlm}|jdd}|s|S|j||}|s|Sg}|ddD]}|jdd}|r|dvr|jd xs|jd d } |jd d } |jdd} | rd| nd} |j | |jdr$t j |jdnd| |dd|dd| d|jddg|d<|D]k}|dj |jd d |jd|jd d|jdd|jdddmt||d<|S)aJEnrich a Reddit item using ScrapeCreators comment API. No rate limit risk. Uses 1 credit per call. Args: item: Reddit item dict (already has engagement from search) token: ScrapeCreators API key timeout: HTTP timeout Returns: Enriched item with top_comments and comment_insights r)redditr r1N r5rIupsr*rr6r7r.rrr-r8rs)r*rpr6r5rur c&|jddSrKrLrMs rrOz'enrich_reddit_item_sc..7sAEE'1$5rTrPrqrprurtrv) r1r}r=fetch_post_commentsr>r rwsortrk) rlr{r reddit_modr raw_commentsrqrNr5r*r6r.rys renrich_reddit_item_scrs"' ((5" C  11#u=L  L #2  uuVR t99 e 1gq 1x-EE+r* :C*9+6 EFUU=EYE++AEE-,@A_c#JDSz    &5tDD   ^##UU7A&EE&MeeHb)uuY+55# %    8 ED  Kr)NrX)r~))Nr~r)rX)rrZtypingrrrr urllib.parserr1r r strr Exceptionrintr%rFrVrkrzrrrrrs ,,!SXc]$ 9 !% ! !~!! ! d38n !H6C6DcN6r#tDz###tDcN?S#&/tDz/#/d3i/h(, = sCx.=tn== =  #s(^ =F= sCx.= == #s(^ =r