?iGJdZddlZddlZddlmZddlmZmZddlmZm Z m Z m Z m Z  ddl ZddlmZdZdd d d d d d d dd d d ddd dZehdZdefdZdede eeffdZdedefdZdedede efdZehdZ d6de e eefdedede efdZde efdZd7d e eefd!ed"ede eeffd#Z d8d$eded%ed&ede e eeff d'Z d8d(ed$eded%ed&ede e eeff d)Z!d*edede e eeffd+Z"d,e e eefde e eeffd-Z# d9ded.ed/ededede eeff d0Z$ d:d1e e eefdedede e eeffd2Z% d9ded.ed/ededede eeff d3Z&d4e eefde e eeffd5Z'y#e$rdZY_wxYw);aReddit search via ScrapeCreators API for /last30days. Uses ScrapeCreators REST API to search Reddit globally, discover relevant subreddits, run targeted subreddit searches, and fetch comment trees. Replaces openai_reddit.py as the primary Reddit search backend. Falls back to openai_reddit.py if SCRAPECREATORS_API_KEY is missing but OPENAI_API_KEY is present. Requires SCRAPECREATORS_API_KEY in config (same key as TikTok + Instagram). API docs: https://scrapecreators.com/docs N)Counter)datetimetimezone)AnyDictListOptionalSet)httpz(https://api.scrapecreators.com/v1/redditweek)global_searchessubreddit_searchescomment_enrichments timeframemonth)quickdefaultdeep>*aaninisofontoareforhownewthetopbestgoodnewstipswhatwithgreatguideusingwhichadvicekillerlatestpromptupdateawesomehottestmethodspopularpromptsupdatesfeaturestrendingtutorial practices prompting approaches strategiesrecommendationsmsgctjjd|dtjjy)zLog to stderr.z [Reddit]  N)sysstderrwriteflush)rDs Q/home/ubuntu/.openclaw/workspace/skills/last30days-official/scripts/lib/reddit.py_logrLBs-JJyR()JJtokenreturnc |ddS)z%Build ScrapeCreators request headers.zapplication/json)z x-api-keyz Content-Type)rNs rK _sc_headersrRHs* rMtopicc\|jj}gd}|D]3}|j|dzs|t|dj}5|j }|Dcgc] }|t vs |}}|rdj |n|}|jdScc}w)zyExtract core subject from verbose query. Strips meta/research words to keep only the core product/concept name. ) zwhat are the bestzwhat is the bestzwhat are the latestzwhat are people saying aboutzwhat do people think aboutz how do i usez how to usezhow tozwhat arezwhat isztips forzbest practices for N?!.)lowerstrip startswithlensplit NOISE_WORDSjoinrstrip)rStextprefixespwordswfilteredresults rK_extract_core_subjectrfPs ;;=   DH ) ??1s7 #A=&&(D) JJLE 9aA[$89H9#+SXXh F == :s / B)=B)depthc^t|}|g}|jjd}|j|jk7r-t |j dkr|j ||dvr|j |d|dk(r|j |d|S)a]Generate multiple Reddit search queries from a topic. Uses local logic (no LLM call needed): 1. Extract core subject (strip noise words) 2. Include original topic if different from core 3. For default/deep: add casual/review variant 4. For deep: add problem/issues variant Returns 1-4 query strings depending on depth. rVr)rrz worth it OR thoughts OR reviewrz$ issues OR problems OR bug OR broken)rfrXr^rWrZr[append)rSrgcorequeriesoriginal_cleans rKexpand_reddit_queriesrmis ! 'DfG[[]))%0N zz|~++--#n6J6J6L2MQR2R~& ##$>?@ $CDE NrM> helpmefind findareddit findthatsong namethatsong tipofmytongue whatsthissongsubredditdramawhatisthissongwhatisthisthingresultsmax_subsc |r t|nd}|r't|jjn t}t }|D]}|j dd}|sd}|j |rt fd|Dr|dz } tvr|dz}|j dxs|j dd } | r | d kDr|d z }||xx|z cc<|j|D cgc]\}} | c} }Scc} }w) aExtract top subreddits from global search results with relevance weighting. Uses frequency + topic-word matching + utility-sub penalties + engagement bonus to find discussion subs rather than utility/meta subs. Args: results: List of post dicts from global search topic: Original search topic (for relevance matching) max_subs: Maximum subreddits to return Returns: Top subreddit names sorted by weighted score subredditg?c3DK|]}t|dkDs|vyw)r N)rZ).0rc sub_lowers rK z&discover_subreddits..sOCFQJa9nOs  g@g333333?upsscorerdg?) rfsetrWr[rgetany UTILITY_SUBS most_common) rwrSrxrj core_wordsscorespostsubbaser_r~s @rKdiscover_subredditsrs$,1  'bD.2TZZ\'')*J YFhh{B' IIK #OjOO CKD  $ CKDhhuo5'1!5 39 CKDs t -0%00: ;FCC ;; ;s6 Dc|sy tjt|tj}|j dS#t ttf$rYywxYw)z%Convert Unix timestamp to YYYY-MM-DD.N)tzz%Y-%m-%d) r fromtimestampfloatrutcstrftime ValueError TypeErrorOSError) created_utcdts rK _parse_datersP   # #E+$68<< H{{:&&  7 +s>AAAridx source_labelc@|jdd}|rd|n|jdd}|rd|vrd}d||jddt|jddj|t|jd djt|jd |jd xs|jd d |jdd |jdddd|dt|jddddd S)z>Normalize a ScrapeCreators Reddit post to our internal format. permalinkrzzhttps://www.reddit.comurlz reddit.comRidtitler{rrrr num_comments upvote_ratio)rrrgffffff?zReddit z searchselftextNi) r reddit_idrrr{date engagement relevance why_relevantr)rstrrXr)rrrrrs rK_normalize_postrsb)I2; "9+ .%QSATC |3&#iXXdB'TXXgr*+113+r2399;DHH]34XXe_<!(< HH^Q7 HH^4 !,w7R01$37 rMquerysortrctstd ddlm}||||d}td|}t |}t j|d<t j||dd }|jd |jd gS tjtd|||dt |d} | j| j}|jd |jd gS#t$r} td | gcYd } ~ Sd } ~ wwxYw#t$r} td| gcYd } ~ Sd } ~ wwxYw)a/Search across all of Reddit via ScrapeCreators global search. Args: query: Search query token: ScrapeCreators API key sort: Sort order (relevance, hot, top, new) timeframe: Time filter (hour, day, week, month, year, all) Returns: List of post dicts z6requests library not installed, falling back to urllibr urlencode)rrrz/search? User-Agentr headerstimeoutretriespostsdatazGlobal search error (urllib): Nz/searchparamsrrzGlobal search error: ) _requestsrL urllib.parserSCRAPECREATORS_BASErRr USER_AGENTr Exceptionraise_for_statusjson) rrNrrrrrrreresps rK_global_searchrs7"  EF  .9UVF()&:C!%(G$(OOGL !88C"aHD88GTXXfb%9: : }}"#7 +"DyI&   yy{xx&"!566  1!5 6I   $QC () s=A5C9 A.D9 DDDD E(D=7E=Er{ctsx ddlm}|||||d}td|}t |}t j |d<t j||dd} | jd | jd gS tjtd||||dt |d} | j| j} | jd | jd gS#t$r} td |d | gcYd } ~ Sd } ~ wwxYw#t$r} td|d | gcYd } ~ Sd } ~ wwxYw)aSearch within a specific subreddit via ScrapeCreators. Args: subreddit: Subreddit name (without r/) query: Search query token: ScrapeCreators API key sort: Sort order timeframe: Time filter Returns: List of post dicts rr)r{rrrz/subreddit/search?rrr rrrz&Subreddit search error (urllib) for r/z: Nz/subreddit/searchrzSubreddit search error for r/ rrrrrRr rrrrLrr) r{rrNrrrrrrrrrs rK_subreddit_searchrsO&   .&9 F)));F8DC!%(G$(OOGL !88C"aHD88GTXXfb%9: : }}"##4 5&&   &   yy{xx&"!566%  9)BqcJ KI &  ,YKr! => s=A6C0A/D0 D9D DD D?"D:4D?:D?rcltsu ddlm}|d|i}td|}t |}t j |d<t j||dd}|jd |jd gS tjtd d|it |d}|j|j}|jd |jd gS#t$r}td |gcYd }~Sd }~wwxYw#t$r}td|gcYd }~Sd }~wwxYw)zFetch comments for a Reddit post via ScrapeCreators. Args: url: Reddit post URL or permalink token: ScrapeCreators API key Returns: List of comment dicts with score, author, body, etc. rrrz/post/comments?rrr rcommentsrzComment fetch error (urllib): Nz/post/commentsrzComment fetch error: r) rrNrrapi_urlrrrrs rKfetch_post_commentsrKs(   .s|,F,-_VHEG!%(G$(OOGL !88GWb!LD88J(<= = }}"#> 23<&   yy{xx DHHVR$899  1!5 6I   $QC () s=A3C*=A,D* D 3DD D  D3D.(D3.D3rct}t}g}|D]k}|jdd}|jdd}|r||vr.|r||vr5|r|j||r|j||j|m|S)z9Deduplicate posts by reddit_id, keeping first occurrence.rrzr)rraddri)rseen_ids seen_urlsuniquerridrs rK _dedupe_postsrtsuHI F hh{B'hhub! 3(?  3)#   LL   MM#  d  MrM from_dateto_datec J|sgddStj|td}|d}t||}td|dt |d|g}|d} t |d | D]a\} } | d k(rd nd } td | dzd| d| d| d t | || |} tdt | d|j| cg}t |D]&\} }t|| dzd}|j|(t|||d}td|t|}|d |dD]}td|d|dt|||d |}tdt |d|t |D]5\}}t|t ||zdzd|}|j|7t|}tdt |d g}d }|D]F}|d!r#||d!cxkr|krnn|j|+|d!|j|B|dz }H|r|}|r'td"|d#ntd$t ||jd%d&'t |D]\} }d(| dz|d)<td*t |d+d,|iS)-aFull Reddit search: multi-query global discovery + subreddit drill-down. This is the main entry point. Replaces openai_reddit.search_reddit(). Args: topic: Search topic from_date: Start date (YYYY-MM-DD) to_date: End date (YYYY-MM-DD) depth: 'quick', 'default', or 'deep' token: ScrapeCreators API key Returns: Dict with 'items' list and optional 'error'. z$No SCRAPECREATORS_API_KEY configured)itemserrorrrz Expanded 'z' into z queries: rNrrr&zGlobal search r /z: 'z' (sort=))rrz -> z resultsglobalr)rSrxzDiscovered subreddits: zSubreddit search: r/z for ''z results from r/zr/z After dedup: z unique postsrz Filtered z posts outside date rangez(No posts within date range, keeping all cN|jdijddxsdS)Nrrrr)xs rKzsearch_reddit..s#quu\2.227A>C!rMTkeyreverserrzFinal: z Reddit postsr) DEPTH_CONFIGrrmrLrZ enumeraterextendrrirrfrrr)rSrrrgrNconfigrrk all_raw_posts max_globalirrr all_itemsritemdiscovered_subsrjr sub_postsjin_range out_of_ranges rK search_redditrs1* &LMM   e\)%< =F{#I$E51G:eWGCL>G9 EFM)*Jgkz23$51f{% ~acU!J"- 01 Y rMrc tj|td}|d}|r|s|S|d|}tdt|d|D]}|jdd}|st ||}|s(g} g} t |ddD]\} } | jd d} | r| d vr | jd xs| jd d }| jdd}| jdd}|rd|nd}| d k(rdnd}| j |t| jd|| d||dt| dk\s|dvs| dd}t| dkDr:t |D]\}}|dvs |dkDs|d|dz}n|jdz}| j |!| jdd| dd|d <| dd|d!<|S)"aEnrich top items with comment data from ScrapeCreators. Args: items: Reddit items from search_reddit() token: ScrapeCreators API key depth: Depth for comment limit Returns: Items with top_comments and comment_insights added. rrNzEnriching comments for z postsrrz body) [deleted] [removed]rrrauthorrrzhttps://reddit.comii,r)rrrexcerptrr)rr AutoModeratorz.!?2r z...c&|jddS)Nrrr)cs rKrz&enrich_with_comments../sgq(9rMTr top_commentscomment_insights) rrrLrZrrrirr^r)rrNrgr max_comments top_itemsrr raw_commentsrinsightscirrrrr comment_url max_excerptinsightrchars rKenrich_with_commentsr s*  e\)%< =F/0L  m|$I "3y>"2& 9:11hhub! *36   |CR01 )EB55$D4#==EE%L5AEE'1$5EUU8[1Fk2.I>G.yk:RK"$q#cK   #AEE-$89  -" ! 4yB61\#\t*t9s?#,W#5;45=QV&-dqsmG!; #*.."2U":(= )B 94H+CR0^#+CR=  c11f LrMctt|||||}|jdg}|r|rt|||}||d<|S)aFull Reddit pipeline: search + comment enrichment. This is the convenience function that does everything. Args: topic: Search topic from_date: Start date (YYYY-MM-DD) to_date: End date (YYYY-MM-DD) depth: 'quick', 'default', or 'deep' token: ScrapeCreators API key Returns: Dict with 'items' list. Items include top_comments and comment_insights. r)rrr )rSrrrgrNrers rKsearch_and_enrichr 7sG*5)WeU CF JJw #E $UE59w MrMresponsec&|jdgS)zParse ScrapeCreators response to item list. Compatibility shim matching openai_reddit.parse_reddit_response() signature. rr)r s rKparse_reddit_responserVs << $$rM)rzr)r)rr)rN)r)(__doc__rerG collectionsrrrtypingrrrr r requestsr ImportErrorrzr rr frozensetr\rrLrRrfrmrintrrrrrrrrr r rrQrMrKrs  '11 @         ,   c stCH~   2ST#Y< .< $sCx. !.< .<.< #Y .%DcN%tDcN7K%AIsFF  F