?i-dZddlZddlmZmZddlmZmZmZmZm Z ddl m Z ddl m Z iddd dd d d d d dddddddddddddddddddddddd ddd!d!d"d"d#d#d$Zd%ed&eefd'Zd(ed&eefd)Zd%ed*ed+ed&e eeeffd,Zhd-Zd%ed&efd.Zd%ed&efd/Z d8d0eeeefd1ed2ed3ed&eeeeff d4Zd5eeeefd2ed3ed&ee j0fd6Zd5ee j0d&ee j0fd7Zy)9aWebSearch module for last30days skill. NOTE: WebSearch uses the assistant's built-in web search tool, which runs inside the host environment. Unlike Reddit/X which use external APIs, web search results are obtained by the assistant directly and passed to this module for normalization and scoring. The typical flow is: 1. The assistant invokes its web search tool with the topic 2. The assistant passes results to parse_websearch_results() 3. Results are normalized into WebSearchItem objects N)datetime timedelta)AnyDictListOptionalTuple)urlparse)schemajanjanuaryfebfebruarymarmarchapraprilmayjunjunejuljulyaugaugustsep sept ) septemberoctoctobernovnovemberdecdecemberurlreturnctjd|}|rc|j\}}}dt|cxkrdkr;nn8dt|cxkrdkr$nn!dt|cxkrdkr nn |d|d|Stjd|}|rc|j\}}}dt|cxkrdkr;nn8dt|cxkrdkr$nn!dt|cxkrdkr nn |d|d|Stjd |}|rc|j\}}}dt|cxkrdkr;ny dt|cxkrdkr$ny dt|cxkrdkr ny |d|d|Sy ) a Try to extract a date from URL path. Many sites embed dates in URLs like: - /2026/01/24/article-title - /2026-01-24/article - /blog/20260124/title Args: url: URL to parse Returns: Date string in YYYY-MM-DD format, or None z/(\d{4})/(\d{2})/(\d{2})/r r(-z/(\d{4})-(\d{2})-(\d{2})[-/]z/(\d{4})(\d{2})(\d{2})/N)researchgroupsint)r0matchyearmonthdays T/home/ubuntu/.openclaw/workspace/skills/last30days-official/scripts/lib/websearch.pyextract_date_from_urlr@&sq II2C 8E  <<>eS 3t9 $ $c%j)>B)>1CCVTVCVV1UG1SE* * II5s ;E  <<>eS 3t9 $ $c%j)>B)>1CCVTVCVV1UG1SE* * II0# 6E  <<>eS 3t9 $ $ *+c%j)>B)> DECCVTVCV V1UG1SE* * textcn|sy|j}tjd|}|rq|j\}}}tj |dd}|rCdt |cxkrdkr/nn,dt |cxkrdkrnn|d|d dt |d Stjd |}|rq|j\}}}tj |dd}|rCdt |cxkrdkr/nn,dt |cxkrdkrnn|d|d dt |d Stjd |}|rc|j\}}}dt |cxkrdkr;nn8dt |cxkrd kr$nn!dt |cxkrdkr nn |d|d|Stj}d |vr |tdz }|jdSd|vr|jdStjd|}|r?t |jd} | dkr |t| z }|jdStjd|}|r|jdSd|vr |tdz }|jdSd|vr |tdz }|jdSy)a6Try to extract a date from text snippet or title. Looks for patterns like: - January 24, 2026 or Jan 24, 2026 - 24 January 2026 - 2026-01-24 - "3 days ago", "yesterday", "last week" Args: text: Text to parse Returns: Date string in YYYY-MM-DD format, or None Nz\b(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)\s+(\d{1,2})(?:st|nd|rd|th)?,?\s*(\d{4})\brr3r4r r5r602dz\b(\d{1,2})(?:st|nd|rd|th)?\s+(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)\s+(\d{4})\bz\b(\d{4})-(\d{2})-(\d{2})\br( yesterday)daysz%Y-%m-%dtodayz\b(\d+)\s*days?\s*ago\b<z\b(\d+)\s*hours?\s*ago\bz last weekrz this week) lowerr7r8r9 MONTH_MAPgetr:rnowrstrftimegroup) rB text_lowerr; month_strr>r<r=rGdaterFs r?extract_date_from_snippetrRLs J II 6   E  $||~ 3 im, TSY.$.1C3FB3FV1U3KqS#7 7 II    E $||~Y im, TSY.$.1C3FB3FV1U3KqS#7 7 II4d ;E  <<>eS 3t9 $ $c%j)>B)>1CCVTVCVV1UG1SE* * LLNEj ya((}}Z((*~~j)) II0* =E 5;;q>" 2:9$//D==, , II1: >E ~~j))j ya((}}Z((j ya((}}Z(( rAsnippettitlecjt|}|r|dfSt|}|r|dfSt|}|r|dfSy)aExtract date from any available signal. Tries URL first (most reliable), then snippet, then title. Args: url: Page URL snippet: Page snippet/description title: Page title Returns: Tuple of (date_string, confidence) - date from URL: 'high' confidence - date from snippet/title: 'med' confidence - no date found: None, 'low' confidence highmed)Nlow)r@rR)r0rSrTurl_date snippet_date title_dates r?extract_date_signalsr\sS*%S)H-W5LU""+51J5  rA>x.com www.x.com reddit.com twitter.comold.reddit.comwww.reddit.comwww.twitter.commobile.twitter.comc t|}|jj}|jdr|dd}|S#t$rYywxYw)z}Extract the domain from a URL. Args: url: Full URL Returns: Domain string (e.g., "medium.com") zwww.rN)r netlocrI startswith Exceptionr0parseddomains r?extract_domainrmsR#$$&   V $ABZF sc7E)R &NJ%", I$"2  Gw JJ{C0  CS% *:!;2 >?EEG    Tw;z L#:& I s#F  F43F4rcg}|D]}tj|d|d|d|d|d|jd|jdd|jd d |jd d  }|j||S)zConvert parsed dicts to WebSearchItem objects. Args: items: List of parsed item dicts from_date: Start of date range (YYYY-MM-DD) to_date: End of date range (YYYY-MM-DD) Returns: List of WebSearchItem objects r}rTr0r~rSrQrrXrwrxr{rfr|)r WebSearchItemrKr)rrsrtrrweb_items r?normalize_websearch_itemsrZsF  ''Dzw-U /O&! HH%6>hh{C0."5    h   MrAct}g}|D]R}|jjjd}||vs1|j ||j |T|S)zRemove duplicate WebSearch items. Deduplication is based on URL. Args: items: List of WebSearchItem objects Returns: Deduplicated list /)setr0rIrstripaddr)r seen_urlsrrurl_keys r?dedupe_websearchr|saI F ((.."))#. ) # MM' " MM$    MrA)rfrf)__doc__r7rrtypingrrrrr urllib.parser rfr rJrr@rRr\rormboolrprrrrrAr?rsb  (33!  1   1 !  1 q  1 q   1   1  a  1 a  1   1 a '( " 2 2  #s#x}#LWCWHSMWt# # # # 8C=#  #N (CD( Y $sCx. !Y YY Y  $sCx. Yx S#X  &   DD!5!564@T@T;UrA