Adaptive weighted crawling of user activity feeds
Embodiments are directed towards determining frequencies for adaptively crawling user activity feeds, by taking into account user activity patterns, as well as a type or nature of a service to be crawled. A statistical model of activity patterns is generated for each user and user activity source based on various parameters. The activity patterns may be weighted to account for variation in relevancy of user content. Various measures are generated that are adaptive by being updated at a rate determined sufficient to detect user activity pattern changes. The statistical model produces clusters of users for each service. A crawling frequency may be assigned to the various clusters such that different users for a given service may have content crawled at different frequencies.
Yahoo! Inc.
G G
G06F G06F
inventional inventional
Mani Pankaj Saurabh
Kumar Kothari Sahni