Crawler
WeNepalBot
How WeNepal's crawler identifies itself, what it reads, how it respects robots.txt, and how to opt out.
What WeNepalBot is
WeNepalBot is the crawler that feeds WeNepal's newsroom. It reads published articles from a fixed list of news outlets and official sources so that our engine can write short summaries in its own words, with a credit and a link to every source. It does not copy articles for republication, it does not index the whole web, and it does not train AI models on what it reads. It runs on WeNepal's technology infrastructure, powered by Lacspace.
How it identifies itself
User agent:
WeNepalBot/1.0 (+https://wenepal.com/bot)Requests come from WeNepal's own servers. If you see this user agent from an address you doubt, write to hello@wenepal.com and we will confirm.
How it behaves
- Respects robots.txt. Rules for
WeNepalBotand for*are honoured; robots.txt is re-read every 24 hours. - Prefers feeds. Where a site offers RSS or Atom, the bot reads the feed first and fetches an article only when it is new.
- Is gentle. One connection per host at a time, at least five seconds between requests, and conditional requests (ETag and Last-Modified) so unchanged pages are not downloaded again.
- Reads only what it needs. Article text, title, byline, date and canonical URL. Image URLs are recorded for reference only; images are never downloaded for reuse.
- Stops on failure. A source that blocks or errors ten times in a row is paused automatically and a person decides whether to contact the outlet.
- Has a small footprint. Each run is limited in time and in the number of new items it fetches.
What we do with what it reads
Our engine clusters reports of the same event, checks that at least two independent outlets agree, and drafts a summary of 80 to 120 words that names and links every source. The AI policy describes the checks in detail. Under Nepal's Copyright Act 2059, facts and news of the day are not protected; the expression is, and our plagiarism check blocks any draft that copies a source's sentences.
Opt out
If you would rather WeNepalBot did not read your site, any of these works:
Add to your robots.txt:
User-agent: WeNepalBot Disallow: /- Email hello@wenepal.com with your domain. We remove it from the source list within two working days and confirm by reply.
Opting out also removes your outlet from our source credits, since we will no longer summarise your reporting.
Takedown requests
If a specific WeNepal story summarises your reporting in a way you object to, write to legal@wenepal.com with the WeNepal URL and your article URL. We acknowledge within 24 hours and assess within 48 hours. If the story fails our copying check on review, we unpublish it and tell you.
Our own robots.txt
wenepal.com allows all well-behaved crawlers, publishes sitemaps and a news sitemap, and describes its AI-content policy in /llms.txt. We disallow crawling of account pages, the API and search result pages.