Everyone

Haven't noticed any appreciable hit on the machine, incoming or outgoing, but there haven't been any real users yet, and the comment feature is still on the sidelines. CPU stays in between 0-2% mostly, peaking at about 12% if I hit reload as fast as I can in a browser (not a realy test, I know). Memory is a steady 30MB on a machine with 1GB. The aggregator has been running for the past month or so without If-Modified-Since/If-None-Match headers, and I haven't noticed any bandwidth drain. The latest update to the codebase added those in, so it should be even better.

I'm currently worried about the concentration of hits on LiveJournal and Xanga. They might decide to get very angry with me, though I'm probably still below the radar. Xanga didn't seem to notice when I was writing my xanga2mt script, which hammered Xanga for several hours straight as I was debugging it :).

In the future, comment parsing will greatly outweigh anything else, as it potentially adds in a factor of N to the bandwidth consumption, where N is the number of entries. Also, LiveJournal and Xanga do not issue Last-Modified or ETag headers for actual entries. I may not support scraping of those sites for that reason, even though I already have to code to do so.

Also, I may have to design the comment feature such that you explicitly have to mark that you want to subscribe to a particular entry.

Squid won't save too much right now, as the cache would have to be very short lived. My blog, for example, only gets 3-6 hits to the front page per hour, even if you look at peak times. The cache expiration would have be set to about ten minutes, which would mean that usually no entries would be served out of the cache, and I would actually be doubling the bandwidth costs. For now, I'll have to see if I get any users first :).

BTW - you officially get credit for being "First Post!" on movabletypo.net. I guess I get "Second Post!" and "Third Post!" :)

That's what I'm hoping, but I know Slashdot, for example, shuts off its feed to any site that reloads to feed too frequently. /. is kinda on the leading edge of this, but I don't want to give other sites reason to, even if I'm only a blip on the radar of a much larger problem.

I'm also a little worried about scraping MT blogs. , but I'm hoping that I can optimize out any of the actual issues. I know that I would certainly notice my aggregator scraping my blog, but I'm also the type of person that looks at my statistics regularly to figure out where my site is being stressed. Say, for example, that your site has ten entries in the feed, which is about par for the course. If I just read your feed every thirty minutes, that's forty-eight hits per day, 1440 per month. If I scrape the comments for that, that's 528 hits per day, or 15840 per month. Also, those hits are loading individual entries, which are much larger.

Currently, no site has registered more than 5000 hits on my site in a single month, so this would be pretty big. The Modified-Since stuff should make most of this go away, but I don't know how well .php sites, for example, support those headers.