Amazon CloudFront Reporting / Log Processing

One of our new customers has just gone live with our service using Amazon CloudFront. They are a pretty high traffic website with a fairly large number of images, i.e. > 50,000 and growing daily.

Amazon CloudFront reporting is non-existent. There simply isn’t any.

We have a custom reporting system so we process the CloudFront log files to generate these reports. CloudFront generates a lot of log files! It generates way too many files for my liking. It generates approximately 3,500 log files a day. This is 35 times more files than our existing CDN.

I am pretty sure when Amazon add more POPs we’ll see the number of files increase. This isn’t the case with our existing CDN.

If we use CloudFront with other customers the problem will only get worse as we have to use a CloudFront Distribution per customer. This will result in another set of files.

3,500 files x 100 customers is 350,000 files per day.

Advertisements

2 Responses to “Amazon CloudFront Reporting / Log Processing”

  1. Amazon CloudFront Wildcard CNAME Support | The diary of a CTO Says:

    […] Amazon CloudFront Reporting / Log¬†Processing […]

  2. Amazon CloudFront wildcard cache invalidation | The diary of a CTO Says:

    […] https://richardyeo.wordpress.com/2012/11/08/amazon-cloudfront-reporting-log-processing/ […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: