Re: Architecture ideas how to summarize values from a large database with multiple conditions

Aside from your storage engine you should make a Job Queue System ( Reference the 'Command' design pattern )

Queue up a report request, then notify when the report is finished.

I process traffic reports with over a billion rows in scope ( I use Google Big Query for my storage engine )

If that's not a solution, then you'll have to start digging into how your data is stored, how you need to view it, and how much you can normalize.

You should look at data warehouse practices/schemas including Fact/Dimension schemas and Star & Snowflake schemas.

On Dec 5, 2012, at 4:11 PM, pseudorandomnoise <janssonsfrestelse@gmail.com> wrote:

Hi all!

I am currently working on an app that has to summarize times for all entries as well as multiple entries (i.e last 30 days). My problem is that this database table will contain a few thousand rows (as well as nested tables). From an architecture point of view - any ideas how to design this the best way?

In order to avoid server time-outs, I might have to use some ajax call which does the actual request? Where cake would then, first check a cached file if the specific summary has been performed already (i.e date range etc.), if not, what would the best approach be?

A. Just loops through the massive db with all different $this->Model->find statements that are needed?

B. Grab the complete table (and nested tables) as an array and then search this array for the statements using the Hash-function instead? Is this even possible? (I suppose this approach might be a little quicker than A, if Hash will work like this?)

C.  ??

Any ideas would be much appreciated since I am stuck solid!

Cheerio

--
Like Us on FaceBook https://www.facebook.com/CakePHP
Find us on Twitter http://twitter.com/CakePHP
 
---
You received this message because you are subscribed to the Google Groups "CakePHP" group.
To post to this group, send email to cake-php@googlegroups.com.
To unsubscribe from this group, send email to cake-php+unsubscribe@googlegroups.com.
Visit this group at http://groups.google.com/group/cake-php?hl=en.