Re: Dealing with large amounts of data, reading and writing

Thanks Quess. I'll have to give that a go and see if it works.

--Carl

On Nov 21, 6:20 am, Quess <miroslaw.niepo...@gmail.com> wrote:
> Hi Carl,
>
> I have similar problem to your with massive mailing list where every
> mail have to be send with some personalized data.
> The solutions which works great was to write an small javascript and
> using ajax calls php to send mails divided in portions (20 mails per
> ajax call) so i've never execute php script more than 5-6 seconds (In
> your case you can export your xml data in parts and append it to file
> every ajax call).
> I hope this give you some clue how to try to solve your problem.
>
> Best Regards,
> Quess
>
> On Nov 20, 9:51 pm, carl67lp <ccander...@gmail.com> wrote:
>
> > I have a database of board games; its data is based on the XML feed
> > from:
>
> >http://boardgamegeek.com/xmlapi/boardgame/1,2,3&stats=1
>
> > Presently it has more than 50,000 game entries, along with associated
> > publisher, artists, designers, and so forth.  In other words--a
> > massive amount of information.
>
> > Everything is structured fairly well with proper model associations, I
> > think, so it's a fairly trivial task to pull the data out of the MySQL
> > database. The issue is that the resulting data array is huge...and I'm
> > running out of memory.
>
> > I can sometimes handle this by pulling 1000 records at a time, say,
> > but then I run into my next issue.  To actually do all the reading and
> > writing to XML (via the SimpleXML class), I'd say it takes around 4 or
> > 5 minutes.  On my local environment, I can change the maximum
> > execution time to be limitless, so it's not a huge issue.  But on my
> > Web host, though, that's not such an easy task--I'm not even sure it
> > can be done.
>
> > So, it gets to the main question: What's the best way to handle
> > processing of tens of thousands of records, reading them from a MySQL
> > database then writing them out via XML?  (Incidentally, the XML file
> > that gets written is some 800MB in size!)
>
> > Any and all suggestions are most welcome.
>
> > --Carl

Check out the new CakePHP Questions site http://cakeqs.org and help others with their CakePHP related questions.

You received this message because you are subscribed to the Google Groups "CakePHP" group.
To post to this group, send email to cake-php@googlegroups.com
To unsubscribe from this group, send email to
cake-php+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php?hl=en