Re: keep hitting mysql memory limit, even in Cake Shell when exporting large xml file

You should make sure, that you free memory between chunks requests.
And do not try to output all this 25 Mb file at once, it also should
be done chunk-by-chunk. Remember, that PHP do not have automatic
garbage collector so you should unset no longer needed variables.

On Jul 26, 10:02 pm, JamesF <usaexportexpe...@gmail.com> wrote:
> i have paginated the results and no matter how i chunk it it still
> fills up the mysqli result set and hits the memory limit. i need to be
> able to flush the result set between each iteration.
>
> On Jul 26, 6:33 am, Jaydeep Dave <jaydipd...@gmail.com> wrote:
>
>
>
> > You can try something like this in your shell....
>
> > <?php
>
> > .......
> > ......
>
> > $page = 1;
>
> > while(1) {
>
> >          $records = $this->findAll("conditions' => array(bla bla bla),
> > 'fields' => 'Model.*' ,'limit' => 50, 'page' => $page);
>
> >          if($records) {
> >               // Write Your XML in a File.
> >          }else{
> >                break;
> >          }
>
> >         $page = $page + 50;
>
> > }
>
> > This will help you a lot.
>
> > Regards,
>
> > Jaydeep Dave
>
> > On Sun, Jul 26, 2009 at 1:41 AM, majna <majna...@gmail.com> wrote:
>
> > > Try with :
> > > $mysqli = ConnectionManager::getDataSource('default');
> > > $mysqli->disconnect();
> > > $mysqli->connect();
>
> > > debug($mysqli->connected);
> > > debug($mysqli->connection);
>
> > > On Jul 26, 7:55 am, JamesF <usaexportexpe...@gmail.com> wrote:
> > > > MY PROBLEM:
> > > > i am running into a wall with this one. basically i have a large xml
> > > > file that i am rendering using XmlHelper. We are talking about a
> > > > 25-50mb file. This is basically a product data feed. I have tried
> > > > quite a few methods to overcome the memory limit but no luck.
>
> > > > MY ERROR:
> > > > Fatal error: Allowed memory size of 83886080 bytes exhausted (tried to
> > > > allocate 53 bytes) in /home/username/usr/cakedev/cake/libs/model/
> > > > datasources/dbo/dbo_mysqli.php on line 402
>
> > > > WHAT I HAVE DONE TO TRY AND FIX IT:
>
> > > > 1) Initially i set this up to render through the web in one
> > > > shot....this cause execution limit timeouts and memory errors.
>
> > > > 2)I broke up the data requests in seperate chunks of 500 records using
> > > > internal method calls, like $listings = $this->getListings($start_id,
> > > > $limit); This caused the same problem
>
> > > > 3) I set the whole thing up via the cake shell interface. It works
> > > > great except for the fact that i keep running up against the same
> > > > limit, even with staggered chunks of records in different method
> > > > calls.
>
> > > > MY GUESS:
> > > > is that Cake is keeping an open connection with the database for the
> > > > life of the script.
>
> > > > MY SPECIFIC QUESTIONS:
> > > > can i disconnect and reconnect from the database inside of the script?
> > > > specifically can i do this in a cake shell script? will this solve my
> > > > memory problem? am i going about this the wrong way entirely?
>
> > > > thanks again anyone who has some advice!
>
> > --
> > Regards,
>
> > Jaydeep Dave
> > Mobile: +919898456445
> > Email: jaydipd...@yahoo.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "CakePHP" group.
To post to this group, send email to cake-php@googlegroups.com
To unsubscribe from this group, send email to cake-php+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---