记一次50万数据处理内存优化

2020-06-19 17:28:55   工作备份

  内存优化  

公司新项目需要进行数据迁移,在使用测试数据库编写迁移脚本时,发现之前的数据库写法太占内存,所以记下更改后的代码,方便以后查看

  1. 原来写法:
  2. $customer = $this->mysqli->query('select user_basic.*,parent_department,black_list.id as is_black,department.store_id as d_store from user_basic left join department on department.id = department_id left join black_list on black_list.phone = user_basic.phone order by user_basic.id');
  3. $count = $this->count($this->mysqli, 'user_basic');
  4. $time = time();
  5. \DB::statement('SET FOREIGN_KEY_CHECKS = 0');
  6. while($row = $customer->fetch_object()){
  7. ...
  8. }
  1. 更改后:
  2. //迭代器
  3. public function cursor($connection, $sth)
  4. {
  5. while ($row = $sth->fetch_object()) {
  6. @mysqli_next_result(${$connection});
  7. yield $row;
  8. }
  9. }
  10. ...
  11. //使用MYSQLI_USE_RESULT 后 再执行 mysql操作 可能会报Commands out of sync; you can't run this command now 目前我采用的是,另起一个数据库连接避免这个问题
  12. $count = $this->count($this->mysqli, 'user_basic');
  13. $customer = $this->mysqli->query('select user_basic.*,parent_department,black_list.id as is_black,department.store_id as d_store from user_basic left join department on department.id = department_id left join black_list on black_list.phone = user_basic.phone order by user_basic.id', MYSQLI_USE_RESULT);
  14. $time = time();
  15. \DB::statement('SET FOREIGN_KEY_CHECKS = 0');
  16. foreach ($this->cursor($this->mysqli, $customer) as $row) {
  17. ...
  18. }

优化前

  1. Max usage:680.53MB

优化后

  1. Max usage:18.53MB