DEV Community

Discussion on: What was your win this week?

Collapse
 
peter279k profile image
peter279k • Edited

At this week, I'm fighting with millions of data records with CSV formats and import them to MySQL database.

To store large data records, some tips are as follows:

  • Enhance the bash script development skills about importing data records to tables.
  • Using partitions to create table and let date field have the range and it can help selection to be optimized.
  • In InnoDB engine, close auto_commit, unique check and foreign check before using LOAD DATA to import CSV data records to MySQL database table.
  • Help our data scientists to optimize their problem of building web service. And optimize slow SQL query when selecting data records in specific table.