database - Generating a massive 150M-row MySQL table -

I have a C program, which is a huge data source (20 GB raw text) to mine and is simple to perform on for the empty table INSERTs generate the load (1 to 4 integer column primary key). A MEMORY as the table in the setup, the entire work is to be completed in 8 hours. After finishing in the table, there are approximately 150 million rows. Eight hours for me is the number of a fully civilized. This is a one-time deal.

The problem occurs when the MEMORY table to return MyISAM tries to change is that (a) I Other processes for memory to free has been, and (b) when I restart the computer I will not kill you data.

  ALTER TABLE memtable ENGINE = MyISAM  

I have two days to the Optional table query run has, and it it is not. I now it has killed

If I was starting in the meyajama as table'm making, the writing speed is too bad there is (especially due to the fact that the query as Duplicate of key update require the use of technology). I can not stop temporarily the keys will be 1000 times larger than the table and then if I start I need the keys again and must be run by a group of 150,000,000,000 lines. Umm, no

to feel the major obstacles in the one: INSERT query record updates if the table has a primary key (a hash) already exists

maisasama strictly try to use the beginning, I have 1,250 lines per second, the speed of getting a fat. The index increased after, I think that this rate will be even more tanks.

---

I have 16 GB of memory is installed in the machine. A large table is the best way to generate a drive that eventually, as indexed in MyISAM table is finished?

---

Explanation: There are a lot of updating queries ( INSERT ... val = val + duplicate key update any ) . It is a way, a raw dump is not a problem. For the first time in a MEMORY to the table to try to change my argument that all the index lookup, and the table-are for each INSERT, was for.

For you to comment on (the last one, probably) Sorry for throwing.

I just got a big table to change is an example of offers misama the inodibi up, while this is not what you are, it is a intermediate storage of table uses and an efficient way of storage memory is used to describe - how to sort the table in memory that InnoDB is hope that in the end it has to be ordered. If you are not tied to maissama you already have a "right" storage table, because it is a value as can be.