替换一个键的整个数据并删除旧值。

huangapple go评论54阅读模式
英文:

Replace whole data for a key and drop old values

问题

I want to replace whole data for a key every one minute:

Suppose we have a table like this:

cid uid
1 a
1 b
2 a
2 c
... ...

Now I want to replace all records with cid=1 with new values which is b and c;
I have two plans:

  1. Remove all existing with delete from table where cid=1 and then insert into table values(1,b),(1,c);
  2. Insert all new records by insert ignore into table values(1,b),(1,c) and delete old data which is not exist in new values delete from table where cid=1 and uid not in (b,c)

I can use partitioning in mysql on cid which causes fast delete of data but I'm not sure if there is a better method since this happens every minute with thousands of values. I want the best method with minimum time and lowest cost (CPU+DISK+Blueprint). I need something like overwrite a file with new content every minute (We don't need old data anymore).

any suggestion?

英文:

I want to replace whole data for a key every one minute:

Suppose we have a table like this:

cid uid
1 a
1 b
2 a
2 c
... ...

Now I want to replace all records with cid=1 with new values which is b and c;
I have two plans:

  1. Remove all existing with delete from table where cid=1 and then insert into table values(1,b),(1,c);
  2. Insert all new records by insert ignore into table values(1,b),(1,c) and delete old data which is not exist in new values delete from table where cid=1 and uid not in (b,c)

I can use partitioning in mysql on cid which causes fast delete of data but I'm not sure if there is a better method since this happens every minute with thousands of values. I want the best method with minimum time and lowest cost (CPU+DISK+Blueprint). I need something like overwrite a file with new content every minute (We don't need old data anymore).

any suggestion?

答案1

得分: 1

Here is the translated content:

如果您想要最大化读取访问并一次加载完整数据集,那么您可以考虑可能的第三种选项:

  • 创建原始表的空克隆

    CREATE TABLE mytable_new LIKE mytable;

  • 用完整数据集填充新表。根据新数据的来源,您可以以不同的方式执行此操作。如果您的数据已经在服务器上,那么使用LOAD DATA是相当高效的;如果您的数据通过应用层传递,可以使用批量的INSERT INTO mytable_new VALUES (1, 'a'), (1, 'b'), ...语句;如果您的新数据已经存在于MySQL表中,可以使用INSERT INTO mytable_new SELECT ...

  • 对旧表和新表执行原子重命名

    RENAME TABLE mytable TO mytable_old, mytable_new TO mytable;

  • 最后,通过删除旧数据进行清理

    DROP TABLE mytable_old;

英文:

If you want to maximise read access and have a full dataset you can load at once, then you might consider a possible third option:

  • create an empty clone of the original table

    CREATE TABLE mytable_new LIKE mytable;

  • populate your new table with a full dataset. There are different ways you could do this, depending on where your new data is coming from. LOAD DATA is pretty efficient if your data is already on the server, or you could use batches of INSERT INTO mytable_new VALUES (1,'a'),(1,'b'),... statements if your data is coming via an application layer, or, if your new data is already elsewhere within MySQL tables, INSERT INTO mytable_new SELECT ...

  • do an atomic rename of your old and new tables

    RENAME TABLE mytable TO mytable_old, mytable_new TO mytable;

  • finally, clean up by dropping your old data

    DROP TABLE mytable_old;

huangapple
  • 本文由 发表于 2023年5月24日 21:52:26
  • 转载请务必保留本文链接:https://go.coder-hub.com/76324278.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定