英文:
Replace whole data for a key and drop old values
问题
I want to replace whole data for a key every one minute:
Suppose we have a table like this:
cid | uid |
---|---|
1 | a |
1 | b |
2 | a |
2 | c |
... | ... |
Now I want to replace all records with cid=1 with new values which is b
and c
;
I have two plans:
- Remove all existing with
delete from table where cid=1
and theninsert into table values(1,b),(1,c)
; - Insert all new records by
insert ignore into table values(1,b),(1,c)
and delete old data which is not exist in new valuesdelete from table where cid=1 and uid not in (b,c)
I can use partitioning in mysql on cid which causes fast delete of data but I'm not sure if there is a better method since this happens every minute with thousands of values. I want the best method with minimum time and lowest cost (CPU+DISK+Blueprint). I need something like overwrite a file with new content every minute (We don't need old data anymore).
any suggestion?
英文:
I want to replace whole data for a key every one minute:
Suppose we have a table like this:
cid | uid |
---|---|
1 | a |
1 | b |
2 | a |
2 | c |
... | ... |
Now I want to replace all records with cid=1 with new values which is b
and c
;
I have two plans:
- Remove all existing with
delete from table where cid=1
and theninsert into table values(1,b),(1,c)
; - Insert all new records by
insert ignore into table values(1,b),(1,c)
and delete old data which is not exist in new valuesdelete from table where cid=1 and uid not in (b,c)
I can use partitioning in mysql on cid which causes fast delete of data but I'm not sure if there is a better method since this happens every minute with thousands of values. I want the best method with minimum time and lowest cost (CPU+DISK+Blueprint). I need something like overwrite a file with new content every minute (We don't need old data anymore).
any suggestion?
答案1
得分: 1
Here is the translated content:
如果您想要最大化读取访问并一次加载完整数据集,那么您可以考虑可能的第三种选项:
-
创建原始表的空克隆
CREATE TABLE mytable_new LIKE mytable;
-
用完整数据集填充新表。根据新数据的来源,您可以以不同的方式执行此操作。如果您的数据已经在服务器上,那么使用
LOAD DATA
是相当高效的;如果您的数据通过应用层传递,可以使用批量的INSERT INTO mytable_new VALUES (1, 'a'), (1, 'b'), ...
语句;如果您的新数据已经存在于MySQL表中,可以使用INSERT INTO mytable_new SELECT ...
。 -
对旧表和新表执行原子重命名
RENAME TABLE mytable TO mytable_old, mytable_new TO mytable;
-
最后,通过删除旧数据进行清理
DROP TABLE mytable_old;
英文:
If you want to maximise read access and have a full dataset you can load at once, then you might consider a possible third option:
-
create an empty clone of the original table
CREATE TABLE mytable_new LIKE mytable;
-
populate your new table with a full dataset. There are different ways you could do this, depending on where your new data is coming from.
LOAD DATA
is pretty efficient if your data is already on the server, or you could use batches ofINSERT INTO mytable_new VALUES (1,'a'),(1,'b'),...
statements if your data is coming via an application layer, or, if your new data is already elsewhere within MySQL tables,INSERT INTO mytable_new SELECT ...
-
do an atomic rename of your old and new tables
RENAME TABLE mytable TO mytable_old, mytable_new TO mytable;
-
finally, clean up by dropping your old data
DROP TABLE mytable_old;
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论