we recently migrated our eventstore on Mysql and I did a small boo boo. I had two tables event and event_details and I had created two indexes on it
CREATE INDEX events_${TBL_SUFFIX}_i2 ON events_${TBL_SUFFIX} (event_detail_event_guid, pid);
CREATE INDEX event_details_${TBL_SUFFIX}_i1 ON events_${TBL_SUFFIX} (event_detail_event_guid, pid);
As you can notice the boo boo in second index Instead of creating it on event_details table I created the same index on event table :(.
Yesterday night a shard event_detail table balloned to 85M records and there were 20 threads doing a full table scan on this table.
so I fixed the copy paste mistake and generated ddls for each shard but the mysql server kept going Out of memory everytime it tried creating index on this 85M record table. Ultimately the only way to get it done was to start mysql on diff port so no one will connect to it and then give innodb more memory and create index, reset innodb settings back and then restart the server.
All in all it took 6 hours in night to fix the after effects of this copy paste mistake.
CREATE INDEX events_${TBL_SUFFIX}_i2 ON events_${TBL_SUFFIX} (event_detail_event_guid, pid);
CREATE INDEX event_details_${TBL_SUFFIX}_i1 ON events_${TBL_SUFFIX} (event_detail_event_guid, pid);
As you can notice the boo boo in second index Instead of creating it on event_details table I created the same index on event table :(.
Yesterday night a shard event_detail table balloned to 85M records and there were 20 threads doing a full table scan on this table.
so I fixed the copy paste mistake and generated ddls for each shard but the mysql server kept going Out of memory everytime it tried creating index on this 85M record table. Ultimately the only way to get it done was to start mysql on diff port so no one will connect to it and then give innodb more memory and create index, reset innodb settings back and then restart the server.
All in all it took 6 hours in night to fix the after effects of this copy paste mistake.
Comments
Post a Comment