优化MySQL数据库以进行FAST计数 [英] Optimizing MySQL database for FAST count
问题描述
我目前正在尝试优化数据库.问题如下: 我有一张桌子，当前存储超过83Mio.时间相关的值.它们由较高的时间戳(ms)索引.我需要做的是计算某个值在给定的时间间隔中出现了多少次-例如说我想知道值1.56787在时间戳x到时间戳y的间隔中出现了多少次.现在，这几乎是永远的. 我正在使用InnoDB，我已经花了很多时间来优化配置文件，这极大地提高了速度.
i'm currently trying to optimize my database. The problem is the following: I have a table which currently stores over 83Mio. timedependent values. They are indexed by a highres (ms) timestamp. What i need to do is count how many times a certain value appears in a given interval of time - for example say i want to know how many times value 1.56787 appeared in the interval form timestamp x to timestamp y. Right now this takes almost forever. Im using InnoDB and i already put a lot of time into optimizing the config files, which increased the speed immensly.
我非常感谢您的任何投入，因为我几乎耗尽了如何实现这一目标的想法.我能想到的唯一解决方法是创建包含固定间隔的预先计数值的表，由于整个事情也应该是完全可更新的(我们谈论的是每几毫秒到达的新值)，所以这将不能真正令人满意.另一个数据库系统会更适合我的问题吗?
Im thankful for any input, as im pretty much running out of ideas how to pull this off. The only workaround i can think of is to create tables which contain pre counted values for fixed intervals, which would not be really satisfying since the whole thing should also be fully updateable (we are talking about new values arriving every few milliseconds). Would another db system be better suited for my problem?
这是解释输出:
Field Type Null Key Default Extra
timestamp bigint(20) NO PRI NULL
ask decimal(6,5) NO NULL
bid decimal(6,5) NO NULL
askvolume decimal(6,5) NO NULL
bidvolume decimal(6,5) NO NULL
# The MySQL server
[mysqld]
port= 3306
socket= "C:/xampp/mysql/mysql.sock"
basedir="C:/xampp/mysql"
tmpdir="C:/xampp/tmp"
datadir="C:/xampp/mysql/data"
pid_file="mysql.pid"
skip-external-locking
key_buffer = 16M
max_allowed_packet = 61M
table_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error="mysql_error.log"
bind-address="192.168.1.2"
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
# commented in by lampp security
#skip-networking
skip-federated
# Replication Master Server (default)
# binary logging is required for replication
# log-bin deactivated by default since XAMPP 1.4.11
#log-bin=mysql-bin
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Point the following paths to different dedicated disks
#tmpdir = "C:/xampp/tmp"
#log-update = /path-to-dedicated-directory/hostname
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 1024M
innodb_additional_mem_pool_size = 20M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
哦，该机器是具有6GB RAM的i7-950，并且系统+数据库位于SSD上.所以我认为那不应该是问题吗?
Oh the machine is an i7-950 with 6GB of RAM and the system+database is on a SSD. So i think that should not be the problem?
感谢您的帮助，我们将不胜感激！
Thanks for your help, it will be highly appreciated!
推荐答案
如果时间范围可以表示为一系列范围(月，日，周等)，则可以引入诸如日期前缀列之类的内容，这将大大减少使用IN()表达式检查的行数.
If the time ranges can be expressed as a series of ranges (months, days, weeks, etc.), you might introduce something like a date-prefix column, that will significatly reduce the number of examined rows using IN() expression.
这里有一篇文章揭示了这个想法: http://www.mysqlperformanceblog.com/2010/01/09/getting-around-optimizer-limitations-with-an-in-list/
Here is an article that exposes the idea: http://www.mysqlperformanceblog.com/2010/01/09/getting-around-optimizer-limitations-with-an-in-list/
这篇关于优化MySQL数据库以进行FAST计数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
