antirez [Sun, 13 May 2012 14:40:29 +0000 (16:40 +0200)]
Redis timer interrupt frequency configurable as REDIS_HZ.
Redis uses a function called serverCron() that is very similar to the
timer interrupt of an operating system. This function is used to handle
a number of asynchronous things, like active expired keys collection,
clients timeouts, update of statistics, things related to the cluster
and replication, triggering of BGSAVE and AOF rewrite process, and so
forth.
In the past the timer was called 1 time per second. At some point it was
raised to 10 times per second, but it still was fixed and could not be
changed even at compile time, because different functions called from
serverCron() assumed a given fixed frequency.
This commmit makes the frequency configurable, so that it is simpler to
pick a good tradeoff between overhead of this function (that is usually
very small) and the responsiveness of Redis during a few critical
circumstances where a lot of work is done inside the timer.
An example of such a critical condition is mass-expire of a lot of keys
in the same second. Up to a given percentage of CPU time is used to
perform expired keys collection per expire cylce. Now changing the
REDIS_HZ macro it is possible to do less work but more times per second
in order to block the server for less time.
If this patch will work well in our tests it will enter Redis 2.6-final.
antirez [Fri, 11 May 2012 17:17:31 +0000 (19:17 +0200)]
More incremental active expired keys collection process.
If a large amonut of keys are all expiring about at the same time, the
"active" expired keys collection cycle used to block as far as the
percentage of already expired keys was >= 25% of the total population of
keys with an expire set.
This could block the server even for many seconds in order to reclaim
memory ASAP. The new algorithm uses at max a small amount of
milliseconds per cycle, even if this means reclaiming the memory less
promptly it also means a more responsive server.
Compare integers in ziplist regardless of encoding
Because of the introduction of new integer encoding types for ziplists
in the 2.6 tree, the same integer value may have a different encoding in
different versions of the ziplist implementation. This means that the
encoding can NOT be used as a fast path in comparing integers.
antirez [Wed, 2 May 2012 20:41:50 +0000 (22:41 +0200)]
syncio.c read / write functions reworked for correctness and performance.
The new implementation start reading / writing before blocking with
aeWait(), likely the descriptor can accept writes or has buffered data
inside and we can go faster, otherwise we get an error and wait.
This change has effects on speed but also on correctness: on socket
errors when we perform non blocking connect(2) write is performed ASAP
and the error is returned ASAP before waiting.
So the practical effect is that now a Redis slave is more available if it
can not connect to the master, previously the slave continued to block on
syncWrite() trying to send SYNC, and serving commands very slowly.
antirez [Wed, 2 May 2012 15:14:45 +0000 (17:14 +0200)]
Use specific error if master is down and slave-serve-stale-data is set to no.
We used to reply -ERR ... message ..., now the reply is
instead -MASTERDOWN ... message ... so that it can be distinguished
easily by the other error conditions.
Every matched key in a KEYS call is checked for expiration. When the key
is set to expire, the call to `getExpire` will assert that the key also
exists in the main dictionary. This in turn causes a rehashing step to
be executed. Rehashing a dictionary when there is an iterator active may
result in the iterator emitting duplicate entries, or not emitting some
entries at all. By using a safe iterator, the rehash step is omitted.
Set LUA_MASKCOUNT hook more selectively. Fixes issue #480.
An user reported a crash with Redis scripting (see issue #480 on
github), inspection of the kindly provided strack trace showed that
server.lua_caller was probably set to NULL. The stack trace also slowed
that the call to the hook was originating from a point where we just
used to set/get a few global variables in the Lua state.
What was happening is that we did not set the timeout hook selectively
only when the user script was called. Now we set it more selectively,
specifically only in the context of the lua_pcall() call, and make sure
to remove the hook when the call returns. Otherwise the hook can get
called in random contexts every time we do something with the Lua
state.
Re-introduce -g -rdynamic -ggdb when linking, fixing strack traces.
A previous commit removed -g -rdynamic -ggdb as LDFLAGS, not allowing
Redis to produce a stack trace wth symbol names on crash.
This commit fixes the issue.
The main reason is that otherwise it is completely pointless that we do
a lot of efforts to print the stack trace on crash, and the content of
the stack and registers as well. Using an alternate stack broken this
feature completely.
Redis test: More reliable BRPOPLPUSH replication test.
Now it uses the new wait_for_condition testing primitive.
Also wait_for_condition implementation was fixed in this commit to properly
escape the expr command and its argument.
Redis test: scripting EVALSHA replication test more reliable.
A new primitive wait_for_condition was introduced in the scripting
engine that makes waiting for events simpler, so that it is simpler to
write tests that are more resistant to timing issues.
1) One integer "immediate" encoding that can encode from 0 to 12 in the
encoding byte itself.
2) One 8 bit signed integer encoding that can encode 8 bit signed small
integers in a single byte.
The idea is to exploit all the not used bits we have around in a
backward compatible way.
Add a 24bit integer to ziplists to save one byte for ints that can
fit in 24 bits (thanks to antirez for catching and solving the two's compliment
bug).
Fix and refactoring of code used to get registers on crash.
This fixes compilation on FreeBSD (and possibly other systems) by
not using ucontext_t at all if HAVE_BACKTRACE is not defined.
Also the ifdefs to get the registers are modified to explicitly test for the
operating system in the first level, and the arch in the second level
of nesting.
1) Up to SLOWLOG_ENTRY_MAX_ARGV arguments are logged.
2) Up to SLOWLOG_ENTRY_MAX_STRING bytes per argument are logged.
3) slowlog-max-len is set to 128 by default (was 1024).
The number of remaining arguments / bytes is logged in the entry
so that the user can understand better the nature of the logged command.
just fieldobj itself as sentinel of the fact a field object is used or
not, instead of using the filed length, that may be confusing both for
people and for the compiler emitting a warning.
lookupKeyByPattern() used by SORT GET/BY rewritten. Fixes issue #460.
lookupKeyByPattern() was implemented with a trick to speedup the lookup
process allocating two fake Redis obejcts on the stack. However now that
we propagate expires to the slave as DEL operations the lookup of the
key may result into a call to expireIfNeeded() having the stack
allocated object as argument, that may in turn use it to create the
protocol to send to the slave. But since this fake obejcts are
inherently read-only this is a problem.
As a side effect of this fix there are no longer size limits in the
pattern to be used with GET/BY option of SORT.
See https://github.com/antirez/redis/issues/460 for bug details.
This reflects that REDIS_*FLAGS will only be used for compilation of
Redis and not for its dependencies. Similarly, that FINAL_*FLAGS are
composed of other variables and holds the options that are finally
passed to the compiler and linker.
Lua global protection can now be simpified becuase we no longer have the
global() function. It's useless to occupy memory with this table, it is
also not faster because the metamethods we use are only called when a
global object does not exist or we are trying to create it from a
script.
After considering the interaction between ability to delcare globals in
scripts using the 'global' function, and the complexities related to
hanlding replication and AOF in a sane way with globals AND ability to
turn protection On and Off, we reconsidered the design. The new design
makes clear that there is only one good way to write Redis scripts, that
is not using globals. In the rare cases state must be retained across
calls a Redis key can be used.
projects / redis.git / log
