The Demikernel is not a unikernel. It is a library OS compiled as a series of shared libraries. It is not compiled together with the application and doesn’t take into account what features the application uses. It is designed to work with kernel-bypass hardware, like DPDK.
It depends on the interface for the drivers to the application. However, UIO doesn't seem to support DMA, which is a non-starter.
RDMA and DPDK both use user-space drivers, which is necessary for kernel-bypass. I'm not advocating for a particular kernel-bypass solution. I'm arguing that if we use kernel-bypass for I/O, we should have a common, efficient, high-level interface for it.
High-level means not exposing hardware limitations to the application. The primary target applications are datacenter services, which spend much of their time processing network I/O. As network latencies lower to a few microseconds, datacenter applications like Redis will need kernel-bypass because the kernel will become too expensive for them. In our experiments with a 25Gb network, the Linux kernel and POSIX interface costs Redis 60% of its latency.
Network I/O is a major bottleneck for Redis but they leave a lot on the table by being single threaded. I can speak from experience because I maintain a Multithreaded Fork: https://github.com/JohnSully/KeyDB
KeyDB can easily get 2-3x the QPS with half the latency.
This is IMHO a wrong analysis. Redis can be scaled by being single threaded by running multiple processes: then if you remove the overhead of the network stack, each process can deliver more QPS, not just better latency. By using threads (which Redis now in parts also does, but and gets 2X performance by making threaded just 0.01% of the code, that is, a single function) you continue to incur in the I/O penalty, just amortized in more threads, but it continues to be a waste. Also the latency you measure as reduced with threads is an illusion: it happens only during benchmarks because the instance is saturated more when running on a single thread. If you measure single-request latencies, they are dominated by the network stack latency.
The lower latency is not an illusion, it is indeed lower latency for servers with high load. If you don't have high load then I agree the need for threads is eliminated - but people using Redis for real work have traffic where this becomes an issue. Multiple processes require clustering or sharding each with its own set of overheads (both in CPU and human terms).
You and I disagree vehemently on this (hence the fork), but I really think your optimizing for your own simplicity not that of the user's. It should be the opposite since the developer has the most insight into the software.
I don't think you understood my comment. What I mean is that regardless of what you think of Redis and threads the fact that doing IO is so wasteful and adds latency and CPU time remains and is a constant.
> High-level means not exposing hardware limitations to the application.
This seems counter-intuitive.
Hardware limitations mean different abstraction than OS-level APIs, as them to applications.
Even POSIX does not expose hardware limitations.
Rather, high-level in the paper is more like some suitable interface to a wide range of applications. I.e., high-level as it's targeted to be used directly by applications as a portable interface.
You said you don't want to make users deal with flow control and hardware details.. does that imply a userspace bypass library which does that stuff for us? Does it look posixy?
Solarflare's OpenOnload or Mellanox's VMA both show up as LD_PRELOADS that overload any traditional socket programming unless you want to code your apps to their API directly.
It looks POSIX-like but uses high-level queues and fixes some issues with epoll. The lack of an atomic data unit and the overhead of the poor epoll interface cost too much to retain for kernel-bypass. Take a look at the paper for more details.
No, that was our key insight. Basically, you can think of it as, using something like Paxos requires all replicas to run the same code. So every replica checks for transaction conflicts. However, you actually only need one replica to detect a conflict, so there is no reason for every replica to run the same checks -- this is wasted work. Instead, we just make sure that each conflict is checked by at least one replica.
The other interesting conclusion is that there are other workloads like this. For example, it is possible to build a reliable and better performing lock server in this way as well (and there is an implementation in the github repo). So, you'd get something similar to Chubby, but where the latency to the server is only a single round-trip in the common case.
Figure 14 in the paper gives comparisons with MongoDB, Redis and Cassandra using YCSB. The git repo has the YCSB bindings for TAPIR, so you can run TAPIR against any of the other systems that also have YCSB bindings.
I don't think there's any claims that it did. The paper and the talk both made it clear that Cassandra outperformed it. I think MongoDB is the only one it beat. But I think the point is that it's able to perform well while being consistent, not that it's faster than "weakly consistent" storage systems like Cassandra.
The leader bottleneck will continue to exist even if there great sharding because the leaders simply process more messages than the replicas, so TAPIR will allow each shard to support more throughput.
The paper has an evaluation for multi-data-center replication in Figure 12. We assume that the clients are web servers, so they are always close to one of the replicas, but not all of them. The result we found is basically that TAPIR performs better in the multi-data center case except when the leader is in the same data center as the client. So it depends on whether you can always guarantee that the leader is in the same data center as the client.
The abort rate continues to essentially track the latency needed for commit. So, TAPIR reduces the abort rate compared to OCC because it reduces the commit latency. At very high contention, locking is likely to make slightly more progress, but no systems with strong consistency will be able to provide high performance. If you are interested in some other ways to optimize for the high contention case, take a look at our work on Claret: http://homes.cs.washington.edu/~bholt/projects/claret.html
We also tested with high clock skew. The paper notes, "with a clock skew of 50 ms, we saw less than 1% TAPIR retries." Since the clients can use the retry timestamps to sync their clocks, it only adds an extra round-trip, so it still leaves TAPIR with the same latency as a conventional system, even in cases of extremely high clock skew.
