20TB in an RDBMS has very different characteristics than 100GB in an RDBMS. Once you can no longer store your dataset in memory, you start to see greater variance in latency.
DynamoDB for us has given us very reliable latency for our datasets that are too large to hold in memory on an RDBMS.
You can get a box with 48TB RAM so 20TB has more to do with time it takes for DB to come up. To provision DynamoDB to the same performance as PG on i3.16xlarge instances you will be paying north of 50K/month vs 5K/month and that's with all limitations that come with DynamoDB.
An i3.16xlarge is 488GB, not 48TB. The x1.32xlarge (largest amazon offers) is 2TB.
Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB.
The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up around $6k. A 3 year reservation will get that down to $3k.
Comparable DynamoDB (3000 write units, 3000 read units, reserved IOPS) comes out to $1500.
The key here is consistent performance and low operational overhead. As I said, we're pretty happy with it.
> The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up around $6k. A 3 year reservation will get that down to $3k.
> Comparable DynamoDB (3000 write units, 3000 read units, reserved IOPS) comes out to $1500.
Note those aren't really comparable numbers - postgres will often collapse writes from concurrent sessions / statements.
Is there any place in the post I claimed i3.16xlarge has 48TB RAM? I said you can get a box with 48TB RAM. Did anywhere in the post I compared to RDS? i3.16xlarge is basically a dedicated box you have 8 NVMe PCI SSDs to be on super conservative side I was using 50,000 write IOPS 200K read IOPS in reality it can do way more.
Not with AWS. If you don't have AWS as a requirement, go for it.
> i3.16xlarge
An i3.16xlarge still won't hold a 20TB dataset in memory. You are going to see more variance in latency if you have a working dataset too big to fit into memory.
As far as getting superior performance out of an i3.16xlarge, that's fine if you have the expertise and resources to run PG yourself. However you're going to need replication, which will increase that cost. You're going to need failover mechanisms, backup, etc.
We have preferred RDS for Postgres because it gives us something operationally simple. We've found DynamoDB to be even simpler operationally and have more predictable performance. We have frequently considered running a self managed PG instance and have decided against it for our use case.
Ok let's try :)
Did I ever claim on 48TB on AWS ?
You are seriously claiming DynamoDB will be holding full 20TB in RAM if yes I have a bridge to sell you :).
The price for i3.16xlarge was quoted for 2 instances the IOPS used for calculation is 1/8th of actual max IOPS that instance can do.
When presenting an alternative to DynamoDB, AWS is an implied part of presenting that alternative. While the original article does suggest an alternative outside of AWS, it properly qualifies it as a competitor offering. Regardless, I don't care what was and wasn't claimed. It's not important.
> You are seriously claiming DynamoDB will be holding full 20TB in RAM
No. I'm claiming that DynamoDB will have predictable and consistent latency characteristics at 20TB. The related claim is that an RDBMS will not have consistent latency characteristics if the working dataset does not fit in memory.
"When presenting an alternative to DynamoDB, AWS is an implied part of presenting that alternative". If that is the way you guys operate well ... We run things in AWS, on prem and in GCP. Our AWS spend is over 3 mil. per month things that require hard latency guarantees are def. not running in AWS. Nothing is AWS has "predictable and consistent latency characteristics" simply due to the nature of how they operate.
