Hey, this is kind of offtopic, but figured still appropriate to ask;
How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and fire up clusters just for processing when needed), but they cannot interface with tools like the regular storage.
Meanwhile, Google's storage is like regular object storage, but you can run map/reduce (dataproc) and Spark on it.
Since Cloud Dataproc is ephemeral it's going to be a (probably) better idea to use GCS over HDFS so there is no data loss.
Technically, Cloud Dataproc clusters have both HDFS (on PD) for write/read-intensive operations (and scratch space) along with the GCS connector. GCS is not the default file system, however.
Most people (in my experience) don't use the old S3 block filesystem protocol anymore, and just use object storage instead (with either S3n, S3a or the proprietary EMRFS).
The Hadoop FileSystem interface doesn't really force an specific underlying implementation, and you can even use "local" filesystems without any issue, in fact, IIRC, MapR-FS is just an optimized NFS drive, i.e. a shared network drive.
How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and fire up clusters just for processing when needed), but they cannot interface with tools like the regular storage.
Meanwhile, Google's storage is like regular object storage, but you can run map/reduce (dataproc) and Spark on it.