Python is ideally suited for data science and machine learning because 80% of the work tends to be data exploration, transformations and feature engineering and 19% of the work is iterative development of models. In such workflows the lack of type checking & simple REPL is a huge feature. Most of the underlying code for ML is anyway written in Fortran or C and Python becomes a very convenient front end glue
For the 1% work that is actually engineering the model's deployment in production, it's not uncommon to use faster languages with the exported model.
If what you're doing is writing a vanilla CRUD server that gets thousands or hundreds of thousands of hits a second, Python is probably a bad choice.
I’m not sure how many people will ever even work on CRUD apps of that scale.
Even a mere 1,000 req/sec adds up to 86.4M per day. That sounds like something at least as popular as Stack Overflow, for example. How many web apps are there, really, that have millions of users?
Isn't that market dominated by a few, or at the max, no more than 10-15 companies? My point is just that are there really more than a few dozen or maybe a hundred companies that see more than 1000 req/sec (on web apps)?
Depends if you're talking maximum or average. Getting to the front page of HN could easily give you a burst of 1000 req/sec. I would agree there's very few CRUD apps that deal with 1000 req/sec on average.
On the other hands, there's various APIs and tracking type of services that probably do handle well in excess of that, being run by not-large companies none of us have ever heard of.
I think Python is at a weird place regarding data science.
If you need easy-to-use tools, R libraries cover significantly more ground than Python. You don't have to worry about 2/3 compatibility of niche libraries. Plus the functional language forces the libraries to have similar API's. You also get to use tidyverse and ggplot2 in all of their majestic beauty. At most you have to write a "%>% do() %>%" to get around weird library API choices.
However if you are in the frontier, you could avoid so much awkward notation with an array-first language like Matlab or Julia. You have "^" instead of "np.linalg.matrix_power", "/","\" instead of "np.linalg.lstsq", the . operator and so much more life-saving tools.
R has so many problems itself! Attempting to run it at any sort of scale is basically impossible. Its dependency management is atrocious. You can't thread or multi-process effectively. The amount of overhead to even install R on a system is incredible, and amazingly its documentation is even worse than Python. We use R as our foundational Data science language and it has caused nothing but immense pain for everyone attempting to wrap the packages in scalable software.
None of the libraries that do the computations are written in python. None of the GUIs or matplotlib are written in python. If you want to build large projects or need performance, python is not the right language. I do like python a lot - for the things it's good at.
no, the trivial frontend is built in python; the real code is usually c++ or c.
Ignore that reality if you want to, but it is a fact.
Big complicated python projects are seldom pure python, they are usually a friendly python frontend to a serious application written in something else.
It seems in no way remarkable that someone wanting to build a serious backend type piece of functionality would pick another language that was, just for example, multithreaded.
I don't understand this logic. The users are learning Python, not C++ when they're trying to learn data science or implement a machine learning model. Should I say Tensor flow isn't written in C++ but CUDA or OpenCL? Any self respecting researcher is training their models on GPU or FPGA, not CPU.
The point is that Python is the entry point for large majority of data scientists currently and its absolutely disingenuous to try to dispute that reality.
The key word here is "users". Python is fantastic for users. It allows users to get things done without having to worry about types or memory or the underlying hardware in any way.
I’d say it’s very disingenuous to claim that Python users don’t have to care about types. Many types are directly related to semantics, so of course they have to care about types!
I think Python made some really good choices with their types from UX that lead to you having to care more about your logic than the machine. Only one integer type, for example! For most users it’s perfect UX.
However, I think (possibly due to the early time when a Python was made) some of the decisions made practical but regrettable trade offs. Duck typing is ultimately just type inference with strong performance penalties and really weak static tooling, for example. _Many_ static type systems are still very far from the user’s actual domain, but the core techniques don’t have to be.
Not just data science: the entire machine learning and scientific computing world, if we’re honest. Need to do huge numerical linear algebra operations very fast with great memory safety and ease of use? Python.
No it isn't? The substantive portions of the software aren't written in Python. Python is a thin interface to the actual library. Users of the library who need to rapidly prototype can leverage the library using its Python frontend, but the authors of the library did not write the library in Python.
To program in CPython is to use thin wrappers around C-level implemented extension types and modules. For example, all major built-in container types.
If you program in Python for performance-critical applications, as I do, then you will frequently hand-write C-level extension modules, write extension modules in Cython, use JIT tools like numba.
even for the crud apps so frequently referred to in this thread the equivalent 'real code' is the sql rdbms written in c or c++ no matter what the webapp language is. i don't advocate using python in million line monoliths despite it being my favorite lang in general at the moment but the volume of cognitive dissonance and false dichotomies in this thread is crushing.
Oddly in biology I’m seeing an uptick in R use as it’s easier to install and run tools in R for non programmers. The python 2/3 confusion (why doesn’t this google result work?), and various install packages (conda/pip...) seem to have made R
more popular. This is despite biopython which is quite good. The newest single sell rna seq analysis tools are in R.
R is a strang thing to me, though it has its moments (graphing). I still think pandas which gives python R like data frames is one of the great tools out there.
pandas gives base R a run for its money, but I find the multi-indexing really confusing, and R tidyverse is just way way nicer than pandas.
The sentiment that R is easier than python to install/manage is common among some R users but I disagree. With R I'm constantly facing dependency hell problems -- one package wants an old version of R, while another needs the newest. Conda/venv solves this problem very nicely in python.
Recently lI've been using Julia more and really like it. One nonstandard case where I've found it really shines is parsing large bioinformatics data, such as pileup files. Python is just so slowww here, but neither do I want to write a C program to do the text parsing. Julia is perfect in this case.
