The locality level as far as I know indicates which type of access to data has been performed. When a node finishes all its work and its CPU become idle, Spark may decide to start other pending task that require obtaining data from other places. So ideally, all your tasks should be process local as … Read more
I can think of a couple of possible explanations: Other applications on your system are using so much memory that there isn’t 300Gb available right now. There could be a resource limit on the per-process memory size. You can check this using ulimit. (Note that according to this bug, you will get the error message … Read more
I have found this article on the subject: http://www.tenereillo.com/GSLBPageOfShame.htm Basically if you do not require long lasting sticky sessions you can configure your DNS servers to return multiple A records (IP addresses) for your website. Web browsers are smart enough to try all the addresses until they find one that works.
As of Python >=3.2, os.makedirs() can take a third optional argument exist_ok: os.makedirs(mydir, exist_ok=True)
Found the answer buried in an old supercluster.org thread: qselect -u <username> | xargs qdel Worked flawlessly.
I’ve looked extensively at Zookeeper/Curator, Eureka, etcd, and consul. Zookeeper/Curator and Eureka are in many ways the most polished and easiest to integrate if you are in the Java world. Etcd is pretty cool and very flexible, but It is really just a HA key store so you would have to write a lot of … Read more
Blocking communication is done using MPI_Send() and MPI_Recv(). These functions do not return (i.e., they block) until the communication is finished. Simplifying somewhat, this means that the buffer passed to MPI_Send() can be reused, either because MPI saved it somewhere, or because it has been received by the destination. Similarly, MPI_Recv() returns when the receive … Read more
I’ve been doing A LOT of reading on the available options. I also got my hands on High Performance MySQL 2nd edition, which I highly recommend. This is what I’ve managed to piece together: Clustering Clustering in the general sense is distributing load across many servers that appear to an outside application as one server. … Read more
There were several options for HPC infrastructure considered: MPICH, Open MPI, and MS MPI. Initially tried to use MPICH2 but gave up as the latest stable release 1.4.1 for Windows dated back by 2013 and no support since those times. Open MPI is not supported by Windows. Then only the MS MPI option is left. … Read more