We use Python in conjunction with h5py, numpy/scipy and boost::python to do data analysis. Our typical datasets have sizes of up to a few hundred GBs. HDF5 advantages: data can be inspected conveniently using the h5view application, h5py/ipython and the h5* commandline tools APIs are available for different platforms and languages structure data using groups … Read more
Wes is of course right! I’m just chiming in to provide a little more complete example code. I had the same issue with a 129 Mb file, which was solved by: import pandas as pd tp = pd.read_csv(‘large_dataset.csv’, iterator=True, chunksize=1000) # gives TextFileReader, which is iterable with chunks of 1000 rows. df = pd.concat(tp, ignore_index=True) … Read more
My two cents: If you are concerned about the edge case failures that may occur with SATA hardware (specifically, lousy SATA controllers), spend the money on real SAS disks. These cards do what they say on the tin: They translate SAS (SCSI) commands to SATA commands, and even implement a few themselves (like power- and … Read more
The sideband cables are used to connect the controller to a managed backplane or disk enclosure, so that it can monitor the controller as well as the disks.
It’s a case of math. I/O-per-second is determined by the following calculation: Random I/O = 1000/ (average latency + seek time) For a 7.2K RPM drive, a seek-time of 8.5ms and latency of 4.16 gives an IOPS number of 78. For a 15K RPM drive, a seek-time of 2.6ms and latency of 2.0ms gives an … Read more
It works. SATA discs are compatible with SAS. I have tons of them in SAS backplanes. Work like a charm.
The “SATA = 7.2K RPM, SAS = 10/15K RPM” mind-set is strong, and (in my opinion anyway) where most of the “SAS is faster than SATA” thinking comes from. There are some slight differences between SAS and SATA drives, notably in their on-board caching algorithms (NCQ vs. TCQ). However, the performance difference of equivalently specced … Read more
Marketing. 7.2K drives are slower and easier to produce, and with higher error thresholds which improves yields (and capacity). However, in terms of I/O operations each discrete disk can support, the 7.2K drives are markedly less performant than their faster brethren. Therefore they get the ‘Nearline’ moniker, as they’ll hit I/O saturation much faster than … Read more
This has been covered here… See the related links on the right pane of this question. Right now, the market conditions are such that you should try to use SAS disks everywhere you can. Enterprise SAS disks are your fastest and most resilient rotating media available at 10,000 and 15,000 RPM. Performance-optimized Nearline or Midline … Read more