Best way to aggregate multiple log files from several servers [closed]

Probably the lightest-weight solution for real-time log watching is to use Dancer’s shell in concurrent mode with tail -f: dsh -Mac — tail -f /var/log/apache/*.log The -a is for all machine names that you’ve defined in ~/.dsh/machines.list The -c is for concurrent running of tail The -M prepends the hostname to every line of output.

Spread vs MPI vs zeromq?

MPI was deisgned tightly-coupled compute clusters with fast, reliable networks. Spread and ØMQ are designed for large distributed systems. If you’re designing a parallel scientific application, go with MPI, but if you are designing a persistent distributed system that needs to be resilient to faults and network instability, use one of the others. MPI has … Read more

What tools do distributed programmers lack?

OK, let me start. A distributed logger with a high-precision global time axis – allowing to register events from different machines in a distributed system with high precision and independent on the clock offset and drift; with sufficient scalability to handle the load of several hundred machines and several thousand logging processes. Such a logger … Read more

.net service bus recommendations? [closed]

NServiceBus is growing in popularity. It is open source as well. Here is a Hanselminutes episode with Scott Hanselman talking with Udi Dahan about NServiceBus to help grok it. You should definitely evaluate using it. UPDATE: There’s also a DNR TV episode which shows what it’s like to build an NServiceBus solution from scratch here: … Read more

In Apache Kafka why can’t there be more consumer instances than partitions?

Ok, to understand it, one needs to understand several parts. In order to provide ordering total order, the message can be sent only to one consumer. Otherwise it would be extremely inefficient, because it would need to wait for all consumers to recieve the message before sending the next one: However, although the server hands … Read more

Real World Use of Zookeeper [closed]

Free Software Projects Powered by ZooKeeper: AdroitLogic UltraESB Akka Eclipse Communication Framework Eclipse Gyrex GoldenOrb Juju Katta KeptCollections Mesos Neo4j Norbert Talend ESB redis_failover Apache Projects Powered by ZooKeeper: Apache Accumulo Apache BookKeeper Apache CXF DOSGi Apache Flume Apache Hadoop MapReduce Apache HBase Apache Hedwig Apache Kafka Apache S4 Apache Solr Source: https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy