Try using EMON profiling in Vtune, or some equivalent tool like oprof
- Vtune for Linux (you can search for the Windows version)
- oprofile
EMON (Event Monitoring) profiling => like a time based tool, but it can tell you what performance event is causing the problem. Although, you should start out with a time based profile first, to see if there is a particular instruction that jumps out. (And possibly the related events that tell you how often there was a retirement stall at that IP.)
To use EMON profiling, you must run through a list of events, ranging from “the usual suspects” to …
Here, I would start off with cache misses, alignment. I do not know if the processor you are using has a counter for RF port limitations – it should – but I added EMON profiling long ago, and I don’t know how well they are keeping up by adding events appropriate for microarchitecture.
It may also be possible that it is a front end, instruction fetch, stall. How many bytes are in these instructions, anyway? There are EMON events for that, too.
Responding to comment that Nehalem VTune can’t see L3 events: not true. Here is stuff I was adding to comment, but did not fit:
Actually, there ARE performance counters for the LL3 / L3$ / so-called Uncore. I would be immensely surprised if VTune doesn’t support them. See http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf points to VTune and other tools such as PTU. In fact, even without LL3 events, as David Levinthal says: “the Intel® Core™ i7 processor has a “latency event” which is
very similar to the Itanium® Processor Family Data EAR event. This event samples
loads, recording the number of cycles between the execution of the instruction and actual
deliver of the data. If the measured latency is larger than the minimum latency
programmed into MSR 0x3f6, bits 15:0, then the counter is incremented. Counter
overflow arms the PEBS mechanism and on the next event satisfying the latency
threshold, the measured latency, the virtual or linear address and the data source are
copied into 3 additional registers in the PEBS buffer. Because the virtual address is
captured into a known location, the sampling driver could also execute a virtual to
physical translation and capture the physical address. The physical address identifies the
NUMA home location and in principle allows an analysis of the details of the cache
occupancies.” He also points, on page 35, to VTune events such as L3 CACHE_HIT_UNCORE_HIT and L3 CACHE_MISS_REMOTE_DRAM. Sometimes you need to look up the numeric codes and program them into VTune’s lower level interface, but I think in this case it is visible in the pretty user interface.
OK, in http://software.intel.com/en-us/forums/showthread.php?t=77700&o=d&s=lr a VTune programmer in Russia (I think) “explains” that you can’t sample on Uncore events.
He’s wrong – you could, for example, enable only one CPU, and sample meaningfully. I also believe that there is the ability to mark L3 missing data as it returns to the CPU. In fact, overall the L3 knows which CPU it is returning data to, so you can definitely sample. You may not know which hyperthread, but again you can disable, go into single thread mode.
But it looks like, as is rather common, you would have to work AROUND VTune, not with it, to do this.
Try latency profiling first. That’s entirely inside the CPU, and the VTune folks are unlikely to have messed it up too much.
And, I say again, likelihood is that your problem is in the core, not in L3. So VTune should bne able to handle that.
Try “Cycle Accounting” per Levinthal.