AMD EPYC<\/a> is the server processor that takes aim at the Intel Xeon and its dominant status on the data center market. The enterprise field is a high margin, high profit area and while AMD once had significant share in this space with Opteron, that has essentially dropped to zero over the last 6+ years. AMD hopes to use the same tactic in the data center as they did on the consumer side to shock and awe the industry into taking notice; AMD is providing impressive new performance levels while undercutting the competition on pricing.<\/p>\r\n\r\n Introducing the AMD EPYC 7000 Series<\/strong><\/p>\r\n\r\n Targeting the single and 2-socket systems that make up ~95% of the market for data centers and enterprise, AMD EPYC is smartly not trying to swing over its weight class. This offers an enormous opportunity for AMD to take market share from Intel with minimal risk.<\/p>\r\n\r\n Many of the specifications here have been slowly shared by AMD over time, including at the recent financial analyst day, but seeing it placed on a single slide like this puts everything in perspective. In a single socket design, servers will be able to integrate 32 cores with 64 threads, 8x DDR4 memory channels with up to 2TB of memory capacity per CPU, 128 PCI Express 3.0 lanes for connectivity, and more.<\/p>\r\n\r\n Worth noting on this slide, and was originally announced at the financial analyst day as well, is AMD’s intent to maintain socket compatibility going forward for the next two generations. Both Rome and Milan, based on 7nm technology, will be drop-in upgrades for customers buying into EPYC platforms today. That kind of commitment from AMD is crucial to regain the trust of a market that needs those reassurances.<\/p>\r\n\r\n Here is the lineup as AMD is providing it for us today. The model numbers in the 7000 series use the second and third characters as a performance indicator (755x will be faster than 750x, for example) and the fourth character to indicate the generation of EPYC (here, the 1 indicates first gen). AMD has created four different core count divisions along with a few TDP options to help provide options for all types of potential customers. It is worth noting that though this table might seem a bit intimidating, it is drastically more efficient when compared to the Intel Xeon product line that exists today, or that will exist in the future. AMD is offering immediate availability of the top five CPUs in this stack, with the bottom four due before the end of July.<\/p>\r\n\r\n\r\n\r\n EPYC will take form in 32-core, 24-core, 16-core and even an 8-core option, with base clock speeds ranging from 2.1 GHz to 2.4 GHz, and Turbo clock rates of up to 3.2 GHz. And that peak Turbo clock is on the 32-core part (!!) though the “all core loaded” turbo will be around 2.7 GHz. TDPs will be either 180 watts or fall into the 155\/170 watt segment, depending on the speed settings of DDR4 memory. The baseline 8-core part, mainly targeted at systems that need storage and IO connectivity, will run at 120 watts. (If you are wondering why that is higher than consumer parts that run at much higher frequency, remember that the added PCIe and DDR4 channels increases the power draw considerably.) Also, because the EPYC 7000 series of CPUs includes the south bridge functionality of USB 3.0, SATA and more, the TDP calculations for system builders do not need to add in the power of a chipset as they do on Intel’s Xeon line.<\/p>\r\n\r\n All the EPYC 7000 series product line will offer 8 channels of DDR4 memory, with speeds hitting 2666 MHz. That is double the current Xeon family and 33% more than the 6 channels expected on the Xeon Scalable product coming out later this year, thus providing substantially higher memory bandwidth and up to 2TB of memory capacity in a 2-socket system. Every EPYC 7000 CPU will feature 128 lanes of PCI Express, allowing data center and workstation builders to utilize substantial resources outside of the processor including graphics cards, networking, storage controllers, and more.<\/p>\r\n\r\n AMD claims to have significant performance advantages in a 2P (two socket) configuration compared to the equitably priced Xeon product line. At the flagship level, comparing the >$4000 EPYC 7601 to the Intel Xeon E5-2699 v4, AMD expects to have 47% faster SPECint performance. Keep in mind that the E5-2699 v4 is a 22-core\/44-thread processor, so that performance advantage AMD holds is a result of having 45% more cores at the same price point. Clock speeds of the Xeon part are comparable, with a 2.2 GHz base and 3.6 GHz Turbo clock.<\/p>\r\n\r\n As you move down the price \/ competitive metrics on the slide above, the advantages to AMD increase slightly, going as high as 70% for the ~$800 product line, again using SPECint as the basis for this comparison. Though I assume we are all intelligent enough to understand that one benchmark does not make the case for an entire platform, AMD seems eager to get more benchmarks and workflow comparisons in front of the media, proving its case for the value EPYC will provide. And even though we know we will see higher core count processors in the upcoming Skylake refresh of the Xeon product family, all indications are that pricing from Intel remains stable (or is going up), so AMD should maintain its performance advantages using price as the target metric.<\/p>\r\n\r\n Though the single socket platform for servers is a significantly smaller market, AMD has a few interesting SKUs built specifically for it.<\/p>\r\n\r\n These solutions are useful for storage systems or GPU-centric builds that don’t need a significant amount of CPU compute power. Customers will still get access to 8-channels of DDR4 memory and 128 lanes of PCI Express, something that comparable Intel single-socket solutions will not be able to match.<\/p>\r\n\r\n AMD is running the comparison of a single EPYC processor against a pair of Intel Xeon processors, with the obvious performance advantage going to AMD in SPECint once again.<\/p>\r\n\r\n The software ecosystem is of critical import for the early success of AMD EPYC, and though there is clearly some additional work and partnerships to come, AMD feels confident that the collection of software partners it has on-board for the launch window is solid.<\/p>\r\n\r\n All the key operating system and hypervisor companies appear to be working with AMD to prepare the software infrastructure for use. Microsoft, Red Hat, Ubuntu, VMware, Citrix, etc. and development tools like Visual Studio and GCC have libraries and compilers in place to help give AMD the steady footing it needs to push upwards with EPYC.<\/p>\r\n\r\n In a similar way, AMD has been working with the hardware ecosystem to prepare for today’s release, bringing many of the industry’s top-level competitors into the fold.<\/p>\r\n\r\n HPE, Dell EMC, Tyan, Supermicro, and even Lenovo are building platforms for EPYC. <\/p>\r\n\n\n The basic Zen architecture found in the Ryzen processor, that we have discussed and debated many times on PC Perspective in the past<\/a>, remains unchanged for EPYC. The core was designed with server and data center placement in mind, according to AMD’s briefing I took part in yesterday. There are a lot of discussions and accusations from outsiders that have called the AMD EPYC processor simply a “glued together” Ryzen CPU, attempting to construe this platform as a desktop part simply repurposed to act in the server environment. Despite the fact that it really doesn’t matter what the pedigree of the architecture was, only how it performs in the necessary workloads and environments, AMD did give us more detail on the die to die and socket to socket communications to counter.<\/p>\r\n\r\n What AMD calls Infinity Fabric is actually a collection of interfaces that range in use from intra-die to inter-die to inter-socket. While the specific details of how these may be similar, or different, are still currently not detailed, we do now know the performance specifications between these to help judge the capability they offer.<\/p>\r\n\r\n AMD EPYC, and the upcoming Threadripper consumer HEDT parts, are multi-die packages. EPYC will have four dies on each CPU package, regardless of the number of enabled cores. AMD is disabling cores in a symmetrical pattern, so a 32-core part will have four dies with all 8 cores enabled on all four. A 24-core processor will have 1 core of each CCX, and thus 2-cores per die, disabled. A 16-core part will have 2 cores of each CCX disabled, 4 per core. And the 8-core part will have 3 of the 4 cores per CCX disabled, leaving only per core, and 2 per die.<\/p>\r\n\r\n I am still waiting for input from AMD on this, but it does bring up concerns of the L3 \/ thread-to-thread latencies of the architecture that we have discussed in the past<\/a>. In the worst-case scenario, the 8-core design, there would only be one core per CCX which would require all inter-core communication to happen through the L3, maximizing latency. Couple that with the unknown quantity that is latency from die to die (or even socket to socket) and you have an interesting comparison point between platforms to dive into. I am working to get more information on this, as well as hardware to test and compare.<\/p>\r\n\r\n What AMD has shared to this point is some impressive bandwidth numbers to help alleviate any concern with the multi-die package implementation.<\/p>\r\n\r\n AMD has built low power, low latency links between the cores that offer as much as 42 GB\/s of bi-directional bandwidth at extremely low power. Every die is connected to every other die, enabling single hop data travel between any two. They can run at extremely low power states when the cross traffic is minimal, keeping TDPs of the processors low.<\/p>\r\n\r\n When we look at the socket to socket bandwidth and connection diagram, each die is connected to the matching peer die on the other socket with a 38 GB\/s bi-directional link. That gives EPYC at most two hops of latency to traverse between any two cores on the system, with a total aggregate bandwidth availability of 152 GB\/s.<\/p>\r\n\r\n The IO and connectivity portion of the processor supports eight x16 PCIe links, getting us to the magic 128 lane number. Each die supports two of them, though one is used for the socket to socket interface in a 2P system, leaving each CPU with 64 lanes of PCIe, a total of 128 for the system. Each of the links supports 32 GB\/s of bandwidth and 256 GB\/s per socket – a substantial amount of potential throughput. Those connections can be divided into as many as 8 PCIe devices per x16 link, totaling 64 possible x1 PCIe connections. How about that for a coin mining server?<\/p>\r\n\r\n When you put it all together, it might look a bit messy, but the bandwidth and connectivity is there to make EPYC a powerful server processor and platform. The only question that remains unanswered for me is the “low latency” part of this slide that hasn’t been quantified…and that we haven’t tested yet.<\/p>\r\n\r\n AMD also claims that there are additional benefits to its multi-die design. For one, the amount of combined die space from these four chips is larger than a reticule can produce with today’s lithography technology. Essentially, AMD claims this product could not have been built as a monolithic die in its current configuration. This method does help AMD increase yields and thus better attack the high end of the server \/ workstation market with more products at the top of the stack. The flexibility to configure dies is clearly an advantage over a single, monolithic die.<\/p>\r\n\r\n A Performance Example<\/strong><\/p>\r\n\r\n In the build up to today's release, AMD brought media to a room full of demos of EPYC at work, one of which stood out particularly to me. Using a single socket system, and access to the full 128 lanes of PCI Express from the EPYC 7601 processor, AMD was able to break world records for IOps using the FIO storage benchmark. Paired with the EPYC processor were 24 NVMe SSDs from Samsung, PM1725A in this case, all running with access to a full x4 of PCIe 3.0 bandwidth. At 3.2TB each, the total capacity of this software defined array was 76.8TB!<\/p>\r\n\r\n These numbers are incredibly impressive – 9.1M IOPS read, 7.1M IOPS write (both running at full random 4K), and 53.3 GB\/s of storage bandwidth when run at 128K random! Even more impressive for the EPYC platform is that the server still has 32 lanes of PCI Express remaining for networking hardware, compute resources like accelerators, or more storage controllers. This is a clear example of how the massive amount of IO connectivity AMD has brought can change the data center TCO landscape.<\/p>\r\n\r\n Closing Thoughts, for now<\/strong><\/p>\r\n\r\n AMD is in a significantly different space today than it was only 4-5 months ago with CPUs. It has gone from a lingering memory in the minds of gamers and DIY builders to a prominent player in the field. It has revamped interest in enthusiasts and OEMs like Dell for high-end gaming PCs and mainstream desktop builds. And today it prepares to make the same shift for the server and enterprise markets, launching the EPYC data center platform that returns competition to a market that has had a single major player for more than half a decade.<\/p>\r\n\r\n Based on the data I have seen, the products as they are described to me, and the ecosystem in the state that it currently resides, it’s hard to imagine AMD not being able to make significant headroom in this field. The definition of “significant” is going to vary depend on who you ask. Those that are wishing for the return of the Opteron peak will target a 20%+ market share as the necessary milestone. Intel might view even a couple percentage points of its highly profitable Xeon market as significant. To me, AMD management should probably be looking at a double-digit goal by 2020. That will shift the views and opinions of AMD from outsiders, help stabilize a financially taut corporation, and will open gates and allow for more customers to feel comfortable with the product line.<\/p>\r\n\r\n But let’s be clear, though it should be an easily attainable goal to gain market share where you have almost none, there are roadblocks. AMD needs to prove the product can perform and in more than just SPECint benchmarks. Considerable work is done by Intel on a yearly basis to optimized its hardware stack to meet the needs of the major platform players (think the Super 7<\/a>). AMD needs to make a performance and a cost argument to these groups that will turn heads.<\/p>\r\n\r\n AMD must also avoid any potential platform pitfalls that plagued the Ryzen consumer launch. Data center customers have zero tolerance for that and mission critical systems and data need to be running at 100%.<\/p>\r\n\r\n Can they do it? Absolutely. Will they? Hopefully we’ll see more in the coming days and weeks to prove they are on the right track.<\/p>\r\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":2,"featured_media":170745,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":""},"categories":[13266],"tags":[8,1854,10959,11123,7777],"_links":{"self":[{"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/posts\/170744"}],"collection":[{"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/comments?post=170744"}],"version-history":[{"count":2,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/posts\/170744\/revisions"}],"predecessor-version":[{"id":231266,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/posts\/170744\/revisions\/231266"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/media\/170745"}],"wp:attachment":[{"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/media?parent=170744"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/categories?post=170744"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pcper.com\/wp-json\/wp\/v2\/tags?post=170744"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"epyc-13.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/1017-epyc-13.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-14.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/2e9e-epyc-14.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-19.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/89c3-epyc-19.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-21.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/6095-epyc-21.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-22.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/b29f-epyc-22.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-27.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/5eca-epyc-27.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-28.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/36d5-epyc-28.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\nArchitectural Outlook<\/h5>\n
![\"epyc-15.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/0da5-epyc-15.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-72.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/d2f9-epyc-72.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-73.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/2737-epyc-73.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-74.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/b487-epyc-74.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-77.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/d11a-epyc-77.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"demo1.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/cfb4-demo1.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n![\"epyc-58.jpg\"](\"https:\/\/pcper.com\/wp-content\/uploads\/2017\/06\/38b1-epyc-58.jpg\")
<\/span><\/a><\/div>\n<\/p>\r\n\r\n