Interact Contact Scratchpad 1 0 5

The ExaLINK Fusion has two internal module bays, one of which is typicallypopulated with an FPGA module running Exablaze firmware enabling the creationof mux & switch objects etc. A firmware development kit is available fromExablaze which enables users with an in-house FPGA development capability to addcustom FPGA functionality into the ExaLINK Fusion. A basic version of thisdevelopment kit is available free of charge from Exablaze.

• Interact Contact Scratchpad 1 0 50
• Interact Contact Scratchpad 1 0 56
• Interact Contact Scratchpad 1 0 58
• Interact Contact Scratchpad 1 0 5 0

There are two known variants of Scratch 1.0: 1.0.1 and 1.0.2. 1.0.1 was never released; only internal builds are available. A build from January 24, 2007 can be downloaded here. Net Scratch was based on this version. 1.0.2 was also not officially released. A build from sometime in 2007, most likely February–March is available here. Interact Scratchpad 1.0.5 – Convert text to contacts. Copy text from email. Signatures, web pages, or any document, and let the Interact Scratchpad turn it into a contact for you! Interact sits in your menu bar, or is available in the 'Share' contextual (right-click) menu on text selections. Sketchpad: Free online drawing application for all ages. Create digital artwork to share online and export to popular image formats JPEG, PNG, SVG, and PDF.

The deployment of user FPGA firmware to the Fusion can be either be done in thefirst FPGA module (ie module X), in which case the user's firmware runs insteadof the Exablaze firmware, or in the second module (ie module Y), in which casethe system is still able to create mux/switch objects in the first module,and the users firmware runs in the second module. It is also possible to populateboth FPGA modules running user firmware. The transceivers in the user's FPGA canbe connected directly to front panel ports, or into the other FPGA. If one FPGAis running Exablaze firmware, this means that the user's FPGA can be connectedto mux/switch objects.

We have reference designs available using this Firmware Dev Kit (FDK), so pleasecontact us for further details if you are interested.

The type of FPGA in the Fusion FPGA module is aXilinx Kintex UltrascaleXCKU115-2FFVD1924C:

Block Diagram of the Fusion FPGA Module

Connectivity

Transceivers

All 52 transceivers are available for use. 48 of these are connected to thelayer 1 Crosspoint, where they can then be patched through to front panelports, or back up into the other internal module. The final 4 transceiversare exposed to a high speed connector that can be directly connected acrossto the other module, bypassing connection through the Crosspoint.

Note that whilst the GTH transceivers in the Kintex Ultrascale FPGA supportoperation to 16Gb/s, the rest of the data path from the FPGA to the frontpanel is limited to 12.5Gb/s.

Sideband Interface

There is a High Speed USB interface between the Mainboard and FPGA module,which is presented to the FPGA as an 8 bit interface along with severalhandshaking bits. Exablaze uses this interface for setting up operatingparameters of the FPGA when running mux or switch firmware. When runningcustom firmware, this communications channel can be exposed out the 10/100Management Interface port on the front panel. A server is setup on the managementprocessor, and a client can connect via TCP. Any data sent/received throughthis connection is routed to the FPGA module, allowing users to interact withtheir design through a non-high speed port for accessing status and configurationparameters. Refer to the serial serversection for more information. Icons8 5 7 2 cr2 file.

Memory

In addition to the ~10MByte of block RAM inside the FPGA, there are a number ofadditional memories available on the FPGA module.

DDR4 DRAM

There is also a dual DDR4 DIMM interface into the FPGA for applications requiringmuch higher memory density, accessed using Xilinx's MIG controller.

• In order to fit within the height constraints of the Fusion chassis, Very Low Profile (VLP) DIMMs must be used. These have a maximum height of 18.8mm
• Data rates of up to 2133MT/s (PC4-17000) are supported
• Dual-rank ECC (72bit) DIMMs are supported in a x8 configuration
• 16GB VLP DIMMs are currently available from several manufacturers.

Please refer to the Xilinx Ultrascale Memory IP Product GuidePG150for more information.

EEPROM

A small 1kbit EEPROM is connected to the FPGA via SPI for non-volatile storageof operating parameters etc.

Oscillators and clocking

• All transceivers can either be clocked from a fixed 161MHz oscillator, or a low jitter programmable oscillator for data rates other than 10Gb/s.
• There is a 10MHz TCXO (100ppb stability) fitted as standard for general purpose clocking. This can be upgraded at the time of purchase to a high stability OCXO with a stability of 0.2ppb.
• A PPS signal can be connected to the front panel and routed to both FPGA modules through the Mainboard.

Setup

In order to run custom firmware, the FPGA module must first be set to functioncustom:

When a module is set to the function custom, the user has the ability topower on/off the module:

A bitfile is then copied across to the Fusion using sftp:

Once the bitfile is transferred, the Fusion is instructed to associate it withthe particular FPGA module, and then start configuration using the reconfig command:

These commands form part of the running-config. This can be saved to thestartup-config in which case the Fusion will setup and reconfigure the FPGAwith the specified bitfile upon reboot/power up.

Note: During development it is not necessary to transfer bitfiles to theFusion - you can configure the FPGA directly from Xilinx's Vivado Design Suite,however the module must still be set to the custom function. Refer to thesection on Debugging with Vivado formore information.

Operation

The 48 transceivers in the FPGA which are connected to the Fusion layer 1Crosspoint are referred to as X1, X2,., X48 and Y1, Y2,., Y48 depending on themodule. These are ports like any other portin the system, so can have an alias and description set, be patched to another port,added to a mux object etc.

Note: Setting the speed of a port in this case just configures the signalrecovery circuitry for optimal performance, there is no way the management interfacecan change the speed at which the FPGA PHY/MAC in the user's application operatesat - this is left to the user to handle.

Connecting to the Fusion

Xilinx Virtual Cable (XVC) is a TCP/IP-based protocol that acts like a JTAGcable and provides a means to access and debug your FPGA design without usinga physical cable. An XVC server can be enabled in the Fusion to allow remoteconnection to the FPGA module.

Vivado connecting to the Fusion

Once the FPGA module has been configured for custom use, the XVC server isenabled as follows:

Warning

Note there is currentlyno authentication or other security provided on this connection (other thanthe Access Control List for the wholemanagement interface).

Note

Note that enabling the XVC serveris not stored as part of the Fusion's config, which means it's not listed underthe running-config and that enabling the XVC server must be done each time theFusion is rebooted. Use the show module command to see the current state.

After the XVC server on the Fusion has been enabled, open a Vivado Hardware Managersession as shown:

Opening the Hardware Manager

• Start a Hardware Server session with the following command in the Tcl Console:

  connect_hw_server

Ignore the error that may appear if you have no local targets.

• Connect to the Fusion with the following command in the Tcl Console.

  open_hw_target -xvc_url 192.168.220.14:2542

The -xvc_url switch tells Vivado the IP address and port to communicate with.

You should now see the xcku115 FPGA inside the Fusion listed, and it can now beoperated on as normal. For example, right click on SysMon and selectDashboard->New Dashboard to view temperature and voltage status:

SysMon dashboard showing Fusion FPGA status

Refer tohttp://www.xilinx.com/products/intellectual-property/xvc.htmlfor more information on Xilinx XVC.

Warning Macos mojave 10 14 4 combo update.

Exablaze makes use of bitstream encryption to protect the configuration imagesshipped with the Fusion. The xcku115 EFUSE registers are burnt with encryptionkey information. The user must NOT attempt to modify any EFUSE settings onthe primary Fusion FPGA or Exablaze firmware can no longer be loaded(unrecoverably) and warranty will be voided. Screens 4 6 8 inch. User bitfiles should be generatedwithout any encryption settings.

Configuring the FPGA

In order to configure the FPGA with your custom design, right click on xcku115_0and select Program Device. A window will appear where you can select the bitfileand click Program. After several seconds, the FPGA will be configured with yourbitfile. If your design includes cores such as ILA/Chipscope, MIG or IBERT,these will be displayed under the FPGA along with SysMon.

Once you have finished development of your custom FPGA image, you can transferthe bitfile to the Fusion for loading via the management processor, ie withoutneeding to maintain a Vivado session. Refer to theConfiguration section for further details.

An example design that instantiates 48 transceivers with IBERT cores is available fordownload here.

First ensure the FPGA module is configured for custom use:

On the Fusion CLI, create several loopbacks by patching internal module portstogether, for example:

Enable the XVC server and connect Vivado as described above, then configure theFPGA with the example IBERT bitfile.After the FPGA is configured, you will be asked if you want to auto-detect serialI/O links for the IBERT cores - select Yes.

Depending on how many patches you created above, you will see a number of links detected.An eye scan can be done on these links by right clicking one and selecting Create Scan,and clicking OK.

Eyescan of loopback link using Vivado and IBERT

Note these ports can be patched to frontpanel ports, enabling all 48 ports to run PRBS tests on external equipment.The PRBS pattern can be selected as appropriate, bit errors can be injected etc.

Connection

There is a communications channel between the FPGA and the Fusion managementprocessor. When running custom firmware, this channel can be exposed out of the10/100 Management Interface on the front panel. A server is setup on themanagement processor, and a client can connect via TCP. Any data sent/receivedthrough this connection is routed to the FPGA module, allowing users to interactwith their design through a non-high speed port for accessing status andconfiguration parameters.

To enable this connection the following command is used:

Any port above 1024 can be used for this connection.

Warning

Note there is currentlyno authentication or other security provided on this connection (other than theAccess Control List for the wholemanagement interface).

Framing Protocol

The firmware development kit example uses theHDLC framingprotocol to send messages over this interface. HDLC is a serial framingprotocol that uses comma and escape codes in order to delimit frameboundaries. The HDLC frame is byte oriented and is structured like so:

Since byte 0x7E is a framing flag, user code should never try to send a bytewith this value in any field apart from the SOF/EOF flag fields. Instead, usercode should first transmit an escape character, 0x7D, followed by the flag bytewith bit 5 flipped (0x5E). If the user code needs to transmit 0x7D, it mustfollow the same procedure and send 0x7D followed by 0x7D with bit 5 flipped(0x5D).

As an example, for the message payload:

Would be encoded:

Example registers

The example design provides an example of using the HDLC stream protocol toexpose a 32 bit memory mapped register interface. This registers interface sitsat address 0 in the HDLC address space, and allows for read/write messages. Thepayload structure for this example is as follows:

As an example, to write to register 1 the value 0x01020304, you would send aHDLC frame that looks like:

For both read and write requests, the FPGA will respond with a HDLC frame thathas the same format as a 'write request', however for register read requests,the register value field will contain the value as read from the register

Overview

Exablaze provides reference designs to get started with FPGA development using the ExaLINK Fusion. These examples also include our ultra low latency PCS/MAC. Depending on the configuration, some or all of the 48 transceiversconnected to the crosspoint are instantiated. Users are given the freedom to choose the example application to build from.

The following examples (referred to as targets) come with our development kit, including the requisite source code for each.

• A 'native loopback example' implements a loop-back application, where the received packets on a particular port is looped back out of the same port
• A 'native trigger example' shows how to pre-load ExaLINK Fusion with a replyahead of time, and send it based on a simple mask/pattern match over received frames

A DDR4 memory controller is also included in the design, however it is commented out by default to reduce build time.

Licensing

The ExaLINK Fusion development kit can ship as a time limited evaluationversion, which is fully functional for up to two hours. After this, features ofthe evaluation version will progressively shut down and stop working. Afterthis time the host can be rebooted/power cycled to reset the two hourtimer. Windows plugins on mac. Should you wish to purchase a full license and have this time limitationremoved, contact your reseller or our sales team todiscuss licensing options.

Exablaze provides a license file which Xilinx Vivado will require in order to synthesize the 'full' version of the development kit. Note that this is not required or provided for 'evaluation' FDKs.

This file, exablaze_fdk.lic, is included at the root of the tarball you will have received. This can be installed either locally or on a license server.

For local installation, the license file can be placed in your ~/.Xilinx/ directory. You may also need to update your ~/.flexlmrc file to ensure that it has the ~/.Xilinx/ directory in the license search path, it should contain a line similar to:

For installation on a license server, the license file should be added as a parameter when starting the FlexLM License Server, for example:

If Vivado is unable to locate the license, an error will be generated similar to this:

Please refer to the Xilinx documentation and this FAQ for further details.

The Fusion FDK can also ship with a 100M & 1G PCS/MAC along with the alreadyincluded 10G PCS/MAC. All the supplied example applications will work for the100M & 1G PCS/MAC without any modifications, as long as all the associated portsare configured with equal line rates. The examples may not work as intendedwhen a different line rate is used on downstream and upstream ports as there isno buffering added. Note that the inclusion of 100M & 1G PCS/MAC inthe FDK will introduce an additional pipeline stage for 10G in order to meet thetiming requirements. This will increase the overall latency of the 10G TX pathby one clock cycle.

Due to this extra clock cycle of latency, we do not include the 100M or 1GPCS/MAC by default. If 100M and 1G support is required, customers will have tocontact Exablaze and ask for a special build that will also include the 100M &1G PCS/MAC. Also note that the default configuration of the FDK will only enable8 ports of Fusion device. Should you require more ports to be enabled, pleasespecify that while contacting Exablaze.

Software Installation

Prior to using the development kit, customers must install Xilinx Vivado 2015.4 or later, which can be downloaded from the Xilinx website. A synthesis license is required for the FPGA device used in the Fusion (XCKU115-2FFVD1924C). A free 30 day 'webpack' license is available from Xilinx.

Warning

Exablaze has observed instances where incorrect logic is synthesized using Vivado 2016.1 and 2016.2 so use of these versions is not recommended.

Build System

The ExaLINK Fusion development kit ships with a build system for various fully functional target example applications. The build system consists of a Makefile and a Vivado TCL script (compile.tcl). The Makefile launches Vivado and instructs it to run the TCL script. All project source files and the Vivado build process are defined within the TCL script.

The Vivado environment must first be sourced by running the following (change path to suit):

The Makefile expects a TARGET to be provided. The default targets (each of which are contained within their own directory under the src/ directory) are:

• native_loopback_example
• native_trigger_example

The development kit is delivered as a tarball. Once this is extracted, the reference design can be built using following command:

Users can also create their own targets in a new directory under src/. The build system will generate a number of files in the outputs/ directory, including a standard firmware image with a .bit extension that be can be transferredto the ExaLINK Fusion.

Exablaze build options

There are several build options that are available for the FDK that Exablazeneeds to set at the FDK build time, rather than at the customer's build time.The file buildlog contains information as to what the build options were setto at the time the particular FDK was generated by Exablaze. The build optionsare:

• FDK Type: Specifies whether this FDK is a Full or Demo.
• Number of Ports: Specified the number of ports that are enabled in Fusion. The minimum value is 8 and the maximum is 48.
• Extra RX Reg: Specifies whether an additional register stage is added to the internal MAC RX path. This does not impact the user interface/timing diagrams. If this additional register is not included, the MAC latency will improve by 3.1ns, however it will make timing closure more difficult.
• Extra CRC Reg: Specifies whether an additional register stage is added to CRC RX path. This improves timing closure, but delays the assertion of the crc_fail flag. If this register is added, crc_flag is valid 2 cycles after eof, otherwise it's valid on the cycle after eof.
• Support for 100M & 1G: Specifies whether 100M and 1G PCS/MAC is included in the FDK. This will also introduce an additional register stage for the 10G PCS/MAC TX path in order to achieve timing closure.

Please contact Exablaze support if you would like a build with any of these options changed.

The reference design is intended to be a simple example of how to get started with FPGA development for the ExaLINK Fusion. The reference design instantiates a number of FPGA transceivers and connects each one of them to an Exablaze low-latency PCS/MAC. https://neondating952.weebly.com/apple-logic-pro-x-10-1-1-download-free.html.

The top module described in fusion_devkit.v instantiates the fusion_devkit_netlistmodule which instantiates the transceivers. This top level module provides connectionsbetween the Fusion FDK netlist and the user's application. The top module also instantiates the DDR MIGs.

An overview of the FDK structure for ExaLINK Fusion is shown below.

Native loopback example

This reference design implements a native loopback, where the frames received from the RX datapath on a particular port is looped back out of the same port. When run at 10G rate, the Frame Check Sequence (FCS) or the CRC bytes in the received frame is removed before retransmitting the data through the same port. Also, 3 cycles of buffering is included in the loopback path to prevent TX underrun issues.

Native trigger example

The native trigger example application allows users to pre-load the Fusion with a pattern, mask and reply frame. The application performs a match on port 0 of any incoming frame against the pattern and mask, and if a match occurs the application will transmit the reply frame on port 1. This application can be used as a starting point for more advanced custom logic.

Note that this example application is only implemented on the FPGA for port 0 and port 1, and all other ports operate as normal network interfaces.

The FPGA transceivers are instantiated in the module xcku115_xcvr_32bit.The transceivers are connected to the low latency PCS/MAC IP from Exablaze, instantiated within the FDK netlist module. Raw PCS data is then transferred between the transceivers and the PCS/MAC.

The network-side interface allows the user application to send and receive packetson the network via the Exablaze low-latency MAC. Note that the interface signals rx_early_sof, tx_eof_no_crc, and tx_abort_frame are only present in the 10G PCS/MAC and not available for the 100M & 1G PCS/MAC.

The received data is provided via the following signals, all of which are inputs:

• rx_data (32 bits per port) - packet data as received from the wire. The first byte appears at byte 0 (bits 7 to 0).
• rx_sof (1 bit per port) - asserted on the same cycle as the first data word received from the wire. rx_data[7:0] will contain the first byte of the destination MAC address.
• rx_early_sof (1 bit per port) - a 'heads up' that the start of the preamble has been received and that rx_sof will be asserted in 2-3 cycles.
• rx_eof (1 bit per port) - asserted on the very last cycle of a received frame. The last bytes seen will include the four byte received CRC.
• rx_len (2 bits per port) - asserted on the same cycle as the EOF, indicates how many bytes in the final data signal are valid. As an example, if this reads 1, then only the bottom byte of data (bits 7 to 0) is valid. If it reads 0, then there are no more valid bytes in this cycle and the packet effectively finished in the previous cycle.
• rx_vld (1 bit per port) - asserted to indicate that receive data is valid. Due to the overhead of the 64b/66b encoding used in 10G Ethernet, there may be cycles intra-frame that do not contain valid data. This signal only applies to rx_data. You can assume that rx_sof, rx_early_sof and rx_eof are always valid.
• rx_err (1 bit per port) - asserted to indicate an abnormal frame termination condition. This can occur when the sender aborts the frame early, or if the link is lost in the middle of a frame. If a frame is terminated with rx_err, there will be no rx_eof for that frame.
• rx_crc_fail (1 bit per port) - asserted after EOF to indicate that the frame CRC check failed. This assertion normally occurs two cycles after EOF, but this depends on Exablaze build options.
• rx_timestamp (32 bit per port) - a counter that serves as the timestamp for the first byte of the received frame with 3.1ns of resolution.

All the above signals, except rx_timestamp, is synchronous to clk_rx. The rx_timestamp data is synchronous to clk_time.

A transmit interface is also exposed to the ExaLINK Fusion development kit application. The user application can monitor and modify frames that are being transmitted by the host, as well as transmit frames of its own. Ethernet frames transmitted by the user application must start with the first byte of the destination MAC address, and end at the last byte of the payload. Logic within Fusion automatically calculates, appends and transmits the CRC.The FPGA application has the following signals which connect through to the Ethernet transmission logic:

• tx_data (32 bits per port) - the packet data to be transmitted. The first byte that will be placed on the wire (e.g. the first byte of the destination MAC address) is located at bits 7 to 0.
• tx_sof (1 bit per port) - to be asserted on the same cycle as the first data word.
• tx_eof (1 bit per port) - to be asserted on the same cycle as the last data word.
• tx_len (2 bits per port) - to be set on the same cycle as EOF, indicating the number of bytes of data valid in the last cycle. Like rx_len, this may be 0 to indicate that the frame effectively ended in the previous cycle.
• tx_ack (1 bit per port) - acknowledge signal provided to the user application. The Fusion can be considered to have accepted transmit data from the application for any rising clock edge during which 'ack' is high. If 'ack' is low the same signals should continue to be presented.
• tx_err (1 bit per port), corrupt the CRC of the current frame. Note that asserting this signal only corrupts the running CRC calculation, it does not terminatethe frame. To terminate the current frame with an invalid CRC, asserttx_eof at least one acknowledged cycle after asserting tx_err.
• tx_eof_no_crc (1 bit per port), ends transmission of the current framebut does not append the frame checksum. Timing of this signal is the same as tx_eof.The result of setting both tx_eof and tx_eof_no_crc in the samecycle is undefined. If this signal is used, it is the responsibility of theuser's firmware to append the CRC to the data stream. (Only available for 32-bit native mode 10G operations.)
• tx_abort_frame (1 bit per port), aborts the current frame withoutsending an EOF symbol. Timing of this signal is the same as tx_eof. Ifthe current 64b/66b block is not full when this signal is asserted, theremainder of the block is filled with zeros. (Only available for 32-bit native mode 10G operations.)
• tx_ifg_compress (1 bit per port), Forces the transmitted interframe gapto the smallest value allowed by the deficit idle count algorithm, i.e. whenthere is no accumulated deficit. Must be set on eof and held constant untilthe sof of the next frame is acknowledged. In practice, we expect the userwill set it to a constant value if they determine that the downstream networkdevice can handle sustained short IFGs. Use with caution.

All the above signals are synchronous to clk_tx.

Note that tx_ack may drop out at any time. Also note that, during the frame, valid data must be presented on every cycle. There is no way to stall packet transmission. This is a property of Ethernet, not a limitation of our implementation.

For the 100M and 1G PCS/MAC, asserting the signals that are not supported bythe interface will not cause any harmful effects. Generally, this will also not result in any useful operations. However, it may be worth pointing out that asserting tx_eof_no_crc will have the same effect as that of tx_eof for 100M and 1G operations.

Note that the width of each of the above signals scales with the number of ports. To select theset of signals for a given port, use bit slicing. For example, RX data for port 0 will occupyrx_data[31:0], and RX data for port 1 will occupy rx_data[63:32]. The example designsshow how to perform this bit slicing or indexing for each of the signals on this bus. This noteapplies to all of the Ethernet frame interfaces in the FPGA development kit.

A Xilinx DDR4 memory controller is instantiated in fusion_devkit.v (refer mig_0_inst)however it's commented out to reduce build time for the majority of customers whomay not need this. Commented out line in compile.tcl will also need to be added backin if the DDR4 MIG is required.

Note that Fusions do not typically ship with a DDR4 DIMM installed. In order to fit within the height constraints of the Fusion chassis, Very Low Profile (VLP) DIMMs must be used, which have a maximum height of 18.8mm.

Please refer to the Xilinx Ultrascale Memory IP Product GuidePG150for more information, and contact Exablaze if you wish topurchase validated VLP DDR4 DIMMs.

Interact Contact Scratchpad 1 0 50

The management_if module implements a register interface that can be read andwritten through software. These registers can be accessed remotely via a serial server optionally enabled by the Fusion management processor.

Several other registers are provided, including status of the MIG calibration tests, port link up status, a synthesis timestamp of the design, and scratchpad registers. A list of available registers and their details are provided in the table below. Refer to management_if.v for further details.

Port rate is 2 bits for each port. The configuration will be as below.

The 1G autonegotiation control is one bit for each port. When set to 1, this will enable autonegotiation for the particular port. The 'autonegotiation complete' status for that port will be available on the corresponding bit in the status register.

Be default, the port rate for each port is set to 10G and the autonegotiation control bits are all set to 1.

Once the FPGA module has been configured for custom use, the XVC server isenabled as follows:

Warning

Note there is currentlyno authentication or other security provided on this connection (other thanthe Access Control List for the wholemanagement interface).

Note

Note that enabling the XVC serveris not stored as part of the Fusion's config, which means it's not listed underthe running-config and that enabling the XVC server must be done each time theFusion is rebooted. Use the show module command to see the current state.

After the XVC server on the Fusion has been enabled, open a Vivado Hardware Managersession as shown:

Opening the Hardware Manager

• Start a Hardware Server session with the following command in the Tcl Console:

  connect_hw_server

Ignore the error that may appear if you have no local targets.

• Connect to the Fusion with the following command in the Tcl Console.

  open_hw_target -xvc_url 192.168.220.14:2542

The -xvc_url switch tells Vivado the IP address and port to communicate with.

You should now see the xcku115 FPGA inside the Fusion listed, and it can now beoperated on as normal. For example, right click on SysMon and selectDashboard->New Dashboard to view temperature and voltage status:

SysMon dashboard showing Fusion FPGA status

Refer tohttp://www.xilinx.com/products/intellectual-property/xvc.htmlfor more information on Xilinx XVC.

Warning Macos mojave 10 14 4 combo update.

Exablaze makes use of bitstream encryption to protect the configuration imagesshipped with the Fusion. The xcku115 EFUSE registers are burnt with encryptionkey information. The user must NOT attempt to modify any EFUSE settings onthe primary Fusion FPGA or Exablaze firmware can no longer be loaded(unrecoverably) and warranty will be voided. Screens 4 6 8 inch. User bitfiles should be generatedwithout any encryption settings.

Configuring the FPGA

In order to configure the FPGA with your custom design, right click on xcku115_0and select Program Device. A window will appear where you can select the bitfileand click Program. After several seconds, the FPGA will be configured with yourbitfile. If your design includes cores such as ILA/Chipscope, MIG or IBERT,these will be displayed under the FPGA along with SysMon.

Once you have finished development of your custom FPGA image, you can transferthe bitfile to the Fusion for loading via the management processor, ie withoutneeding to maintain a Vivado session. Refer to theConfiguration section for further details.

An example design that instantiates 48 transceivers with IBERT cores is available fordownload here.

First ensure the FPGA module is configured for custom use:

On the Fusion CLI, create several loopbacks by patching internal module portstogether, for example:

Enable the XVC server and connect Vivado as described above, then configure theFPGA with the example IBERT bitfile.After the FPGA is configured, you will be asked if you want to auto-detect serialI/O links for the IBERT cores - select Yes.

Depending on how many patches you created above, you will see a number of links detected.An eye scan can be done on these links by right clicking one and selecting Create Scan,and clicking OK.

Eyescan of loopback link using Vivado and IBERT

Note these ports can be patched to frontpanel ports, enabling all 48 ports to run PRBS tests on external equipment.The PRBS pattern can be selected as appropriate, bit errors can be injected etc.

Connection

There is a communications channel between the FPGA and the Fusion managementprocessor. When running custom firmware, this channel can be exposed out of the10/100 Management Interface on the front panel. A server is setup on themanagement processor, and a client can connect via TCP. Any data sent/receivedthrough this connection is routed to the FPGA module, allowing users to interactwith their design through a non-high speed port for accessing status andconfiguration parameters.

To enable this connection the following command is used:

Any port above 1024 can be used for this connection.

Warning

Note there is currentlyno authentication or other security provided on this connection (other than theAccess Control List for the wholemanagement interface).

Framing Protocol

The firmware development kit example uses theHDLC framingprotocol to send messages over this interface. HDLC is a serial framingprotocol that uses comma and escape codes in order to delimit frameboundaries. The HDLC frame is byte oriented and is structured like so:

Since byte 0x7E is a framing flag, user code should never try to send a bytewith this value in any field apart from the SOF/EOF flag fields. Instead, usercode should first transmit an escape character, 0x7D, followed by the flag bytewith bit 5 flipped (0x5E). If the user code needs to transmit 0x7D, it mustfollow the same procedure and send 0x7D followed by 0x7D with bit 5 flipped(0x5D).

As an example, for the message payload:

Would be encoded:

Example registers

The example design provides an example of using the HDLC stream protocol toexpose a 32 bit memory mapped register interface. This registers interface sitsat address 0 in the HDLC address space, and allows for read/write messages. Thepayload structure for this example is as follows:

As an example, to write to register 1 the value 0x01020304, you would send aHDLC frame that looks like:

For both read and write requests, the FPGA will respond with a HDLC frame thathas the same format as a 'write request', however for register read requests,the register value field will contain the value as read from the register

Overview

Exablaze provides reference designs to get started with FPGA development using the ExaLINK Fusion. These examples also include our ultra low latency PCS/MAC. Depending on the configuration, some or all of the 48 transceiversconnected to the crosspoint are instantiated. Users are given the freedom to choose the example application to build from.

The following examples (referred to as targets) come with our development kit, including the requisite source code for each.

• A 'native loopback example' implements a loop-back application, where the received packets on a particular port is looped back out of the same port
• A 'native trigger example' shows how to pre-load ExaLINK Fusion with a replyahead of time, and send it based on a simple mask/pattern match over received frames

A DDR4 memory controller is also included in the design, however it is commented out by default to reduce build time.

Licensing

The ExaLINK Fusion development kit can ship as a time limited evaluationversion, which is fully functional for up to two hours. After this, features ofthe evaluation version will progressively shut down and stop working. Afterthis time the host can be rebooted/power cycled to reset the two hourtimer. Windows plugins on mac. Should you wish to purchase a full license and have this time limitationremoved, contact your reseller or our sales team todiscuss licensing options.

Exablaze provides a license file which Xilinx Vivado will require in order to synthesize the 'full' version of the development kit. Note that this is not required or provided for 'evaluation' FDKs.

This file, exablaze_fdk.lic, is included at the root of the tarball you will have received. This can be installed either locally or on a license server.

For local installation, the license file can be placed in your ~/.Xilinx/ directory. You may also need to update your ~/.flexlmrc file to ensure that it has the ~/.Xilinx/ directory in the license search path, it should contain a line similar to:

For installation on a license server, the license file should be added as a parameter when starting the FlexLM License Server, for example:

If Vivado is unable to locate the license, an error will be generated similar to this:

Please refer to the Xilinx documentation and this FAQ for further details.

The Fusion FDK can also ship with a 100M & 1G PCS/MAC along with the alreadyincluded 10G PCS/MAC. All the supplied example applications will work for the100M & 1G PCS/MAC without any modifications, as long as all the associated portsare configured with equal line rates. The examples may not work as intendedwhen a different line rate is used on downstream and upstream ports as there isno buffering added. Note that the inclusion of 100M & 1G PCS/MAC inthe FDK will introduce an additional pipeline stage for 10G in order to meet thetiming requirements. This will increase the overall latency of the 10G TX pathby one clock cycle.

Due to this extra clock cycle of latency, we do not include the 100M or 1GPCS/MAC by default. If 100M and 1G support is required, customers will have tocontact Exablaze and ask for a special build that will also include the 100M &1G PCS/MAC. Also note that the default configuration of the FDK will only enable8 ports of Fusion device. Should you require more ports to be enabled, pleasespecify that while contacting Exablaze.

Software Installation

Prior to using the development kit, customers must install Xilinx Vivado 2015.4 or later, which can be downloaded from the Xilinx website. A synthesis license is required for the FPGA device used in the Fusion (XCKU115-2FFVD1924C). A free 30 day 'webpack' license is available from Xilinx.

Warning

Exablaze has observed instances where incorrect logic is synthesized using Vivado 2016.1 and 2016.2 so use of these versions is not recommended.

Build System

The ExaLINK Fusion development kit ships with a build system for various fully functional target example applications. The build system consists of a Makefile and a Vivado TCL script (compile.tcl). The Makefile launches Vivado and instructs it to run the TCL script. All project source files and the Vivado build process are defined within the TCL script.

The Vivado environment must first be sourced by running the following (change path to suit):

The Makefile expects a TARGET to be provided. The default targets (each of which are contained within their own directory under the src/ directory) are:

• native_loopback_example
• native_trigger_example

The development kit is delivered as a tarball. Once this is extracted, the reference design can be built using following command:

Users can also create their own targets in a new directory under src/. The build system will generate a number of files in the outputs/ directory, including a standard firmware image with a .bit extension that be can be transferredto the ExaLINK Fusion.

Exablaze build options

There are several build options that are available for the FDK that Exablazeneeds to set at the FDK build time, rather than at the customer's build time.The file buildlog contains information as to what the build options were setto at the time the particular FDK was generated by Exablaze. The build optionsare:

• FDK Type: Specifies whether this FDK is a Full or Demo.
• Number of Ports: Specified the number of ports that are enabled in Fusion. The minimum value is 8 and the maximum is 48.
• Extra RX Reg: Specifies whether an additional register stage is added to the internal MAC RX path. This does not impact the user interface/timing diagrams. If this additional register is not included, the MAC latency will improve by 3.1ns, however it will make timing closure more difficult.
• Extra CRC Reg: Specifies whether an additional register stage is added to CRC RX path. This improves timing closure, but delays the assertion of the crc_fail flag. If this register is added, crc_flag is valid 2 cycles after eof, otherwise it's valid on the cycle after eof.
• Support for 100M & 1G: Specifies whether 100M and 1G PCS/MAC is included in the FDK. This will also introduce an additional register stage for the 10G PCS/MAC TX path in order to achieve timing closure.

Please contact Exablaze support if you would like a build with any of these options changed.

The reference design is intended to be a simple example of how to get started with FPGA development for the ExaLINK Fusion. The reference design instantiates a number of FPGA transceivers and connects each one of them to an Exablaze low-latency PCS/MAC. https://neondating952.weebly.com/apple-logic-pro-x-10-1-1-download-free.html.

The top module described in fusion_devkit.v instantiates the fusion_devkit_netlistmodule which instantiates the transceivers. This top level module provides connectionsbetween the Fusion FDK netlist and the user's application. The top module also instantiates the DDR MIGs.

An overview of the FDK structure for ExaLINK Fusion is shown below.

Native loopback example

This reference design implements a native loopback, where the frames received from the RX datapath on a particular port is looped back out of the same port. When run at 10G rate, the Frame Check Sequence (FCS) or the CRC bytes in the received frame is removed before retransmitting the data through the same port. Also, 3 cycles of buffering is included in the loopback path to prevent TX underrun issues.

Native trigger example

The native trigger example application allows users to pre-load the Fusion with a pattern, mask and reply frame. The application performs a match on port 0 of any incoming frame against the pattern and mask, and if a match occurs the application will transmit the reply frame on port 1. This application can be used as a starting point for more advanced custom logic.

Note that this example application is only implemented on the FPGA for port 0 and port 1, and all other ports operate as normal network interfaces.

The FPGA transceivers are instantiated in the module xcku115_xcvr_32bit.The transceivers are connected to the low latency PCS/MAC IP from Exablaze, instantiated within the FDK netlist module. Raw PCS data is then transferred between the transceivers and the PCS/MAC.

The network-side interface allows the user application to send and receive packetson the network via the Exablaze low-latency MAC. Note that the interface signals rx_early_sof, tx_eof_no_crc, and tx_abort_frame are only present in the 10G PCS/MAC and not available for the 100M & 1G PCS/MAC.

The received data is provided via the following signals, all of which are inputs:

• rx_data (32 bits per port) - packet data as received from the wire. The first byte appears at byte 0 (bits 7 to 0).
• rx_sof (1 bit per port) - asserted on the same cycle as the first data word received from the wire. rx_data[7:0] will contain the first byte of the destination MAC address.
• rx_early_sof (1 bit per port) - a 'heads up' that the start of the preamble has been received and that rx_sof will be asserted in 2-3 cycles.
• rx_eof (1 bit per port) - asserted on the very last cycle of a received frame. The last bytes seen will include the four byte received CRC.
• rx_len (2 bits per port) - asserted on the same cycle as the EOF, indicates how many bytes in the final data signal are valid. As an example, if this reads 1, then only the bottom byte of data (bits 7 to 0) is valid. If it reads 0, then there are no more valid bytes in this cycle and the packet effectively finished in the previous cycle.
• rx_vld (1 bit per port) - asserted to indicate that receive data is valid. Due to the overhead of the 64b/66b encoding used in 10G Ethernet, there may be cycles intra-frame that do not contain valid data. This signal only applies to rx_data. You can assume that rx_sof, rx_early_sof and rx_eof are always valid.
• rx_err (1 bit per port) - asserted to indicate an abnormal frame termination condition. This can occur when the sender aborts the frame early, or if the link is lost in the middle of a frame. If a frame is terminated with rx_err, there will be no rx_eof for that frame.
• rx_crc_fail (1 bit per port) - asserted after EOF to indicate that the frame CRC check failed. This assertion normally occurs two cycles after EOF, but this depends on Exablaze build options.
• rx_timestamp (32 bit per port) - a counter that serves as the timestamp for the first byte of the received frame with 3.1ns of resolution.

All the above signals, except rx_timestamp, is synchronous to clk_rx. The rx_timestamp data is synchronous to clk_time.

A transmit interface is also exposed to the ExaLINK Fusion development kit application. The user application can monitor and modify frames that are being transmitted by the host, as well as transmit frames of its own. Ethernet frames transmitted by the user application must start with the first byte of the destination MAC address, and end at the last byte of the payload. Logic within Fusion automatically calculates, appends and transmits the CRC.The FPGA application has the following signals which connect through to the Ethernet transmission logic:

• tx_data (32 bits per port) - the packet data to be transmitted. The first byte that will be placed on the wire (e.g. the first byte of the destination MAC address) is located at bits 7 to 0.
• tx_sof (1 bit per port) - to be asserted on the same cycle as the first data word.
• tx_eof (1 bit per port) - to be asserted on the same cycle as the last data word.
• tx_len (2 bits per port) - to be set on the same cycle as EOF, indicating the number of bytes of data valid in the last cycle. Like rx_len, this may be 0 to indicate that the frame effectively ended in the previous cycle.
• tx_ack (1 bit per port) - acknowledge signal provided to the user application. The Fusion can be considered to have accepted transmit data from the application for any rising clock edge during which 'ack' is high. If 'ack' is low the same signals should continue to be presented.
• tx_err (1 bit per port), corrupt the CRC of the current frame. Note that asserting this signal only corrupts the running CRC calculation, it does not terminatethe frame. To terminate the current frame with an invalid CRC, asserttx_eof at least one acknowledged cycle after asserting tx_err.
• tx_eof_no_crc (1 bit per port), ends transmission of the current framebut does not append the frame checksum. Timing of this signal is the same as tx_eof.The result of setting both tx_eof and tx_eof_no_crc in the samecycle is undefined. If this signal is used, it is the responsibility of theuser's firmware to append the CRC to the data stream. (Only available for 32-bit native mode 10G operations.)
• tx_abort_frame (1 bit per port), aborts the current frame withoutsending an EOF symbol. Timing of this signal is the same as tx_eof. Ifthe current 64b/66b block is not full when this signal is asserted, theremainder of the block is filled with zeros. (Only available for 32-bit native mode 10G operations.)
• tx_ifg_compress (1 bit per port), Forces the transmitted interframe gapto the smallest value allowed by the deficit idle count algorithm, i.e. whenthere is no accumulated deficit. Must be set on eof and held constant untilthe sof of the next frame is acknowledged. In practice, we expect the userwill set it to a constant value if they determine that the downstream networkdevice can handle sustained short IFGs. Use with caution.

All the above signals are synchronous to clk_tx.

Note that tx_ack may drop out at any time. Also note that, during the frame, valid data must be presented on every cycle. There is no way to stall packet transmission. This is a property of Ethernet, not a limitation of our implementation.

For the 100M and 1G PCS/MAC, asserting the signals that are not supported bythe interface will not cause any harmful effects. Generally, this will also not result in any useful operations. However, it may be worth pointing out that asserting tx_eof_no_crc will have the same effect as that of tx_eof for 100M and 1G operations.

Note that the width of each of the above signals scales with the number of ports. To select theset of signals for a given port, use bit slicing. For example, RX data for port 0 will occupyrx_data[31:0], and RX data for port 1 will occupy rx_data[63:32]. The example designsshow how to perform this bit slicing or indexing for each of the signals on this bus. This noteapplies to all of the Ethernet frame interfaces in the FPGA development kit.

A Xilinx DDR4 memory controller is instantiated in fusion_devkit.v (refer mig_0_inst)however it's commented out to reduce build time for the majority of customers whomay not need this. Commented out line in compile.tcl will also need to be added backin if the DDR4 MIG is required.

Note that Fusions do not typically ship with a DDR4 DIMM installed. In order to fit within the height constraints of the Fusion chassis, Very Low Profile (VLP) DIMMs must be used, which have a maximum height of 18.8mm.

Please refer to the Xilinx Ultrascale Memory IP Product GuidePG150for more information, and contact Exablaze if you wish topurchase validated VLP DDR4 DIMMs.

Interact Contact Scratchpad 1 0 50

The management_if module implements a register interface that can be read andwritten through software. These registers can be accessed remotely via a serial server optionally enabled by the Fusion management processor.

Several other registers are provided, including status of the MIG calibration tests, port link up status, a synthesis timestamp of the design, and scratchpad registers. A list of available registers and their details are provided in the table below. Refer to management_if.v for further details.

Port rate is 2 bits for each port. The configuration will be as below.

The 1G autonegotiation control is one bit for each port. When set to 1, this will enable autonegotiation for the particular port. The 'autonegotiation complete' status for that port will be available on the corresponding bit in the status register.

Be default, the port rate for each port is set to 10G and the autonegotiation control bits are all set to 1.

In order to run custom firmware, the FPGA module must first be set to functioncustom:

After building the reference design, you will end up with a bitfile in outputs/ which we'll need to copy across to the Fusion:

Note: For the native trigger example, the bitfile will be fusion_native_trigger_example.bit.

Once the bitfile is transferred, the Fusion is instructed to associate it withthe particular FPGA module, and then start configuration using the reconfig command:

Note: As an alternative to copying the bitfile to the Fusion and having the management processor load it onto the FPGA, a remote connection between Vivado on your workstation and the Fusion can be established. You can then use Vivado to configure the FPGA with your bitstream, debug using ILA etc. Refer to Debugging with Vivado for more information.

In order to access the register interface in the reference design remotely, the serial server on the Fusion must first be configured:

Several python scripts are included that can read and write to registers from a remote host to the the FPGA running the reference design via the Fusion's management processor. These are located in the sw/ directory.

Note that you can copy these scripts over to a Fusion, and after the serial serverhas been enabled, drop to a bash shell and run the scripts directly on theFusion itself.

To read/write to the scratch register (address 0), one would do the following:

Interact Contact Scratchpad 1 0 56

The time/date of synthesis can be read as follows from register 13:

In order to find the date this version of the FDK was originally built by Exablaze, you can read register 12:

Now let's connect the front panel port A1 to the FPGA transceiver Y3:

Then, send packets from a remote host into A1. As the transceiver ports are all looped back in the reference design, you will see these same packets received back on the host.

Finally, let's try configuring 1G. First, set the line card port speed to 1G as below.

Since A1 is connected to Y3, we need to set the bits [5:4] in register 4to 0b10 (note that these bits correspond to transceiver port 2).This can be achieved with the help of the python script:

This will configure the rate for transceiver port Y3 to 1G. Now you will be able tosend and receive packets on port A1 at 1G rate.

The ExaLINK Fusion development kit is provided with a full functional model for all of the individual interfaces. This can be found in the tb/ directory of the package. The testbench consists of the following files:

• test_bench.v, the top level harness that wraps the various modules contained in the functional simulation.
• control.v, contains various control tasks and generates the timestamp counter.
• transmit_sim_32bit.v, simulates the Fusion Ethernet transmit interface. Will log frames that have been transferred successful and indicate error conditions.
• frame_sim_32bit.v, simulates the frames received from the wire.
• test_cases.v, container for user test cases. Users can add their own simulation directives here.
• start_sim.sh, a shell script that starts the Xilinx simulator in console mode. To start in graphical mode, use the switch -gui.
• test_bench.prj, a project file for the Xilinx simulator that lists all files that make up the simulation. This file is generated automatically by start_sim.sh when a simulation is run.

The example in test_cases.v shows how users can exercise the various elements of the functional mode, and provides a test case for the example design. Users can add their own test cases to this file as necessary.

There are two example applications that can be simulated by the users viz.native_loopback_example and native_trigger_example. Users can start the example testbench by running:

Mailraider pro 3 6975. This will cause the testbench to be compiled and xsim to start in command line mode. From the xsim prompt, the simulation can be run for 1 microsecond by entering:

For the graphical mode simulation, run:

v2.2.0, 19-Mar-2019

• Support Vivado 2018.3
• Add dual DDR MIG (commented out by default)
• Add Vivado licensing requirement for 'full' FDKs
• Workaround Vivado issue which caused multi-bit signals to be split up in chipscope

v2.1.0, 5-Dec-2018

• Support Vivado 2018.2
• Add support for 32bit 1G PCS/MAC on all ports
• Add support for 100M Ethernet
• 10G PCS: Add the tx_eof_no_crc_net signal to end a frame without appending the frame checksum
• 10G PCS: Add the tx_abort_frame_net signal to abort the current frame without sending an EOF symbol on the wire
• 10G PCS: Improve SOF selection algorithm in MAC to minimise interpacket gap
• Add extra pipeline stage for 10G to meet timing (only when 1G logic is added)
• Make 1G feature selectable during build time (at Exablaze)

v2.0.1

• Mismatch in tx_err_net signal length fixed

Interact Contact Scratchpad 1 0 58

v2.0.0, 18-Jul-2018

• Upgrade PCS/MAC to 32b width with reduced latency, 10G only currently
• Add DDR/QDR MIGs (commented out by default)
• Selectable number of transceivers to add to design (at Exablaze build time)
• Add a second example design - trigger
• Add support for synthesis with Vivado up to 2018.2
• Add test bench for example applications

Interact Contact Scratchpad 1 0 5 0

This page was last updated on Jun-23-2020.