Skip to content

Latest commit

 

History

History
1085 lines (941 loc) · 59.6 KB

isp1040.md

File metadata and controls

1085 lines (941 loc) · 59.6 KB

QLogic ISP1040 (QL1040B) SCSI controller on the Octane

The Octane's two "Integral SCSI controller" ports are QLogic ISP1040B chips (SGI calls the part QL1040B) sitting on the IP30 board's Bridge at PCI slots 0 and 1. To boot IRIX on an emulated Octane, iris has to present a chip that the PROM's and IRIX's ql drivers accept: PCI configuration space, a 256-byte register block, the mailbox protocol of the chip's firmware, and DMA'd request/response queues in host memory. This document is the register map, the firmware interface, and a description of how each OS we have source for (and IRIX, which we don't) actually drives it.

Nothing executes on the real chip's RISC processor in an emulator. What the drivers see is the firmware's behaviour, so this document describes the chip as the firmware makes it look, not the internal RISC.

Sources

Everything here was derived from these, read side by side:

Source Licence What it gave
OpenBSD sys/dev/ic/qlw.c, qlwreg.h, sys/dev/pci/qlw_pci.c (d82e6535c6, the last OpenBSD/sgi tree) ISC The simplest complete ISP1040 driver, and the one that ran on real Octanes (#ifdef __sgi__: initiator ID 0).
FreeBSD stable/12 sys/dev/isp/isp.c, ispreg.h, ispmbox.h, ispvar.h (Matthew Jacob) BSD-2 The full register map (BIU/DMA/SXP/RISC blocks), the mailbox in/out register tables, every completion code. FreeBSD 13 dropped parallel SCSI, so use stable/12.
NetBSD sys/dev/ic/isp*.c, sys/dev/pci/isp_pci.c, sys/dev/microcode/isp/asm_1040.h BSD Same driver family; PCI attach; the 4.66.00 initiator firmware image.
Linux drivers/scsi/qla1280.c, qla1280.h; linux-firmware qlogic/1040.bin GPL-2 QLogic's own driver lineage: reset dance, 64-bit (A64) IOCBs, firmware 7.65.
IRIX 6.5.30 /usr/include/sys/ql.h (shipped header) — SGI's driver structures: the register struct as IRIX sees it, IOCB layouts, mailbox command/status enums, IP30 compile options.
IRIX 6.5.30 /var/sysgen/boot/ql.o (shipped kernel module, with symbols), unix, PROM 4.17 image — Disassembled for interoperability: the exact register sequence IRIX performs, which mailbox commands it issues and in what order, the firmware images it carries. Strings only from the PROM.
QLogic ISP1040B / ISP1040C data sheets (bitsavers pdf/qlogic/) — Block diagram, FIFO sizes, clock. No register-level manual for the 1040 has surfaced; the data sheets are marketing-level.
SGI Bridge ASIC spec rev 4.0 (docs/datasheets/octane/), IRIX sys/PCI/{bridge,pciio,pcibr}.h, OpenBSD arch/sgi/xbow/xbridge*.{c,h}, Linux arch/mips/pci/pci-xtalk-bridge.c — How the Octane's Bridge presents the chip: byte swapping, 64-bit DMA attributes, devio windows.

No leaked IRIX source was used. IRIX behaviour comes from its shipped header and from disassembling the shipped binaries.

1. The chip

The ISP1040 is a PCI SCSI I/O processor: a 16-bit RISC core running firmware out of external SRAM, a SCSI sequencer (the "SXP") and a dual-channel DMA engine (command DMA for IOCBs/status, data DMA for SCSI data) behind a PCI bus interface unit (BIU). The host talks to the firmware through eight 16-bit mailbox registers and through request and response queues of 64-byte I/O control blocks (IOCBs) in host memory.

Variants

Name PCI device ID PCI revision ID CONF0 bits 3:0 Clock Notes
ISP1020 0x1020 0 1 40 MHz Fast SCSI
ISP1020A 0x1020 1 2 40 MHz some Ultra-capable (RISC PSR bit 7)
ISP1040 0x1020 2 3 60 MHz Ultra (Fast-20)
ISP1040A 0x1020 3 4 60 MHz "busted FIFO": drivers leave the FIFO threshold at 0
ISP1040B 0x1020 4 5 60 MHz
ISP1040B rev 2 / "1040C" 0x1020 5 6 60 MHz the Octane's chip
ISP1240 0x1240 60 MHz dual bus, 1080-style register banks
ISP1080 / 1280 / 12160 0x1080 / 0x1280 / 0x1216 100 MHz Ultra2/Ultra3, different DMA/SXP banking

The PCI revision table is OpenBSD's and IRIX's (REV_ISP1040BV2 = 5, printed by hinv as "QL1040B (rev. 2)"); the CONF0 column is what FreeBSD and Linux decode from the hardware revision field, which runs one ahead of the PCI revision ID. The real Octane chip's CONF0 value has not been read back; 6 is what the tables imply for PCI revision 5.

PCI vendor ID is 0x1077. Only the 0x1020 family matters for the Octane; the 1080/1240/12160 differences are noted where a driver code path branches on them so the reader knows what to ignore.

What the Octane has

From hinv -v on an Octane running IRIX 6.5.30:

Integral SCSI controller 0: Version QL1040B (rev. 2), single ended
Integral SCSI controller 1: Version QL1040B (rev. 2), single ended
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 0
  PCI Adapter ID (vendor 0x1077, device 0x1020) PCI slot 1

Both chips are devices 0 and 1 on the Bridge at xtalk widget 15 (the IP30 board), beside the IOC3 (device 2) and RAD audio (device 3). Controller 0 is the internal bus holding the system disks; controller 1 is the external connector. They are single-ended, and the host adapter uses SCSI ID 0 (SGI convention; INITIATOR_SCSI_ID 0 in ql.h). The onboard chips have no NVRAM: IRIX never touches the NVRAM register, and OpenBSD's driver falls back to defaults when the NVRAM reads back invalid.

2. PCI configuration space

Offset Register Value / use
0x00 Vendor / Device ID 0x1077 / 0x1020
0x04 Command / Status drivers set I/O space, memory space, bus master, MWI, parity and SERR enables
0x08 Revision ID, class revision 5 on the Octane; class 0x01 0x00 (SCSI storage)
0x0C Cache line size, latency timer IRIX writes 0x80 / 0x40 (pciio_config_set of 1 byte at 12 and 13); NetBSD and OpenBSD set 0x10 / 0x80
0x10 BAR0: I/O space 256 bytes of registers. IRIX uses this one (USE_IOSPACE for IP30, PCIIO_SPACE_WIN(0)).
0x14 BAR1: memory space the same 256 registers; Linux, NetBSD prefer it
0x2C Subsystem vendor / ID supported by the 1040B
0x30 Expansion ROM BAR BIOS ROM on PC adapters; drivers disable it (clear bit 0)
0x3C / 0x3D Interrupt line / pin pin A
0x40 "ISP_Config_1" vendor-specific; named in IRIX's PCI_REG struct, not written by any driver read here

IRIX reads vendor/device (2 bytes at 0), the revision byte (at 8), reads and writes the command register (2 bytes at 4, OR-ing in bit 0), writes the latency timer and cache line size, then allocates Bridge read response buffers for the slot (pcibr_rrb_alloc; the PROM prints "Cannot allocate %d RRBs for ql %d" when that fails).

3. Register map

All registers are 16 bits wide at even offsets from the BAR. Offsets are PCI offsets; see §8 for how they appear to a big-endian CPU through the Bridge. Unless stated, bit meanings come from FreeBSD's ispreg.h and Linux's qla1280.h, which agree.

3.1 Bus interface unit, 0x00–0x1F

Offset Name Access Bits
0x00 ID_LO R bus ID low
0x02 ID_HI R bus ID high
0x04 CONF0 R bits 3:0 hardware revision (table above)
0x06 CONF1 RW bit 2 BURST_ENABLE; bit 3 SXP register select (maps the SXP registers over 0x80–0xFF); bits 6:4 PCI FIFO threshold: 0x10 = 16 B, 0x20 = 32 B, 0x30 = 64 B, 0x40 = 128 B; bits 1:0 the old SBus FIFO threshold; bits 9:8 (1080 family only) DMA/SXP bank select (0x300)
0x08 ICR RW bit 0 SOFT_RESET; bit 1 ENABLE_ALL_INTS; bit 2 ENABLE_RISC_INT; bit 3 ENABLE_SXP_INT; bit 4 ENABLE_CDMA_INT; bit 5 ENABLE_DMA_INT; bit 8 flash enable, bit 9 flash upper bank (Linux)
0x0A ISR R bit 1 interrupt pending (global); bit 2 RISC_INT; bit 3 SXP int; bit 4 CDMA int; bit 5 DMA int; bit 14 "PCI 64-bit slot" (Linux)
0x0C SEMA RW bit 0 LOCK: the outgoing mailboxes hold a mailbox completion or async event; bit 1 status
0x0E NVRAM RW bit 0 clock, bit 1 chip select, bit 2 data out (host → EEPROM), bit 3 data in; bit 7 busy on later chips
0x10 FLASH_BIOS_DATA RW (OpenBSD/Linux)
0x12 FLASH_BIOS_ADDR RW

Writes of 0 to ISR (IRIX does this in its parity routine) are harmless.

3.2 DMA channels, 0x20–0x63 (1040 layout)

The 1080/1240/12160 move these to a bank at 0x80 selected by CONF1 bits 9:8; on the 1040 they are fixed here. "RW*" means the RISC must be paused.

Offset Name Bits
0x20 CDMA_CONF (RW*) bit 0 direction (1 = host → FIFO), bit 1 BURST_ENABLE, bit 2 enable interrupts to RISC, bit 3 enable SXP → DMA
0x22 CDMA_CONTROL (RW*) bit 0 strobe (start), bit 1 reset interrupt, bit 2 clear FIFO, bit 3 clear channel (and reset DMA registers), bit 4 suspend
0x24 CDMA_STATUS (R) 0x8000 interrupt, 0x4000 retry status, 0x3000 channel status (0x1000 active to host, 0x2000 suspended, 0x3000 transfer in progress), 0x0100 FIFO overrun, 0x0080 FIFO underrun, 0x0040 bus error, 0x0020 bus parity, 0x0010 clear pending, 0x0008 terminal count, 0x0004 suspended, 0x0003 pipeline state
0x26 CDMA_FIFO_STS (R) 0x200 overrun, 0x100 underrun, 0xFF byte count
0x28 CDMA_COUNT (RW*) transfer count
0x2C–0x32 CDMA_ADDR0..3 (RW*) DMA address words 0..3
0x40 DDMA_CONF (RW*) as CDMA_CONF
0x42 DDMA_CONTROL (RW*) as CDMA_CONTROL
0x44 DDMA_STATUS (R) as CDMA_STATUS
0x46 DDMA_FIFO_STS (R)
0x48 / 0x4A DDMA_COUNT_LO / HI
0x4C / 0x4E DDMA_ADDR0 / 1
0x50 / 0x52 DDMA_ADDR2 / 3 1040A and later
0x60 / 0x62 DFIFO_COMMAND / DFIFO_DATA FIFO access ports

Drivers only touch these to clear the channels after reset (CONTROL = CLEAR_CHAN | RESET_INT, IRIX writes 0x0E = clear channel + clear FIFO + reset int) and to set the burst-enable bits. The firmware programs them internally; an emulator need not model the DMA engine itself, only return sane values.

3.3 Mailboxes, 0x70–0x7E

Offset Register
0x70 MAILBOX0 — command (in) / status (out)
0x72 MAILBOX1
0x74 MAILBOX2
0x76 MAILBOX3
0x78 MAILBOX4 — doubles as the request queue in-pointer (host writes) and out-pointer (firmware writes)
0x7A MAILBOX5 — doubles as the response queue in-pointer (firmware writes) and out-pointer (host writes)
0x7C MAILBOX6
0x7E MAILBOX7

Each mailbox is really two registers: a write lands in the incoming mailbox the RISC reads, a read returns the outgoing mailbox the RISC wrote. Writing MAILBOX4 does not disturb what the host reads back from MAILBOX4. This matters for the queue pointers (§4.4).

3.4 RISC processor registers, 0x80–0xBF (CONF1 bit 3 = 0)

Offset Name Notes
0x80 ACC accumulator
0x82–0x9E R1–R15 general registers
0xA0 PSR 0x8000 force true, 0x4000 loop count done, 0x2000 RISC int, 0x1000 timer rollover, 0x0800 ALU overflow, 0x0400 ALU MSB, 0x0200 carry, 0x0100 zero, 0x0080 PCI Ultra capable, 0x0020 SBus Ultra, 0x0010 DMA int, 0x0008 SXP int, 0x0004 host int, 0x0002 int pending, 0x0001 force false
0xA2 IVR interrupt vector
0xA4 PCR processor control
0xA6 / 0xA8 RAR0 / RAR1 RAM address registers
0xAA LCR loop counter
0xAC PC program counter (R)
0xAE MTR memory timing; FreeBSD writes 0x1212, or 0x1313 for "fast memory timing"
0xB0 EMB external memory boundary (1040); 0x8 = dual bank (1080)
0xB2 SP stack pointer
0xBE HRL hardware revision level (R)
0xC0 HCCR host command and control, below
0xC2 / 0xC4 BP0 / BP1 breakpoints
0xC6 / 0xC8 TCR / TMR test control / test mode (W)
0xCC / 0xCE GPIO data / enable 1080-class parts (termination control). ql.h documents GPIO use for SGI's XIO "MSCSI" boards, not IP30.
0xF2 SCSI control pins (Linux scsiControlPins)

All RISC registers except HCCR and the breakpoints are only accessible while the RISC is paused (HCCR PAUSE, HCCR read bit 0x20 set).

HCCR (0xC0) — a write is a command in bits 15:12, a read is status:

Write Command
0x0000 NOP
0x1000 reset RISC
0x2000 pause RISC
0x3000 release RISC (from pause or reset)
0x4000 single step
0x5000 set host interrupt (tells the RISC a mailbox command is waiting)
0x6000 clear host interrupt
0x7000 clear RISC interrupt (acknowledge; the host does this after every interrupt)
0x8000 change breakpoint enables
0x9000 | bit 0 write BIOS enable (0x9000 = disable BIOS so the RISC runs from RAM)
0xA000 | bits 8,9 write SRAM parity enable, bit 8 = bank 0, bit 9 = bank 1 (IRIX writes 0xA100)
0xE000 generate a parity error
0xF000 test mode
Read bit Meaning
0x0080 host interrupt set (a mailbox command is pending)
0x0040 RISC reset in progress
0x0020 RISC paused
0x0100 / 0x0200 parity enabled on bank 0 / 1
0x0400 parity error detected (1040; IRIX HCCR_SRAM_PARITY_ERROR_DETECTED_1040)

3.5 SXP (SCSI executive processor) registers, 0x80–0xFF (CONF1 bit 3 = 1)

Set CONF1 bit 3 while the RISC is paused and the SXP bank replaces the RISC bank. Only two are read by hosts, but the full map (FreeBSD) is:

Offset Name Offset Name
0x80 PART_ID 0xC0 SCSI_ID (0x0F00 (re)selecting id, 0x000F select id)
0x82 CONFIG1 (0xF000 async setup, 0x0600 selection timeout, 0x00E0 clock factor, 0x000F SCSI id) 0xC2 DEV_CONFIG1 (sync hold/setup/offset)
0x84 CONFIG2 (0x40 disable filter, 0x20 REQ/ACK pullups, 0x10 data pullups, 0x08 config autoload, 0x02 reselect, 0x01 select) 0xC4 DEV_CONFIG2 (0x400 wide, 0x200 parity, 0x100 block mode)
0x86 CONFIG3 0xC8 PHASE_PTR
0x8C INSTRUCTION 0xCC BUF_PTR
0x90 RETURN_ADDR 0xD0 BUF_CTR
0x94 COMMAND (0x300B = reset SCSI bus) 0xD2–0xD8 BUFFER, BUF_BYTE, BUF_WD, BUF_WD_TRAN
0x98 INTERRUPT (0x8000 parity err, 0x4000 gross err, 0x2000 function abort, 0x1000 condition failed, 0x0800 FIFO empty, 0x0400 buf counter zero, 0x0200 xfer zero, 0x0080 int pending, 0x0040 cmd running, 0x000F return code) 0xDA–0xE0 FIFO, FIFO_STATUS, FIFO_TOP, FIFO_BOTTOM
0x9C SEQUENCE 0xE4 TRAN_REG
0x9E GROSS_ERR 0xE8–0xEE TRAN_CNT_LO/HI, TRAN_CTR_LO/HI
0xA0 EXCEPTION enables 0xF0 ARB_DATA
0xA4 OVERRIDE (0x0008 FIFO reset, 0x0004 terminate, 0x0002 reset regs, 0x0001 reset module, …) 0xF2 PINS_CTRL (0x8000 phase valid, 0x100 REQ, 0x80 ACK, 0x40 RST, 0x20 BSY, 0x10 SEL, 0x08 ATN, 0x04 MSG, 0x02 C/D, 0x01 I/O)
0xA8 LIT_BASE 0xF4 PINS_DATA
0xAC / 0xB0 / 0xB4 USER_FLAGS / USER_EXCEPT / BREAKPOINT 0xF6 PINS_DIFF: 0x0200 DIFFSENS, 0x0100 differential mode, 0x0080 enable output, 0x007C diff control pins, 0x0002 target, 0x0001 initiator. Ultra2 adds 0x1000 LVD, 0x0800 HVD, 0x0400 SE.

Hosts read PINS_DIFF to learn the bus type: FreeBSD prints "Differential Mode" if bit 8 is set; IRIX's ql_diffsense_sp pauses the RISC, sets CONF1 bit 3, reads 0xF6 and reports "SCSI Adapter %x in bus mode : %d" (1 = single-ended, 2 = HVD, 4 = LVD). For an Octane return bit 8 clear. FreeBSD also reads RISC_PSR (0xA0) bit 7 to decide the part is Ultra capable and runs the clock at 60 MHz.

4. Principles of operation

4.1 Reset

Every driver begins the same way (Linux qla1280_chip_diag, FreeBSD isp_reset, OpenBSD qlw_softreset, IRIX ql_download_fw / ql_reset_interface):

  1. Disable interrupts (ICR = 0), pause the RISC (HCCR 0x2000) if the driver wants to read SXP/RISC registers first.
  2. ICR = SOFT_RESET (bit 0). The bit clears itself when the chip is back (Linux waits up to 5 s polling it; FreeBSD too). The chip does not respond on PCI during the reset — Linux notes a read-back right after the write machine-checks on some hosts, so emulate the bit clearing after a short delay rather than instantly if you want to be faithful, but instantly works for every driver read here.
  3. Clear the DMA channels: CDMA_CONTROL and DDMA_CONTROL = CLEAR_CHAN | RESET_INT (FreeBSD) or 0x0E (IRIX, 1040 only).
  4. CONF1 = 0 (FreeBSD/OpenBSD), then HCCR 0x1000 (reset RISC), SEMA = 0, HCCR 0x3000 (release). Linux does it in one write: HCCR = RESET | RELEASE | DISABLE_BIOS = 0x1000|0x3000|0x9000 — the chip takes that as the individual commands.
  5. Wait for the ROM firmware's self test: MAILBOX0 reads 4 (MBOX_BUSY) while it runs, then 0 (FIRMWARE_ALIVE; 1 = checksum error, 2 = shadow load error). The other mailboxes then hold the product ID: MAILBOX1 = 0x4953 "IS", MAILBOX2 = 0x5020 "P " (0x0000 on some old parts), MAILBOX3 = 0x2020 " ", MAILBOX4 = 0x0001. Linux and OpenBSD check these; IRIX does not. IRIX polls HCCR bit 0x40 (reset in progress) clear instead, then waits while MAILBOX0 == 4.
  6. CONF1 = burst enable | FIFO threshold (FreeBSD/OpenBSD 0x34 = 64 B, IRIX and Linux 0x44 = 128 B), and the CDMA/DDMA_CONF burst bits. On a 1040A the threshold bits are left 0.
  7. HCCR 0x9000 (BIOS disable), enable interrupts (ICR = 6, bits 1 and 2), then a mailbox NOP (FreeBSD/OpenBSD) and a MAILBOX REGISTER TEST (everyone; see §4.2) to prove the ROM firmware answers.

After this the chip runs its boot ROM, which services the small set of mailbox commands needed to load firmware (0x00–0x08, 0x09/0x0A, 0x40–0x42). Anything else returns 0x4001 (invalid command).

The RAM contents survive ICR soft reset and HCCR RISC reset. IRIX relies on that: ql_download_fw loads the firmware once, then every ql_reset_interface (initial setup and every error recovery) resets the chip and issues EXECUTE FIRMWARE again without reloading. An emulator must keep the loaded image across resets.

4.2 The mailbox protocol

A mailbox command is at most eight 16-bit words. The host:

  1. writes the parameters to the incoming mailboxes (MAILBOX0 = command code),
  2. writes HCCR 0x5000 (set host interrupt). HCCR reads back with bit 0x80 until the RISC takes the command,
  3. waits for an interrupt (or polls): ISR bit 2 set and SEMA bit 0 set means the outgoing mailboxes hold a mailbox completion or an async event. MAILBOX0 ≥ 0x8000 is an async event (§4.6), otherwise it is the command's status,
  4. reads the outgoing mailboxes it cares about, writes SEMA = 0, and writes HCCR 0x7000 (clear RISC interrupt).

Only one mailbox command is outstanding at a time; while the semaphore is held the firmware will not post another completion. Response-queue interrupts (§4.4) arrive with SEMA bit 0 clear; that is how drivers tell the two apart. OpenBSD and NetBSD/FreeBSD share this logic (qlw_read_isr, isp_pci_rd_isr), Linux's qla1280_isr and IRIX's ql_service_interrupt do the same.

Status codes in MAILBOX0:

Code Meaning
0x4000 command complete
0x4001 invalid command
0x4002 host interface error
0x4003 test failed
0x4005 command error (MAILBOX1 = subcode)
0x4006 command parameter error
0x4007 bus configuration error (IRIX ql.h)
0x4010 checksum error (FC firmware; the 1040 ROM reports a bad checksum as a failed VERIFY CHECKSUM)

Drivers wait between 0.4 s (OpenBSD, polled during attach) and 20 s (Linux) for a completion; IRIX retries after each 1 s by writing HCCR 0x7000 and prints "Mailbox timeout: ... resetting ISR and retrying".

FreeBSD keeps a table of which mailboxes each command reads and writes; it is reproduced here (bit n = MAILBOXn), with the parameter meanings collected from all the drivers.

Code Name In Out Parameters
0x00 NO-OP 0x01 0x01
0x01 LOAD RAM 0x1F 0x01 1 = RISC address, 2/3 = host address 31:16 / 15:0, 4 = word count. DMA from host to RISC RAM.
0x02 EXECUTE FIRMWARE 0x03 0x01 1 = start address (0x1000)
0x03 DUMP RAM 0x1F 0x01 as LOAD RAM, RISC → host
0x04 WRITE RAM WORD 0x07 0x07 1 = address, 2 = data
0x05 READ RAM WORD 0x03 0x07 1 = address → 2 = data
0x06 MAILBOX REGISTER TEST 0x3F 0x3F echoes 1..5 (FreeBSD checks 1..5, Linux 1..7, IRIX writes 1..7 with 0x1111..0x7777, the PROM compares)
0x07 VERIFY CHECKSUM 0x07 0x07 1 = start address. The 16-bit sum of words [start, start + len) where len = word[start+3] must be 0. Returns the sum in 1 (OpenBSD prints it, the PROM prints "theirs %x, mine %x").
0x08 ABOUT FIRMWARE 0x01 0x0F → 1 = major, 2 = minor, 3 = subminor; 6 = attributes (0 on 1040)
0x09 LOAD RAM A64 (ROM) 0xDF 0x01 as LOAD RAM plus 6/7 = host address 63:48 / 47:32. Served by the boot ROM.
0x0A DUMP RAM A64 (ROM)
0x0E CHECK FIRMWARE 0x01 0x05
0x10 INIT REQUEST QUEUE 0x1F 0x1F 1 = entries, 2/3 = address 31:16 / 15:0, 4 = initial in-pointer (0) → 4 = firmware's out-pointer
0x11 INIT RESPONSE QUEUE 0x3F 0x3F 1 = entries, 2/3 = address, 5 = initial out-pointer (0) → 5 = firmware's in-pointer
0x12 EXECUTE IOCB 0x0F 0x0F 2/3 = address of a single IOCB
0x13 WAKE UP 0x03 0x03
0x14 STOP FIRMWARE 0x01 0x3F
0x15 ABORT 0x0F 0x0F 1 = (target << 8) | lun (bit 15 = channel), 2/3 = handle
0x16 ABORT DEVICE 0x03 0x03 1 = target/lun; outstanding commands return completion 5
0x17 ABORT TARGET 0x07 0x07
0x18 BUS RESET 0x07 0x07 1 = reset delay (s), 2 = channel. Async 0x8001 follows.
0x19–0x1C STOP / START / SINGLE STEP / ABORT QUEUE 0x03 0x07 1 = target/lun
0x1D GET DEVICE QUEUE STATUS 0x03 0x4F
0x1F GET FIRMWARE STATUS 0x01 0x07 → 2 = maximum outstanding commands (OpenBSD caps at 512, allocates half as many response entries)
0x20–0x2A GET … 0x01/0x03 readback twins of the SET commands below; 0x28 GET TARGET PARAMETERS: 1 = target<<8 → 2 = flags, 3 = offset<<8 | period, 6 = PPR info (12160)
0x30 SET INITIATOR SCSI ID 0x03 0x03 1 = id | channel << 7
0x31 SET SELECTION TIMEOUT 0x07 0x07 1 = ms (bus 0), 2 = ms (bus 1); 250
0x32 SET RETRY COUNT 0xC7 0xC7 1 = count, 2 = delay (bus 0); 6/7 bus 1
0x33 SET TAG AGE LIMIT 0x07 0x07 1/2 per bus; 8
0x34 SET CLOCK RATE 0x03 0x03 1 = MHz (40 or 60; Linux writes 40 even for a 1040)
0x35 SET ACTIVE NEGATION STATE 0x07 0x07 1/2 per bus: bits 4 and 5 enable REQ/ACK and data-line active negation (drivers disagree which is which; everyone sets both)
0x36 SET ASYNC DATA SETUP TIME 0x07 0x07 1/2 per bus, in clocks: 6 at 40 MHz, 8 or 9 at 60
0x37 SET PCI CONTROL PARAMETERS 0x07 0x07 1 = data DMA channel, 2 = command DMA channel; bit 1 = burst enable
0x38 SET TARGET PARAMETERS 0x4F 0x4F 1 = target << 8 | channel << 15, 2 = flags (§4.7), 3 = sync offset << 8 | sync period
0x39 SET DEVICE QUEUE PARAMETERS 0x0F 0x0F 1 = target << 8 | channel << 15 | lun, 2 = max queue depth, 3 = execution throttle
0x3A SET RESET DELAY PARAMETERS 0x07 0x07 1/2 per bus
0x40–0x42 RETURN BIOS BLOCK ADDR, WRITE FOUR RAM WORDS, EXEC BIOS IOCB BIOS support
0x45 / 0x46 SET / GET SYSTEM PARAMETER 0x03 / 0x01 0x03 1 = parameter word (IRIX writes 1, Linux 0 on a 1040)
0x48 / 0x49 GET / SET SCAM CONFIGURATION
0x4A / 0x4B SET / GET FIRMWARE FEATURES 0x03 0x03 1 = bit 0 fast posting, bit 1 report LVD bus transitions, bit 5 disable synchronous backoff, bit 8 disable reselection fairness, bit 9 "SGI cache flush" (IRIX)
0x50 / 0x51 LOAD / DUMP RAM A64 0xDF 0xDF firmware-mode versions of 0x09/0x0A
0x52 INIT REQUEST QUEUE A64 0xDF 0xFF as 0x10 plus 6/7 = address 63:48 / 47:32
0x53 INIT RESPONSE QUEUE A64 0xEF 0xFF as 0x11 plus 6/7
0x54 EXECUTE IOCB A64 0xCF 0x01
0x55 / 0x56 ENABLE TARGET MODE / GET TARGET STATUS
0x5A / 0x5B SET / GET DATA OVERRUN RECOVERY MODE 0x03 0x03 1 = 0: pad data until the target leaves data phase; 1: hang the bus and post 0x800C; 2: reset the bus, post 0x800D and return all I/O (IRIX and Linux use 2)
0x5C / 0x5D SET / GET HOST DATA

4.3 Firmware images and loading

The RISC runs code from external SRAM, 16-bit words addressed 0x0000– 0xFFFF. Drivers download an image to 0x1000 (ISP_CODE_ORG, QLW_CODE_ORG) and start it there. The image is an array of 16-bit words with a header the loaders rely on:

Word Meaning
0 first instruction (0x0078 in the QLogic-released images, 0x0804 in some)
1 image identifier: 0x1041 initiator firmware, 0x103A initiator/target firmware (NetBSD's 7.65, Linux's 1040.bin), 0x102D the PROM's self-test image
2 0
3 length in words, including the header
4 0
5… ASCII copyright banner, stored as big-endian pairs: " COPYRIGHT 1991,1992,1993,1994 QLOGIC CORPORATION …"
30… "ISP1020 Firmware Version 02.55 Customer No. 00 Product No. 00 $" (word 30 in every initiator image inspected)
len−1 a checksum word that makes the 16-bit sum of words [0, len) zero

Linux's qlogic/1040.bin has a 3-word file header in front of this (word 2 = load address 0x1000); NetBSD/OpenBSD embed the bare array.

Images seen:

Image ID Words Banner version
IRIX 6.5.30 ql.o risc_1040_code01 0x1041 11143 ISP1020 Firmware Version 02.55
IP30 PROM 4.17 (two copies, at 0x1C398 and 0xB4998 in the 1 MB image) 0x1041 11190 ISP1020 Firmware Version 02.55
IP30 PROM 4.17, at 0xB3D30 0x102D 1335 ISP1020 Self Test Firmware Version 01.00
NetBSD/OpenBSD asm_1040.h (initiator) 0x1041 11472 04.66 (dated Sep 2000)
NetBSD asm_1040.h with ISP_TARGET_MODE, linux-firmware 1040.bin 0x103A 16471 / 16728 07.65

ABOUT FIRMWARE returns the banner's version: IRIX prints "Firmware version: %d.%d.%d" from mailboxes 1–3 (2.55.0 for its image), FreeBSD reports "loaded F/W Revision 4.66.0". An emulator that does not run the RISC can parse the banner at word 30 of the loaded image to answer this, and should answer GET FIRMWARE STATUS with a plausible command limit (256 is safe: IRIX and Linux size their rings at 256/255 without asking, OpenBSD divides this value by 3 for its CCB count).

Two ways to load:

  • PIO: WRITE RAM WORD (4) for every word. The 1040 path in Linux (qla1280_load_firmware_pio), OpenBSD (qlw_load_firmware_words), FreeBSD/NetBSD for SCSI chips, and IRIX for the 1040 (ql_download_fw: 11143 mailbox commands at boot). The PROM reports "QL: write data failure at %d word" for this path.
  • DMA: LOAD RAM (1, or the A64 form 9 with the upper address words in mailboxes 6/7), 1000 words per command from a bounce buffer (Linux for 1080+, IRIX for 1080/1240/12160; the PROM prints "mailbox command for dma load"). Linux's DUMP_IT_BACK debug option reads it back with DUMP RAM; the PROM does read back ("Firmware Data Mismatch [0x%x]: his = %x, mine = %x"), so READ RAM WORD / DUMP RAM must return what was loaded.

Then VERIFY CHECKSUM (7) with mailbox 1 = 0x1000 (Linux, FreeBSD, OpenBSD, the PROM — "ISP Firmware Checksum mismatch: theirs %x, mine = %x"; IRIX's kernel skips it), EXECUTE FIRMWARE (2) with mailbox 1 = 0x1000, a 250 ms pause in OpenBSD, then ABOUT FIRMWARE. From here the full command set is live and the ROM-only commands may stop working (nobody issues them again except IRIX's parity routine, §5.1).

4.4 Request and response queues

Both queues are rings of 64-byte entries in host memory, physically contiguous, set up with INIT REQUEST QUEUE / INIT RESPONSE QUEUE (or the A64 versions when the ring may live above 4 GB — which on the Octane it always does, see §8). Sizes used: IRIX 256/256, Linux 255/63, OpenBSD "max commands" (from GET FIRMWARE STATUS, capped 512) and half that, FreeBSD RQUEST_QUEUE_LEN/RESULT_QUEUE_LEN.

Queue pointers live in mailboxes 4 and 5, using the in/out duality:

Host writes (incoming) Host reads (outgoing)
MAILBOX4 request in-pointer: index of the next free entry after the ones the host just filled request out-pointer: how far the firmware has consumed
MAILBOX5 response out-pointer: how far the host has consumed response in-pointer: index after the last entry the firmware posted

To issue commands the host fills entries at its in-pointer, then writes the new in-pointer to MAILBOX4 — no HCCR host interrupt. The firmware polls the incoming MAILBOX4 and processes entries from its out-pointer up to it, advancing the outgoing MAILBOX4. Linux computes free space from the outgoing MAILBOX4 (qla1280_req_pkt); OpenBSD and IRIX track it themselves (IRIX marks a free entry by writing 0xDEAD into its handle word).

When a command completes the firmware writes a status entry at the response in-pointer, advances the outgoing MAILBOX5, and raises the RISC interrupt without the semaphore. The host walks entries from its out-pointer to the new in-pointer, then writes its out-pointer to MAILBOX5 and HCCR 0x7000. The firmware does not overwrite entries the host has not released.

Entry header (first four bytes of every IOCB):

Byte Field
0 entry type
1 entry count (1 + number of continuation entries that follow)
2 system defined (drivers put a sequence number or ring index; firmware ignores)
3 flags, written by the firmware in responses: 0x01 continuation, 0x02 ring full / busy, 0x04 bad header, 0x08 bad payload, 0x10 bad count, 0x20 bad order

Types:

Type Entry
1 command, 32-bit addresses, 4 data segments
2 continuation, 7 × 32-bit segments
3 status (response queue)
4 marker
5 extended command (88-byte CDB)
6 / 7 ATIO / CTIO (target mode)
8 SCAM
9 command, 64-bit addresses, 2 data segments — what IRIX and 64-bit Linux use
0xA continuation, 5 × 64-bit segments
0xB–0xF target-mode entries (enable LUN, modify LUN, immediate notify, notify ack, CTIO1)
0x10 status continuation (60 more sense bytes)

Everything in an IOCB is little-endian. IRIX builds IOCBs in big-endian structs and byte-swaps each 32-bit word (munge()) before DMA so the memory image is the layout below.

Command entry (type 1):

Offset Size Field
0 4 header
4 4 handle — returned in the status entry; IRIX packs channel (bit 12), target (bits 11:8) and a 0–255 slot index
8 1 LUN
9 1 target ID; bit 7 = channel on dual-bus parts
10 2 CDB length
12 2 control flags (below)
14 2 reserved
16 2 timeout, seconds (0 = none)
18 2 data segment count (over this entry and its continuations)
20 12 CDB
32 4 × 8 data segments: address (32), length (32)

Command entry A64 (type 9): identical through the CDB; offsets 32–39 reserved; then two segments of {address 31:0, address 63:32, length} at 40 and 52.

Continuation (type 2): header, 4 bytes reserved, seven {address, length} pairs from offset 8. Continuation A64 (type 0xA): header, then five {lo, hi, length} triples from offset 4.

Control flags:

Bit Meaning
0x0001 no disconnects
0x0002 / 0x0004 / 0x0008 head-of-queue / ordered / simple queue tag
0x0010 target routine (target mode)
0x0020 data in (read)
0x0040 data out (write)
0x0080 no fast posting for this command (OpenBSD)
0x0100 disable auto request sense
0x0200 force asynchronous
0x0400 force synchronous
0x0800 initiate wide negotiation
0x1000 disable data parity
0x2000 stop queue on error
0x4000 extended sense supported
0x8000 priority

Marker (type 4): header; 4 bytes reserved; LUN at 8; target at 9 (bit 7 channel); modifier at 10: 0 = sync this ID/LUN, 1 = sync this ID, 2 = sync all. After a SCSI bus reset (async 0x8001, or the host's own BUS RESET) the firmware expects a marker before it will run new commands; every driver sets a "send marker" flag on 0x8001 and queues a SYNC ALL marker ahead of the next command. IRIX also sends one after an ABORT DEVICE.

Status entry (type 3):

Offset Size Field
0 4 header
4 4 handle of the completed command
8 2 SCSI status: low byte is the SCSI status byte; bit 9 (0x200) = sense data valid (RQCS_SV)
10 2 completion status (below)
12 2 state flags: 0x0100 got bus, 0x0200 got target, 0x0400 sent CDB, 0x0800 transferred data, 0x1000 got status, 0x2000 got sense, 0x4000 transfer complete
14 2 status flags: 0x0001 disconnected, 0x0002 synchronous, 0x0004 parity error, 0x0008 bus reset, 0x0010 device reset, 0x0020 aborted, 0x0040 timeout, 0x0080 negotiation
16 2 time
18 2 request sense length
20 4 residual byte count
24 8 reserved
32 32 sense data (auto request sense)

Completion status codes (1x00 family; IRIX SCS_*, FreeBSD RQCS_*):

Code Meaning Code Meaning
0x00 complete 0x12 device reset message failed
0x01 incomplete (selection timeout when state flags lack "got target") 0x13 identify message failed
0x02 DMA error 0x14 unexpected bus free
0x03 transport error 0x15 data underrun (residual valid)
0x04 reset occurred 0x16 command underrun (IRIX)
0x05 aborted 0x18 queued transaction with disconnect not set
0x06 timeout 0x19 queued transaction to target routine
0x07 data overrun 0x1A queued command when queueing disabled
0x08 command overrun 0x1B invalid IOCB entry type
0x09 status overrun 0x1C queue full
0x0A bad message 0x1D SCSI phase skipped
0x0B no message out 0x1E auto request sense failed
0x0C extended identify failed 0x1F wide negotiation failed
0x0D initiator detected error rejected 0x20 sync negotiation failed
0x0E abort message failed 0x21 LVD bus error
0x0F message reject rejected 0x22 data CRC (IRIX)
0x10 NOP rejected
0x11 message parity error rejected

Drivers treat 0x1F/0x20 as "fall back to narrow/async and retry", 0x15 and 0x07 as good with a residual, 0x01 without "got target" as selection timeout, 0x04/0x05 as requiring a marker.

4.5 Fast posting

With firmware feature bit 0 set, a command that completed with good status is reported as async event 0x8020 with the handle in MAILBOX1 (low) and MAILBOX2 (high) instead of a status entry (events 0x8021/0x8022 and 0x8031–0x8035 batch 32- and 16-bit handles on later firmware). Linux's default NVRAM settings enable it; OpenBSD, FreeBSD (for 1x40) and IRIX leave it off. An emulator must honour the bit if Linux is a target; it can ignore it for IRIX.

4.6 Asynchronous events

Posted in MAILBOX0 with the semaphore held:

Event Meaning Extra
0x8001 SCSI bus reset seen MAILBOX6 = channel (dual bus)
0x8002 system error (firmware crash) MAILBOX1–3 = RISC state; IRIX re-initialises the chip
0x8003 request queue transfer error
0x8004 response queue transfer error
0x8005 request queue wake-up (space freed)
0x8006 execution timeout reset
0x8007 bus device reset
0x800A extended message underrun
0x800B SCAM interrupt
0x800C bus hung after data overrun (recovery mode 1)
0x800D bus reset by the ISP after data overrun (mode 2)
0x800E bus mode transition (SE/HVD/LVD) MAILBOX1/2 = mode
0x800F target transfer rate reduced
0x8010 no memory
0x8020 fast-post completion MAILBOX1/2 = handle
0x8021 / 0x8022 RIO: one / two 32-bit handles MAILBOX1/2, 6/7
0x8031–0x8035 RIO: one to five 16-bit handles MAILBOX1–5

4.7 Target parameters and defaults

SET TARGET PARAMETERS mailbox 2 flags (OpenBSD QLW_TARGET_*, Linux TP_*, IRIX capability byte << 8):

Bit Meaning
0x8000 disconnect allowed
0x4000 parity checking
0x2000 wide
0x1000 synchronous
0x0800 tagged queuing
0x0400 auto request sense
0x0200 stop queue on check condition
0x0100 renegotiate on error
0x0080 narrow (negotiate down)
0x0040 asynchronous (negotiate down)
0x0020 PPR (12160)

OpenBSD starts every target at SAFE (0xC500: disconnect, parity, auto sense, renegotiate) plus narrow and async, and raises it to 0xFD00 after INQUIRY; Linux starts at renegotiate | auto sense | tags | wide | parity | disconnect; IRIX's defaults from ql.h are everything on (0xFD), sync period 25 (100 ns) or 12 (Fast-20, 50 ns), offset 8, execution throttle 255, 8 LUNs per target, 256 queue depth. Mailbox 3 carries the sync period in the low byte (in 4 ns units: 25 = 10 MHz, 12 = 20 MHz) and the offset in the high byte; 0 means asynchronous.

Common chip-wide defaults: selection timeout 250 ms, tag age limit 8, retry count/delay 0/0 (IRIX) or 0/1 (Linux), bus reset delay 3 s, async data setup 6 clocks at 40 MHz / 8–9 at 60 MHz, both active-negation enables on, both DMA burst enables on, clock rate 60 for a 1040.

4.8 NVRAM

Add-in QLA1040 boards carry a 128-byte serial EEPROM (93C46) reached through register 0x0E. Reading word n (OpenBSD qlw_read_nvram, FreeBSD isp_rdnvram_word): write SELECT, then SELECT|CLOCK; clock out 9 bits MSB first — the read opcode 6 in bits 8:6 and the 6-bit word address — each bit as three writes (SELECT[|DATA_OUT], then with CLOCK, then without); then clock in 16 result bits MSB first, sampling DATA_IN after each SELECT|CLOCK write; finally write 0. The 1080 family uses a 10-bit request (opcode << 8 | address) for a 256-byte part.

Layout of the 1040 image (FreeBSD ISP_NVRAM_*):

Byte Field
0–3 "ISP "
4 version (OpenBSD requires ≥ 2)
5 bits 1:0 FIFO threshold, bit 2 BIOS disable, bit 3 HBA enable, bits 7:4 initiator ID
6 bus reset delay
7 / 8 retry count / delay
9 bits 3:0 async data setup, bit 4 REQ/ACK active negation, bit 5 data active negation, bit 6 data DMA burst, bit 7 command DMA burst
10 tag age limit
11 termination and clock bits (bit 3 "enable 60 MHz", bit 4 SCSI reset disable, bit 6 FIFO threshold 128)
12–13 selection timeout (LE)
14–15 max queue depth (LE)
16 bus size/type, clock speed, soft termination, flash present
28 + 6·t per target t: byte 0 flags (bit 0 renegotiate, 1 stop queue, 2 auto sense, 3 tags, 4 sync, 5 wide, 6 parity, 7 disconnect), byte 1 execution throttle, byte 2 sync period, byte 3 bits 3:0 sync offset, bit 4 device enable, bit 5 LUN disable
127 checksum: the byte sum of all 128 bytes is 0

The Octane's chips have none. OpenBSD's probe reads garbage, fails the "ISP " / checksum test, prints "nvram corrupt" and uses defaults, with initiator ID 0 on sgi. Returning DATA_IN = 0 for every bit is enough. Linux does not read a 1040's NVRAM at all (driver_setup.no_nvram = 1).

5. How each OS programs it

5.1 IRIX 6.5.30 ql (kernel)

The driver's view of the registers is ql.h's ISP_REGS, a struct of u_shorts laid out at PCI offset XOR 2 ("These register definitions assume a 32 bit PCI interface (for PIO) on a big endian machine. For 32 bit PCI little endian, macros used to access them need to XOR the address with 2"). So in the disassembly a sh to 0xC2(base) is HCCR (0xC0), 0x72(base) is MAILBOX0, 0x7A(base) is MAILBOX5, 0x0A(base) is ICR and 0x08(base) is ISR. §8 explains why.

Compile-time choices for IP30 (ql.h): USE_IOSPACE (BAR0), QL_SRAM_PARITY 1, BUS_RESET_ON_INIT 0, LOAD_VIA_DMA (used for the 1080 family only), PROBE_ALL_LUNS, request and response depth 256, IOCB_SEGS 2, CONTINUATION_SEGS 5 (type 9 / 0xA entries), up to 254 continuation entries per command.

qlattach: map BAR0 with pciio_pio_addr(dev, NULL, PCIIO_SPACE_WIN(0), 0, 256, &map, 0); set PCI command bit 0, latency timer 0x40, cache line 0x80; pcibr_rrb_alloc; interrupt via pciio_intr_alloc(..., INT A); then ql_init_board → ql_init_isp → ql_download_fw and ql_reset_interface.

ql_init_board identifies the chip from PCI vendor 0x1077 and device: 0x1020 → clock 60, one channel, firmware risc_1040_code01 (risc_1040_code_addr01 = 0x1000, length 11143); 0x1240 → 60 MHz, two channels; 0x1080 → 100; 0x1280 → 100, two channels; 0x1216 → 80, two channels. It records the Bridge revision (pcibr_asic_rev) for the BRIDGE_B_PREFETCH_DATACORR_WAR workaround.

ql_download_fw (register offsets are PCI offsets):

  1. ICR (0x08) = 0x0005 (soft reset + RISC int enable); 30 µs.
  2. HCCR = 0x1000; HCCR = 0x3000; 30 µs. Poll HCCR bit 0x40 clear, up to 10000 × 100 µs; failure → error 11.
  3. Set AFLG_CHIP_DOWNLOAD_FIRMWARE | AFLG_CHIP_SRAM_PARITY_ENABLE and run ql_enable_sram_parity(chip, check = 1):
    • HCCR = 0xA100 (parity enable, bank 0).
    • HCCR = 0x2000 (pause). With the RISC paused write R3 (0x86) = 0xBEEF, R4 (0x88) = 0x8000, R8 (0x90) = 0x0016, R9 (0x92) = 0x0449, RAR1 (0xA8) = 0x1000, CDMA_ADDR0..3 (0x2C, 0x2E, 0x30, 0x32) = 0x43A4, 0x8421, 0x08C2, 0x097A. HCCR = 0x3000; 30 µs.
    • MAILBOX1 = 0x0016, MAILBOX0 = 0x0002 (EXECUTE FIRMWARE at address 0x16, i.e. a routine in the chip's boot ROM that initialises the SRAM parity bits), HCCR = 0x5000. Poll MAILBOX0 == 0x4000 for up to 4000 ms (QL_COMPUTE_PARITY_MEMORY). No interrupt is used.
    • SEMA = 0, ISR = 0 (harmless), HCCR = 0x7000, HCCR = 0x3000; 30 µs.
    • 32 × READ RAM WORD (5) at 0x1000, 0x1001, …; after each, read HCCR and fail if bit 0x400 (parity error) is set or the command failed. On failure: ICR = 5, HCCR = 0x1000, restore HCCR, HCCR = 0x7000, 0x3000, print "Q-logic sram parity error on bank 0 hccr=%x" and continue without parity (error 7 if the chip is unusable). An emulator should answer EXECUTE FIRMWARE at 0x0016 with 0x4000, stay in ROM mode, keep READ RAM WORD working, and never set HCCR bit 0x400.
  4. ql_enable_intrs: ICR = 0x0006.
  5. MAILBOX REGISTER TEST with 0x1111, 0x2222, 0x3333, 0x4444, 0x5555, 0x6666, 0x7777 in mailboxes 1–7 (8 in, 8 out). Failure → error 1. The kernel only checks that the command completed.
  6. HCCR = 0x2000; CONF1 (0x06) = 0x0044 (128-byte FIFO, burst); 100 µs; SEMA = 0; HCCR = 0x3000; 30 µs.
  7. For device 0x1020: WRITE RAM WORD (4) for each of the 11143 words, mailbox 1 = 0x1000 + i, mailbox 2 = word (3 in, 3 out). Failure → error 4. Other chips: copy the image to a bounce buffer, byte-swap it 16-bit-wise (munge16), map it with pciio_dmatrans_addr(…, PCIBR_VCHAN1 | PCIIO_BYTE_STREAM | PCIIO_DMA_A64 | PCIIO_DMA_DATA) and issue LOAD RAM A64 (9) with mailbox 1 = 0x1000, 2/3 = address 31:0, 4 = length, 6/7 = address 63:32.
  8. EXECUTE FIRMWARE (2), mailbox 1 = 0x1000 (2 in, 1 out). Failure → error 5. No VERIFY CHECKSUM.
  9. If showconfig: ABOUT FIRMWARE (1 in, 3 out), print "Firmware version: %d.%d.%d".

ql_reset_interface(chip, bus_reset), run at init and on every recovery:

  1. Flush queued requests; zero request_in/out, response_in/out; mark all 256 request entries free (handle word = 0xDEAD).
  2. ICR = 0; HCCR = 0x2000; poll HCCR bit 0x20 set (10000 × 100 µs; error 1).
  3. ICR = 0x0001 (soft reset); 30 µs; HCCR = 0x1000; HCCR = 0x3000; 30 µs; poll HCCR bit 0x40 clear (error 2).
  4. Device 0x1020: CDMA_CONTROL (0x22) = 0x000E, DDMA_CONTROL (0x42) = 0x000E. (1080 family: CONF1 |= 0x300, the banked DMA control registers at 0x80/0xA0/0xC0 = 0xE, then OR 0x2 into the bank's config registers and HCCR.)
  5. Wait while MAILBOX0 == 0x0004 (busy), up to 1000 × 1 ms.
  6. ql_enable_sram_parity(chip, 0): HCCR = 0xA100 if parity was enabled.
  7. CONF1 = 0x0044; SEMA = 0; HCCR = 0x3000; 30 µs; ICR = 0x0006.
  8. EXECUTE FIRMWARE at 0x1000 again (error 3) — the image loaded earlier must still be in RAM.
  9. SET PCI CONTROL PARAMETERS (0x37): 1 = 2, 2 = 2 (error 4).
  10. SET CLOCK RATE (0x34): 60 (error 5).
  11. Zero the 16 KB response ring, map it with pciio_dmatrans_addr(…, PCIBR_VCHAN1 | PCIIO_BYTE_STREAM | PCIIO_DMA_A64 | PCIIO_DMA_CMD) and INIT RESPONSE QUEUE A64 (0x53): 1 = 256, 2/3 = address 31:0, 4 = 0, 5 = response_in (0), 6/7 = address 63:32 (8 in, 8 out; error 6).
  12. Same for the request ring with INIT REQUEST QUEUE A64 (0x52): 1 = 256, 4 = request_in (0), 5 = 0 (error 7).
  13. If bus_reset (from ql_bus_reset_on_boot, default off for IP30): BUS RESET (0x18) per channel, 1 = that bus's reset delay, 2 = channel (error 8).
  14. SET INITIATOR SCSI ID (0x30): 1 = (channel << 7) | (hostid & 0xF), per channel (error 9).
  15. SET SELECTION TIMEOUT (0x31) 250/250 (error 10); SET RETRY COUNT (0x32) (error 11); SET TAG AGE LIMIT (0x33) 8/8 (error 12); SET ACTIVE NEGATION (0x35) 0x30/0x30 (error 13); SET ASYNC DATA SETUP TIME (0x36) 8/8 (error 14); SET DATA OVERRUN RECOVERY MODE (0x5A) 2 (error 15); SET RESET DELAY PARAMETERS (0x3A) (error 16); SET SYSTEM PARAMETER (0x45) 1 (error 17); SET FIRMWARE FEATURES (0x4A) 0x0002 for the 1040, 0x0322 for the 12160 (error 18).
  16. Set AFLG_CHIP_INITIALIZED.

Then ql_probe_bus does, per target, SET TARGET PARAMETERS (0x38) with 1 = target << 8 | channel << 15, 2 = capability flags << 8, 3 = offset << 8 | period, and SET DEVICE QUEUE PARAMETERS (0x39) with 1 = target << 8 | channel << 15 | lun, 2 = 256, 3 = throttle, for each of 8 LUNs; then INQUIRY commands through the request queue. GET TARGET PARAMETERS (0x28) and ABORT DEVICE (0x16) are used later.

Mailbox command path (ql_mbox_cmd(chip, out[], nin, nout, cmd, m1, m2, m3, m4, m5, m6, m7)): writes mailboxes 7, 6, 5 (skipped for 0x38 and 9), 4 (skipped for 0x32, 0x53, 0x38), 3, 2, 1, 0 in that order, then HCCR = 0x5000. In interrupt, dump, watchdog or parity-check context it polls: ISR bit 2, then SEMA bit 0; if the semaphore is clear it treats the interrupt as a response-queue update (reads MAILBOX5, sets AFLG_CHIP_ASYNC_RESPONSE_IN) and writes HCCR = 0x7000. 100 retries of 1 s; on each timeout it prints the ISR/SEMA/HCCR/ICR values and writes HCCR = 0x7000. Otherwise it sleeps on a semaphore that ql_service_mbox_interrupt posts.

Interrupt path (qlintr → ql_service_interrupt): loop while ISR bit 2: if SEMA bit 0, ql_service_mbox_interrupt — MAILBOX0 == 0x4000 copies mailboxes 0–7 to the chip's save area and wakes the waiter; 0x8001 → "SCSI bus RESET seen", set the send-marker flag; 0x8002 → "Unrecoverable SCSI bus state/condition (mbox1=0x%x mbox2=0x%x)" and ql_reset_interface; 0x800E → record bus mode from MAILBOX1; others print their message; then SEMA = 0. Then read MAILBOX5 (response in) and walk status entries: munge() the 64 bytes, check header flags 0x0E ("Detected busy condition / a bad header / a bad payload"), require type 3 ("Invalid driver command issued" otherwise), decode the handle (channel = bit 12, target = bits 11:8, index = bits 7:0), write 0xDEAD into the entry's handle, complete the request; finally write MAILBOX5 = new response out and HCCR = 0x7000.

Request path (ql_start_scsi): a type 9 entry (type at byte 0 after munge, count, sequence, handle, LUN, target | channel << 7, CDB length, control flags 0x20 read / 0x40 write plus tag bits 0x2/0x4/0x8 and 0x200/0x400/0x800 for forced async/sync/wide, timeout in seconds, segment count, CDB, two 64-bit segments), type 0xA continuations with five segments each, then MAILBOX4 = new request in-pointer. A SYNC ALL marker (modifier 2) precedes the next command when the send-marker flag is set. "Transfer length (0x%x) too large; spans more than %d continuation entries" caps a command at 254 continuations.

What IRIX never does on the Octane: read NVRAM, VERIFY CHECKSUM, LOAD RAM by DMA (1040), enable fast posting, read CONF0 or RISC_PSR (it trusts the PCI IDs), issue BUS RESET at boot.

5.2 IRIX IP30 PROM 4.17

Only strings and embedded images were examined; the PROM's driver is in the flash image twice (main and recovery halves). It carries the same firmware 2.55 as the kernel (11190 words — a slightly different build) and a 1335-word "ISP1020 Self Test Firmware Version 01.00" (image ID 0x102D). Messages show the sequence: "QLISP1040" probe; mailbox register test ("ql_init_isp - mailbox compare error" — it checks the echo); self-test load via WRITE RAM WORD ("QL: write data failure at %d word"), EXECUTE FIRMWARE ("QL: EXECUTE FIRMWARE Command Failed.") and result check ("SCSI self test failed (mbox val %x)."); then the 2.55 load by DMA ("mailbox command for dma load"), a read-back compare ("Firmware Data Mismatch [0x%x]: his = %x, mine = %x"), VERIFY CHECKSUM ("ISP Firmware Checksum mismatch: theirs %x, mine = %x"), SET SELECTION TIMEOUT ("ql_init_isp - SET_SELECTION_TIMEOUT Command Failed"), BUS RESET ("qlreset - SCSI BUS RESET Command Failed", "ql_mbox_cmd - ISP SCSI bus RESET seen"). What the self-test image is expected to leave in the mailboxes is not known from the strings; the first run of the PROM against an emulated chip (IRIS_IP30_TRACE=1) will show which value it waits for after EXECUTE FIRMWARE of that image. Expect it to want 0x4000 in MAILBOX0 and possibly a result word in MAILBOX1.

5.3 Linux qla1280

qla1280_initialize_adapter: SEMA = 0, HCCR 0x7000, HCCR 0x6000; qla1280_chip_diag (ICR soft reset, wait up to 5 s; CONF1 = 0; HCCR = 0x1000|0x3000|0x9000; wait MAILBOX0 != 4; check product ID 0x4953 / 0x5020-or-0 / 0x2020 / 1; enable interrupts; register test with 0xAAAA 0x5555 0xAA55 0x55AA 0xA5A5 0x5A5A 0x2525). For a 1040: PIO load from qlogic/1040.bin (fw_data[2] = 0x1000 start, code from byte 6); 1080+: LOAD RAM A64 1000 words at a time through the request ring buffer. VERIFY CHECKSUM, EXECUTE FIRMWARE. qla1280_nvram_config (defaults on a 1040): CONF1 = FIFO threshold << 4 | burst << 2 (threshold 3 → 0x30 on a 1040 unless hardware revision 4 = 1040A), CDMA/DDMA_CONF |= burst; SET SYSTEM PARAMETER 0; SET CLOCK RATE 40 (IS_ISP1x40); SET FIRMWARE FEATURES (fast posting on, sync backoff disabled → 0x21); SET RETRY COUNT 0/1; SET ASYNC DATA SETUP 6; SET ACTIVE NEGATION 0x30; SET DATA OVERRUN RECOVERY 2; SET PCI CONTROL 2/2; SET TAG AGE LIMIT 8; SET SELECTION TIMEOUT 250; per bus SET INITIATOR ID 7 and per target SET TARGET PARAMETERS (renegotiate | auto sense | tags | wide | parity | disconnect, sync disabled until INQUIRY) and 8 × SET DEVICE QUEUE (32, throttle 16). INIT REQUEST/RESPONSE QUEUE A64 with 255/63 entries. BUS RESET per bus (twice before declaring the bus dead). Commands are type 9/0xA on 64-bit kernels (QLA_64BIT_PTR), type 1/2 otherwise; qla1280_isr handles 0x8020 fast-post completions (handle = MAILBOX2 << 16 | MAILBOX1).

5.4 FreeBSD / NetBSD isp

isp_reset: pause; read CONF0 for the revision and SXP_PINS_DIFF / RISC_PSR for differential and Ultra; ICR soft reset; CDMA/DDMA CONTROL clear; wait ICR bit 0 clear; CONF1 = 0; HCCR 0x1000; SEMA = 0; CONF1 = burst (and FIFO unless 1040A); CDMA/DDMA_CONF burst; RISC_MTR = 0x1212 (or 0x1313 with HCCR STEP for "fast MTTR" boards); HCCR 0x3000; request / response pointers = 0 (mailboxes 4 and 5); HCCR 0x9000; wait MAILBOX0 != 4; NOP; register test (0xdead 0xbeef 0xffff 0xa5a5 0x5a5a); PIO load with WRITE RAM WORD; VERIFY CHECKSUM; EXECUTE FIRMWARE; ABOUT FIRMWARE; GET FIRMWARE STATUS (MAILBOX2 = max commands). isp_scsi_init: SET RETRY COUNT, ASYNC DATA SETUP, ACTIVE NEGATION, TAG AGE LIMIT, SELECTION TIMEOUT, per channel SET INITIATOR ID and SET TARGET PARAMETERS (safe defaults, narrow+async when firmware was loaded) and SET DEVICE QUEUE for 8 LUNs, INIT RESPONSE then REQUEST QUEUE (32-bit forms on a 1040), SET FIRMWARE FEATURES only for Ultra2+. NetBSD's isp_pci.c keeps 1040 DMA segments from crossing a 16 MB boundary (dbound = 1 << 24) and leaves RQSTYPE_A64 disabled. Interrupt: ISR & RISC_INT, SEMA & LOCK, MAILBOX0.

5.5 OpenBSD qlw (ran on real Octanes)

qlw_attach: if mailboxes 1–3 don't read "ISP ", HCCR release; HCCR pause; qlw_softreset (ICR = 0; ICR = 1; wait clear; CONF1 = 0; HCCR 0x1000; SEMA = 0; HCCR 0x3000; mailboxes 4/5 = 0; ICR = 6; HCCR 0x9000; NOP); defaults (CONF1 0x34, initiator 0 on sgi, 7 elsewhere); NVRAM read (fails on the Octane → defaults); PIO firmware load + VERIFY CHECKSUM; EXECUTE FIRMWARE; 250 ms; ABOUT FIRMWARE; GET FIRMWARE STATUS; CONF1 = config; CDMA/DDMA_CONF burst; SET FIRMWARE FEATURES only if LVD notify wanted; SET CLOCK RATE 60; SET RETRY COUNT; SET ASYNC DATA SETUP 9; SET ACTIVE NEGATION; SET TAG AGE LIMIT; SET SELECTION TIMEOUT; per bus SET INITIATOR ID and per target SET TARGET PARAMETERS (safe | narrow | async) + 8 × SET DEVICE QUEUE; INIT REQUEST then RESPONSE QUEUE (32-bit); BUS RESET per bus; wait reset delay. Commands are type 1/2 with a SYNC ALL marker after a reset; no fast posting.

6. Minimal emulation model

What a device model has to do for the PROM, IRIX, Linux, the BSDs:

State: PCI config (IDs, command, BARs, revision 5); 64K words of RISC RAM (persistent across soft and RISC resets); the 16-bit registers; incoming and outgoing mailboxes (8 each); SEMA; ICR/ISR; HCCR status bits (host int, reset, pause, parity enables); "ROM mode" vs "firmware running"; per-queue base address, length, in/out indices; per-target and per-device parameters (store and echo); send-marker expectation (optional).

Reset (ICR bit 0 or HCCR 0x1000 → 0x3000): clear ISR/SEMA, outgoing mailboxes = 0, 0x4953, 0x5020, 0x2020, 0x0001, back to ROM mode, queues undefined, HCCR bit 0x40 reads clear (or clears after a few µs). Keep RAM.

HCCR: 0x5000 → process the mailbox command (immediately or after a short delay), set the host-int status bit until processed; 0x2000/0x3000 toggle the pause bit; 0x7000 clears the RISC interrupt (ISR bit 2); 0xA000 stores the parity enables; 0x4000, 0x6000, 0x8000, 0x9000, 0xE000, 0xF000 accepted and ignored.

Mailbox completion: write outgoing mailboxes, SEMA |= 1, ISR |= 4, raise the PCI interrupt if ICR bits 1 and 2 are set. The host clears SEMA and writes HCCR 0x7000. Do not post a response-queue interrupt while SEMA bit 0 is held.

ROM mode: commands 0–8, 9/0xA, 0x40–0x42 answer; EXECUTE FIRMWARE at 0x1000 (or any address other than 0x0016) switches to firmware mode; EXECUTE at 0x0016 answers 0x4000 and stays in ROM mode; everything else 0x4001.

Firmware mode: all SET/GET commands succeed and echo their inputs; ABOUT FIRMWARE from the banner; GET FIRMWARE STATUS → 256; INIT queues record base/length and reset the pointers (outgoing MAILBOX4/5 = incoming); BUS RESET completes, then posts async 0x8001 (MAILBOX6 = channel) after the delay; ABORT DEVICE completes outstanding commands of that LUN with status 0x05.

Request processing: on a write to incoming MAILBOX4 (and after an INIT REQUEST QUEUE), read entries from the out index to the new in index (wrap at length), handle types 1, 2, 4, 9, 0xA; for each command run the SCSI command against the target model using the data segments (64-bit addresses on the Octane), then post a type 3 status entry with the handle, SCSI status, completion 0 (or 0x15 with the residual on underrun, 0x01 with state flags 0 for no such target), auto-sense data with bit 9 of the SCSI status set when the target returned CHECK CONDITION; advance the outgoing MAILBOX5 and raise ISR bit 2 (without SEMA). Advance the outgoing MAILBOX4 as entries are consumed. Markers and unknown entry types are consumed silently (or answered with flags 0x04 for bad header).

Timing: IRIX waits up to 1 s per mailbox command, 4 s for the parity routine, 1 s for MAILBOX0 to leave 4; Linux 5 s for reset bits; OpenBSD polls for 400 ms during attach. Completing everything instantly is fine.

Things that bite: mailbox 4/5 in/out duality; RAM surviving reset; EXECUTE at 0x16; READ RAM WORD after load; the type 9 segment layout (address low, address high, length); the status entry's sense-valid bit; IRIX requiring type 3 entries only; the PROM comparing the register-test echo and the checksum; Linux's SET CLOCK RATE 40; the Bridge byte swap (next section) making every IRIX register access appear at offset ^ 2.

7. Endianness summary

  • Register PIO: the chip is little-endian PCI. The Bridge is configured by the PROM and IRIX so that a 16-bit access from the big-endian CPU to byte address A reaches PCI address A ^ 2 with the value unchanged (word-swapped, "data invariant"). That is why ql.h lays the struct out at offset ^ 2, and why IRIX writes HCCR at 0xC2. Linux and OpenBSD instead clear the Bridge's swap bits and swap in software, so they address 0xC0 directly. The device model should be written against PCI offsets and let the Bridge model do the swap.
  • IOCBs and firmware DMA: little-endian in memory, IRIX swaps them in software (munge, munge16) and maps the memory PCIIO_BYTE_STREAM (no Bridge swap), so DMA bytes are exact.
  • Firmware images in C sources and in ql.o are arrays of 16-bit words; on disk 1040.bin is little-endian words. The banner text is stored big-endian within words (hence "C PORYGITH" when a little-endian file is dumped as bytes).

8. The Octane side: Bridge, DMA addresses, interrupts

  • Location: Bridge at xtalk widget 15, PCI devices 0 and 1. Their registers are reached through the Bridge's per-device "devio" windows (2 MB each for devices 0 and 1) in PCI I/O space; IRIX maps BAR0 with PCIIO_SPACE_WIN(0).
  • Byte swapping: the Bridge has per-device PIO swap (BRIDGE_DEV_DEV_SWAP, bit 13 of the device(x) register) and large-window swap bits (BRIDGE_CTRL_IO_SWAP/MEM_SWAP, bits 23/22 of widget control), plus DMA swap bits BRIDGE_DEV_SWAP_DIR (bit 19, direct window) and BRIDGE_DEV_SWAP_PMU (bit 20, translated/ATE). IRIX's register struct shows PIO swapping is on for the QL devices. OpenBSD clears the large-window swap bits ("IOC serial console relies on this") and uses software-swapping bus_space tags for the QLogic but not for IOC3/IOC4/ RAD; it sets the DMA swap bits for everything but those SGI parts. Linux clears the large-window swap bits, sets SWAP_DIR for every slot and then clears it again for the IOC3. For the IOC3 the natural mode is no swap; for the QLogic it is swap.
  • DMA addresses: IRIX maps queues and data with PCIIO_DMA_A64, so the ISP issues 64-bit PCI addresses (dual address cycles). The Bridge treats bits 63:48 as attributes and 47:0 as the xtalk address: bits 63:60 target widget (8 = HEART, i.e. memory; Linux uses masterwid << 60), bit 59 prefetch, bit 58 precise, bit 57 virtual channel 1 (IRIX passes PCIBR_VCHAN1 for the QL, so this is set), bit 56 barrier (Linux sets PCI64_ATTR_BAR), bit 55 byte swap (clear for the QL: PCIIO_BYTE_STREAM), bits 55:48 "RMF". The address is the physical address as the CPU sees it (Linux: dma = baddr + paddr, paddr = dma & ~(0xff << 56)). The 32-bit direct window (0x8000_0000–0xFFFF_FFFF on PCI, via the Bridge direct-map register) and the ATE-translated window (0x4000_0000–0x7FFF_FFFF) are what OpenBSD and Linux's IOC3 use; the IRIX QL driver does not.
  • Read response buffers: both the PROM and IRIX allocate Bridge RRBs for the QL slots and fail the controller if they cannot; the Bridge model needs working even/odd response buffer registers.
  • Interrupts: pin A of devices 0 and 1 → Bridge interrupt bits 0 and 1 → the Bridge's interrupt destination (HEART), as configured by the PROM/IRIX in the Bridge INT_ADDR/INT_ENABLE registers. IRIX asks for one line (pciio_intr_alloc(dev, desc, 1, …)).
  • Bus type: PINS_DIFF bit 8 clear (single-ended) for both controllers; initiator ID 0; three disks at IDs 1–3 on controller 0 in the reference machine.

9. Open items

  • The real chips' CONF0, ID_LO/HI, RISC_HRL and SXP_PART_ID values have not been captured from the Octane (needs a user-space PCI PIO on the running machine, or a trace from the PROM once the emulator gets that far). The tables give the values the drivers expect.
  • The PROM self-test firmware's result protocol (§5.2).
  • What IRIX passes in SET RESET DELAY PARAMETERS and the exact 12160 branches are not decoded; neither matters for a 1040 model.
  • Firmware timing (reselection, tag queueing order) is the firmware's business; a model that completes commands in order is enough for IRIX.