Proven C Book←↑→

Appendix M — How machines talk: wires, buses and interruption

In the appendix on C without an operating system, devices were handled by address. Write *(volatile uint8_t *)0x10000000 = 'A' and a character went out. But what happens after that one write? Which wires carry which voltages in what order, and how does the other end turn them back into a character?

This appendix is that story. The aim is not to become a hardware engineer but to be able to read the first page of a datasheet.

Platform note. what this appendix rests on, and its limits

This machine has no I2C device and no PCI card attached. So there is exactly one thing here that can be called “measured” — the cost of the operating system’s signals. The rest is handled two ways: protocols are built byte by byte to the specification and read back to check, and speeds and times are computed. Tables and text mark which is “computed” and which is “measured”.

Words to know first#

WordMeaningEasily confused
line, signalone strand carrying a voltage“data lines” and “control lines” do different work
busa bundle of lines shared by several devicespoint-to-point links (PCIe, USB) are often buses in name only
protocolthe agreement about what goes on the wire and whensame wires, different agreement, and nothing is understood
frameone chunk sent at a timenot “one byte” — markers are attached front and back
baudsignal changes per secondnot necessarily bits per second (it depends on the modulation)
throughputbytes actually moved per secondalways below the theoretical figure, because of the frame’s overhead
latencythe time from asking to being answeredthroughput can be large while latency is large too
controller / targetthe side that speaks / the side that answersmodern specifications prefer these words to the older ones
full duplex / half duplexboth sides at once / by turnsthe number of wires usually decides this

Table 105.1 — The words used in this appendix

★ The third row from the bottom is the most frequently misunderstood in practice. Throughput and latency are different axes. Fill a truck with hard drives and drive it and the throughput is enormous while the latency is hours. Conversely I2C is slow (100 kbit/s) and the latency of exchanging one byte is under a millisecond.

Four axes that separate connections#

There are dozens of ways to connect things, and only four questions to ask. Settle these four and the rest is detail.

AxisOne sideThe otherWhat it settles
number of wiresserial — queued on one strandparallel — at once on severalpin count and top speed (Figure 105.1)
timingasynchronous — no clock linesynchronous — a separate clock linewhether both sides must agree the speed in advance
addressingnone — there is only one other endpresent — one of several is calledwhether many devices can hang off one wire
who startsonly one side (controller-target)anyone (peer, arbitration)how collisions are prevented

Table 105.2 — Four axes that separate connections

serial-parallel

Figure 105.1 — Sending the same byte serially and in parallel.

Lay the common schemes out along those four axes and this is the result. The sections that follow unpack the rows one at a time.

NameWiresClockAddressIn a phrase
UART (universal asynchronous receiver-transmitter) / RS-232serial, 2 strands (+ flow control)none (asynchronous)nonethe oldest and the simplest
parallel portparallel, 8 strands + controlyes (a strobe)noneit was fast, then hit the wall of speed
I2Cserial, 2 strandsyes (SCL)yes (7 bits)many devices on two wires
SPIserial, 4 strandsyes (SCK)chosen by a wire (CS)fast and simple, and it eats wires
CANserial, 2 strands (differential)nonecarried in the messageanyone speaks and arbitration settles it
PCIeserial lanes ×N (differential)embedded (carried in the coding)yes (configuration space)the motorway inside the machine
USBserial, 2 to 4 strandsembeddedyes (the host assigns it)the host commands everything

Table 105.3 — The connections covered in this appendix

A common misconception. serial is slow and parallel is fast

It was so once. Today it is the reverse, because the faster things go the harder it is to line up the arrival times of several lines (Figure 105.1). So today’s fast links — PCIe, USB, SATA, Ethernet — are all serial, and bundle several of those serial lines (lanes) instead. “Several serial lines rather than parallel” is today’s answer.

The oldest wire — UART and RS-232#

The simplest way to join two machines is this. Send the bits along one wire in turn. There is no clock line either. Instead both sides agree in advance on “so many bits per second” and each counts on its own clock.

The device that handles this is a UART (universal asynchronous receiver-transmitter).

Frames — cutting characters out of an idle wire#

Without a clock, how is “a character starts now” recognised? By an agreed shape.

OrderNameValueWhatWithout it
—the idle state1 (high)the wire when nothing is being sentthere is no baseline against which a start can be seen
1the start bit0 (low)“it begins now” — the receiver syncs its clock herecharacter boundaries cannot be found
2–98 data bitsthe valuethe low-order bit goes first—
(optional)paritycomputedone bit making the count of ones even (or odd)a single-bit error goes unnoticed
10the stop bit1 (high)“it ends here” — distinguishable from the next start bitsending back to back loses the boundary

Table 105.4 — The ten bits of an 8N1 frame

The notation 8N1 is that table abbreviated.

PositionMeaningCommon valuesNote
first digitdata bits8, rarely 77 bits is a trace of the days of ASCII alone
middle letterparityN (none), E (even), O (odd)N is today’s default
last digitstop bits1, rarely 22 gives a slow receiver room to breathe

Table 105.5 — How to read a notation like 8N1

examples-en/apx-links/uart_frame.c

/* UART 프레임을 비트로 짓고, *보율이 어긋난 수신기*로 다시 읽어 본다.
   하드웨어는 없다 --- 선 위의 전압을 잘게 썬 배열로 흉내 낸다.
   한 비트를 16칸으로 나누어(흔한 UART 가 정말 이렇게 샘플링한다) 시간을 표현한다. */
#include <stdio.h>
#include <stdint.h>
#include <string.h>

#define OS      16u        /* oversampling --- 한 비트를 몇 칸으로 나눌까 */
#define MAXW    4096u

/* 8N1: 시작 1 + 자료 8 + 정지 1 = 10비트. 자료는 *낮은 자리부터* 나간다. */
static size_t make_frame(unsigned char *w, uint8_t byte, unsigned parity_bits)
{
    size_t n = 0;
    for (unsigned i = 0; i < OS * 2; i++) w[n++] = 1;      /* 놀고 있을 때는 높다 */
    for (unsigned i = 0; i < OS; i++)     w[n++] = 0;      /* 시작 비트: 떨어뜨린다 */
    unsigned ones = 0;
    for (int b = 0; b < 8; b++) {
        unsigned bit = byte >> b & 1;                       /* LSB first */
        ones += bit;
        for (unsigned i = 0; i < OS; i++) w[n++] = (unsigned char)bit;
    }
    if (parity_bits) {                                      /* 짝수 패리티라면 */
        unsigned p = ones & 1;                              /* 1의 개수를 짝수로 맞춘다 */
        for (unsigned i = 0; i < OS; i++) w[n++] = (unsigned char)p;
    }
    for (unsigned i = 0; i < OS; i++)     w[n++] = 1;      /* 정지 비트: 다시 높다 */
    for (unsigned i = 0; i < OS * 2; i++) w[n++] = 1;
    return n;
}

/* 수신기: 시작 비트의 내려감을 찾고, 제 비트 길이로 한가운데를 찍어 읽는다.
   rx_bit 이 16 이 아니면 그만큼 보율이 어긋난 것이다. */
static int receive(const unsigned char *w, size_t n, double rx_bit,
                   uint8_t *out, int *stop_ok)
{
    size_t edge = 0;
    while (edge < n && w[edge] != 0) edge++;                /* 내려가는 자리 */
    if (edge >= n) return -1;

    double t0 = (double)edge;
    uint8_t v = 0;
    for (int b = 0; b < 8; b++) {
        double at = t0 + rx_bit * (b + 1) + rx_bit / 2.0;   /* (b+1)번째 비트의 한가운데 */
        size_t idx = (size_t)(at + 0.5);
        if (idx >= n) return -1;
        v |= (uint8_t)(w[idx] << b);
    }
    double sat = t0 + rx_bit * 9 + rx_bit / 2.0;            /* 정지 비트 자리 */
    *stop_ok = (sat < n) && w[(size_t)(sat + 0.5)] == 1;
    *out = v;
    return 0;
}

static void show_wave(const unsigned char *w, size_t n)
{
    printf("  ");
    for (size_t i = 0; i < n; i += OS / 2) putchar(w[i] ? '-' : '_');
    printf("\n  ");
    /* 비트 경계에 이름을 붙인다 */
    const char *lab[] = { "  ", "  ", "St", "d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "Sp", "  ", "  " };
    for (size_t i = 0, k = 0; i < n; i += OS, k++)
        printf("%-*s", (int)(OS / (OS / 2)), k < sizeof lab / sizeof *lab ? lab[k] : "  ");
    printf("\n");
}

int main(void)
{
    unsigned char w[MAXW];
    const uint8_t byte = 'K';       /* 0x4B = 0100 1011 */

    printf("== the byte to send ==\n");
    printf("  '%c' = 0x%02X = binary %c%c%c%c%c%c%c%c (most significant first)\n\n", byte, byte,
           "01"[byte >> 7 & 1], "01"[byte >> 6 & 1], "01"[byte >> 5 & 1], "01"[byte >> 4 & 1],
           "01"[byte >> 3 & 1], "01"[byte >> 2 & 1], "01"[byte >> 1 & 1], "01"[byte & 1]);

    size_t n = make_frame(w, byte, 0);
    printf("== the shape on the wire (8N1) --- low=_ high=- ==\n");
    show_wave(w, n);
    printf("  * data goes least significant bit first, so it looks reversed to the eye.\n\n");

    printf("== timing ==\n");
    printf("  %-10s %-14s %-14s %s\n", "baud", "one bit", "one frame (10 bits)", "bytes per second");
    const long bauds[] = { 300, 9600, 19200, 115200, 921600 };
    for (unsigned i = 0; i < sizeof bauds / sizeof *bauds; i++) {
        double bit_us = 1e6 / (double)bauds[i];
        printf("  %-10ld %-14.3f %-14.1f %.0f\n", bauds[i], bit_us, bit_us * 10,
               (double)bauds[i] / 10.0);
    }
    printf("  (microseconds. 8N1 spends 10 bits on one byte, so 20%% is overhead)\n\n");

    printf("== when the receiver's baud rate is off ==\n");
    printf("  the receiver samples the middle using its own bit length; the error shifts it.\n\n");
    printf("  %-12s %-10s %-8s %-8s %s\n", "rx baud", "error", "value read", "stop bit", "result");
    struct { const char *name; double factor; } rx[] = {
        { "9600",   1.00 }, { "9700",   9600.0 / 9700 }, { "9900", 9600.0 / 9900 },
        { "10100",  9600.0 / 10100 }, { "10600", 9600.0 / 10600 }, { "19200", 9600.0 / 19200 },
    };
    for (unsigned i = 0; i < sizeof rx / sizeof *rx; i++) {
        uint8_t got; int stop_ok;
        double rx_bit = OS * rx[i].factor;
        if (receive(w, n, rx_bit, &got, &stop_ok) != 0) { printf("  %-12s could not read\n", rx[i].name); continue; }
        double err = (1.0 / rx[i].factor - 1.0) * 100.0;
        printf("  %-12s %+7.1f%%  0x%02X     %-8s %s\n", rx[i].name, err, got,
               stop_ok ? "ok" : "broken",
               got == byte && stop_ok ? "'K' --- correct"
               : got == byte ? "value right, frame lost"
               : "character corrupted");
    }

    /* 임계점을 *찾아본다* --- 통설을 옮겨 적는 대신 프로그램이 재게 한다 */
    double lo = 0, hi = 0;
    for (double e = -20.0; e <= 20.0; e += 0.05) {
        double rx_bit = OS / (1.0 + e / 100.0);
        uint8_t g; int ok;
        int good = receive(w, n, rx_bit, &g, &ok) == 0 && g == byte && ok;
        if (good && lo == 0 && hi == 0) lo = e;
        if (good) hi = e;
    }
    printf("\n  range of error where the character survives here: %+.1f%% to %+.1f%%\n", lo, hi);
    printf("  * the arithmetic agrees. The stop bit is sampled at 9.5 bit times,\n");
    printf("    and that instant must fall within that bit (one bit wide), so\n");
    printf("    0.5 / 9.5 = %.1f%% is the margin on one side.\n", 0.5 / 9.5 * 100.0);
    printf("  * yet practice uses 2 to 3%% as the rule. This %.1f%% is shared by both ends\n",
           0.5 / 9.5 * 100.0);
    printf("    (the sender drifts too), oscillators move with temperature and supply,\n");
    printf("    and a real receiver samples three points and takes a majority.\n");
    printf("    The theoretical limit and the design budget are different numbers.\n");

    printf("\n== what parity catches ==\n");
    size_t np = make_frame(w, byte, 1);
    uint8_t got; int stop_ok;
    receive(w, np, OS, &got, &stop_ok);
    unsigned ones = 0; for (int b = 0; b < 8; b++) ones += byte >> b & 1;
    printf("  '%c' has %u ones -> even parity bit = %u\n", byte, ones, ones & 1);
    printf("  one flipped bit changes the parity, so it is caught.\n");
    printf("  two flipped bits leave the parity unchanged, so it is not --- parity\n");
    printf("  is a device for noticing an error, not for correcting one.\n");
    return 0;
}

Output

== the byte to send ==
  'K' = 0x4B = binary 01001011 (most significant first)

== the shape on the wire (8N1) --- low=_ high=- ==
  ----__----__--____--__------
      Std0d1d2d3d4d5d6d7Sp    
  * data goes least significant bit first, so it looks reversed to the eye.

== timing ==
  baud       one bit        one frame (10 bits) bytes per second
  300        3333.333       33333.3        30
  9600       104.167        1041.7         960
  19200      52.083         520.8          1920
  115200     8.681          86.8           11520
  921600     1.085          10.9           92160
  (microseconds. 8N1 spends 10 bits on one byte, so 20% is overhead)

== when the receiver's baud rate is off ==
  the receiver samples the middle using its own bit length; the error shifts it.

  rx baud      error      value read stop bit result
  9600            +0.0%  0x4B     ok       'K' --- correct
  9700            +1.0%  0x4B     ok       'K' --- correct
  9900            +3.1%  0x4B     ok       'K' --- correct
  10100           +5.2%  0x4B     ok       'K' --- correct
  10600          +10.4%  0x8B     broken   character corrupted
  19200         +100.0%  0x9E     ok       character corrupted

  range of error where the character survives here: -5.2% to +5.9%
  * the arithmetic agrees. The stop bit is sampled at 9.5 bit times,
    and that instant must fall within that bit (one bit wide), so
    0.5 / 9.5 = 5.3% is the margin on one side.
  * yet practice uses 2 to 3% as the rule. This 5.3% is shared by both ends
    (the sender drifts too), oscillators move with temperature and supply,
    and a real receiver samples three points and takes a majority.
    The theoretical limit and the design budget are different numbers.

== what parity catches ==
  'K' has 4 ones -> even parity bit = 0
  one flipped bit changes the parity, so it is caught.
  two flipped bits leave the parity unchanged, so it is not --- parity
  is a device for noticing an error, not for correcting one.

The demonstration shows four things.

First, data goes low-order bit first. Which is why reading the shape on the wire makes the binary number look reversed.

Second, 8N1 spends ten bits on one byte. So 9600 baud is 960 bytes per second — 20% of it is overhead holding the frame together.

Third, a baud mismatch accumulates. The clock is synced at the start bit, but by the tenth bit the error has grown nine and a half times. The demonstration searched for the threshold itself and answered about ±5%, and the arithmetic agrees: 0.5 / 9.5 = 5.3%.

Fourth, and yet the rule in practice is “2 to 3%”. That these two numbers differ matters — the theoretical limit is shared between the two ends, oscillators drift with temperature and supply, and a real receiver samples several points and takes a majority. In the words of chapter 53, a contract is not the limit but a budget with margin left.

Q. Why such awkward numbers as 9600, 19200, 115200?

A. The crystal old UART chips used was 1.8432 MHz, and dividing it by integers gives these numbers (1.8432 MHz ÷ 16 ÷ 12 = 9600). What became standard was not “a round decimal” but a number an integer divider can make. Which is why some clocks cannot produce exactly 115200, and the error that follows eats into the budget above.

RS-232 — the same frame, different voltages#

A UART is logic; RS-232 is the electrical specification for putting that logic on a wire. Mixing the two burns hardware often enough to deserve a table.

TTL/CMOS UARTRS-232
1 (mark)3.3 V or 5 V−3 V to −15 V
0 (space)0 V+3 V to +15 V
logichigh is 1inverted — negative is 1
reachcentimetres on a boardtens of metres
wired together directly—the voltages differ and the chip is damaged — a converter must sit between

Table 105.6 — TTL-level UART and RS-232

★ The last row is where hardware gets destroyed in practice. A computer’s serial port must not go straight onto a microcontroller’s UART pin — something like a MAX232 has to be in between.

NameDirectionWhatToday
TxDoutdata being sentalways used
RxDindata being receivedalways used
GND—the reference voltage — without it nothing worksalways used
RTS / CTSout / in“may I send” — hardware flow controlused on fast links
DTR / DSRout / in“I am powered” — device readinessa trace of the modem era
DCDinthe modem has a connectionalmost never used
RIinthe phone is ringingalmost never used

Table 105.7 — The signals of RS-232 (on a 9-pin connector)

Flow control — how to say “wait”#

If the receiving side cannot keep up, data simply disappears. A UART has no retransmission.

KindHowGoodBad
hardware (RTS/CTS)a separate line is lowered to say “stop”it does not touch the data, and stops at oncetwo more wires are needed
software (XON/XOFF)0x13 and 0x11 are inserted into the data streamthree wires sufficebinary data cannot be sent — those bytes are misread

Table 105.8 — Two kinds of flow control

Counter-example. sending binary data over an XON/XOFF line

A 0x13 byte in the middle of the data is read by the other end as “stop”. The transfer freezes then and there, and may never resume. On a line that will carry binary data, use hardware flow control or encode the data as text.

The four errors a UART reports#

NameWhenWhat it usually means
framing errorthe stop bit position was lowthe baud rates differ, or the line was disturbed
parity errorthe count of ones does not match the agreementnoise — one bit flipped
overrunthe next byte arrived before the last was readmy code was late — an interrupt or DMA is needed
breakthe line stayed low for longer than a framea deliberate signal from the other end, or a wire broken to ground

Table 105.9 — Errors met in a UART’s status register

★ The third row joins on to the end of this appendix. An overrun is a software problem, not a hardware one — when taking an interrupt per byte becomes too much, DMA (direct memory access) is what is needed.

Why parallel disappeared#

Sending eight bits at once looks eight times faster. For a while it was. The Centronics port that connected printers worked that way.

GroupLinesWhatDirection
dataD0 to D7eight bits at onceout
controlSTROBE“the data is valid now” — effectively the clockout
statusACK“received”in
statusBUSY“still working, wait”in
statusPAPER OUT, SELECT, ERRORthe printer’s circumstancesin
referenceGND ×8a ground paired with each data line to reduce noise—

Table 105.10 — The signals of a parallel (Centronics) port

Why it vanished despite being fast is in Figure 105.1. The requirement that all eight lines arrive at the same instant becomes the wall.

ProblemWhat it isAs speed rises
skewlines differ in length and load, so arrival times spreadonce one bit’s time is shorter than the spread, they cannot be lined up
crosstalka change on one line leaks into its neighbourthe faster the change, the worse it is
pins and costeight lines means eight of everything: connector, board, shieldingexpensive, thick and unbendable
terminationeach line’s impedance must be matched or it reflectsbeyond a few hundred MHz it is effectively impossible

Table 105.11 — Why parallel loses as things get faster

★ So today’s answer is not “parallel was abandoned” but “make one serial line very fast, and bundle several of them”. PCIe’s lanes, USB 3′s several pairs and SATA all have that shape. Lining the lines up is solved by hardware recovering each line separately and then reassembling — possible because each line carries its own clock inside its coding.

Many devices on two wires — I2C#

The scheme for when wires must be saved above all. Two wires (data SDA, clock SCL) carry several devices, chosen by address.

NameWhatHow it is madeWhy so
SCLthe clockthe controller drives itsynchronous — no need to agree a speed in advance
SDAdatathe two sides use it by turnshalf duplex — one direction at a time
START“a transaction begins”SDA is pulled low while SCL is higha deliberate violation of “data changes only while SCL is low”, which is what makes it a marker
STOP“it ends”SDA is released high while SCL is highlikewise
repeated START“do not end, only turn round”another START without a STOPno other controller can cut in meanwhile
ACK / NACK“received” / “absent, or enough”on the ninth clock the receiver pulls SDA low, or does notpossible precisely because a line can only be pulled down

Table 105.12 — The signals and markers of I2C

examples-en/apx-links/i2c_frame.c

/* I2C 거래 하나를 신호 차례로 짓고, 그 차례를 다시 읽어 해독한다.
   선은 둘뿐이다: SCL(클록)과 SDA(자료). 둘 다 「끌어내리기만」 할 수 있다(오픈 드레인). */
#include <stdio.h>
#include <stdint.h>
#include <string.h>

enum { EV_START, EV_BIT, EV_ACK, EV_NACK, EV_RSTART, EV_STOP };

struct ev { int kind; int val; const char *note; };

static struct ev log_[256];
static int n_ev;

static void put(int kind, int val, const char *note)
{ log_[n_ev++] = (struct ev){ kind, val, note }; }

/* 바이트 하나 --- 높은 자리부터 여덟 비트, 그다음 아홉 번째 클록이 ACK 자리 */
static void put_byte(uint8_t b, const char *what, int acked)
{
    for (int i = 7; i >= 0; i--) put(EV_BIT, b >> i & 1, i == 7 ? what : NULL);
    put(acked ? EV_ACK : EV_NACK, acked ? 0 : 1, NULL);
}

static const char *kind_name(int k)
{
    switch (k) {
    case EV_START:  return "START";
    case EV_RSTART: return "repeated START";
    case EV_STOP:   return "STOP";
    case EV_ACK:    return "ACK";
    case EV_NACK:   return "NACK";
    default:        return "bits";
    }
}

int main(void)
{
    const uint8_t dev = 0x3C;      /* 7비트 장치 주소 */
    const uint8_t reg = 0x00, val = 0xAF;

    printf("== why the address is confusing ==\n");
    printf("  the 7-bit address 0x%02X is shifted one place on the wire (last bit is read/write).\n", dev);
    printf("    write: 0x%02X << 1 | 0 = 0x%02X\n", dev, dev << 1);
    printf("    read : 0x%02X << 1 | 1 = 0x%02X\n", dev, dev << 1 | 1);
    printf("  so datasheets call the same device 0x%02X in one place and 0x%02X in another.\n\n",
           dev, dev << 1);

    /* ── 쓰기 거래: 장치에게 「레지스터 0 에 0xAF 를 써라」 ── */
    put(EV_START, 0, "the controller takes the bus");
    put_byte((uint8_t)(dev << 1 | 0), "device address + write", 1);
    put_byte(reg, "register number", 1);
    put_byte(val, "value to write", 1);
    put(EV_STOP, 0, "release the bus");

    /* ── 읽기 거래: 반복 START 로 방향만 바꾼다 ── */
    put(EV_START, 0, "take it again");
    put_byte((uint8_t)(dev << 1 | 0), "device address + write", 1);
    put_byte(reg, "say which register to read first", 1);
    put(EV_RSTART, 0, "* START again without STOP --- nobody can cut in meanwhile");
    put_byte((uint8_t)(dev << 1 | 1), "device address + read", 1);
    put_byte(val, "the value the device returned", 0);   /* 마지막 바이트는 주인이 NACK 로 끝을 알린다 */
    put(EV_STOP, 0, "end");

    printf("== the order on the wire ==\n");
    int bitpos = 0; uint8_t acc = 0;
    for (int i = 0; i < n_ev; i++) {
        struct ev *e = &log_[i];
        if (e->kind == EV_BIT) {
            acc = (uint8_t)(acc << 1 | e->val);
            if (++bitpos == 8) {
                printf("  %-12s 0x%02X  %s\n", "byte", acc,
                       log_[i - 7].note ? log_[i - 7].note : "");
                bitpos = 0; acc = 0;
            }
        } else {
            printf("  %-12s %s%s\n", kind_name(e->kind),
                   e->kind == EV_ACK ? "the device pulled SDA down = received" :
                   e->kind == EV_NACK ? "nobody pulled it down = absent, or done" : "",
                   e->note ? e->note : "");
        }
    }

    printf("\n== what means what ==\n");
    printf("  START : SDA falls while SCL is high   (deliberately breaking the rule that\n");
    printf("  STOP  : SDA rises while SCL is high    data changes only while SCL is low)\n");
    printf("  ACK   : on the ninth clock the receiver pulls SDA down\n");
    printf("  NACK  : nobody pulls it down and the line stays high --- absent and enough look alike\n");

    printf("\n== timing ==\n");
    int bits = 0, extra = 0;
    for (int i = 0; i < n_ev; i++)
        (log_[i].kind == EV_BIT) ? bits++ : (log_[i].kind == EV_ACK || log_[i].kind == EV_NACK) ? bits++ : extra++;
    printf("  clocks in these two transactions: %d (data and ACK bits), %d markers\n", bits, extra);
    printf("  %-14s %-14s %s\n", "speed mode", "clock", "time for these two transactions");
    struct { const char *name; double hz; } modes[] = {
        { "standard", 100e3 }, { "fast", 400e3 }, { "fast plus", 1e6 }, { "high speed", 3.4e6 },
    };
    for (unsigned i = 0; i < sizeof modes / sizeof *modes; i++)
        printf("  %-14s %-14.1f %.1f microseconds\n", modes[i].name, modes[i].hz / 1000,
               bits / modes[i].hz * 1e6);
    printf("  (clock in kHz; markers and wait states are not counted)\n");

    printf("\n== why the lines are only ever pulled down ==\n");
    printf("  if two devices speak at once, one pushing 5V and one pushing 0V, current\n");
    printf("  flows straight through and damages the chips. I2C lets nobody push (open drain)\n");
    printf("  and leaves the raising to a resistor. So speaking at once is safe,\n");
    printf("  and the property that zero wins gives arbitration and ACK for free.\n");
    return 0;
}

Output

== why the address is confusing ==
  the 7-bit address 0x3C is shifted one place on the wire (last bit is read/write).
    write: 0x3C << 1 | 0 = 0x78
    read : 0x3C << 1 | 1 = 0x79
  so datasheets call the same device 0x3C in one place and 0x78 in another.

== the order on the wire ==
  START        the controller takes the bus
  byte         0x78  device address + write
  ACK          the device pulled SDA down = received
  byte         0x00  register number
  ACK          the device pulled SDA down = received
  byte         0xAF  value to write
  ACK          the device pulled SDA down = received
  STOP         release the bus
  START        take it again
  byte         0x78  device address + write
  ACK          the device pulled SDA down = received
  byte         0x00  say which register to read first
  ACK          the device pulled SDA down = received
  repeated START * START again without STOP --- nobody can cut in meanwhile
  byte         0x79  device address + read
  ACK          the device pulled SDA down = received
  byte         0xAF  the value the device returned
  NACK         nobody pulled it down = absent, or done
  STOP         end

== what means what ==
  START : SDA falls while SCL is high   (deliberately breaking the rule that
  STOP  : SDA rises while SCL is high    data changes only while SCL is low)
  ACK   : on the ninth clock the receiver pulls SDA down
  NACK  : nobody pulls it down and the line stays high --- absent and enough look alike

== timing ==
  clocks in these two transactions: 63 (data and ACK bits), 5 markers
  speed mode     clock          time for these two transactions
  standard       100.0          630.0 microseconds
  fast           400.0          157.5 microseconds
  fast plus      1000.0         63.0 microseconds
  high speed     3400.0         18.5 microseconds
  (clock in kHz; markers and wait states are not counted)

== why the lines are only ever pulled down ==
  if two devices speak at once, one pushing 5V and one pushing 0V, current
  flows straight through and damages the chips. I2C lets nobody push (open drain)
  and leaves the raising to a resistor. So speaking at once is safe,
  and the property that zero wins gives arbitration and ACK for free.

Three things are immediately useful in practice.

First, why an address is written two ways. A 7-bit address is shifted one place left on the wire with the read/write bit appended. So the same device appears in datasheets as 0x3C and as 0x78. Both are right — one is the address, the other is “the first byte on the wire”.

Second, NACK means two things. “No such device” and “stop sending now” look the same on the wire. The convention is that the controller NACKs the last byte of a read to say it is done.

Third, a design where lines are only ever pulled down makes much of this free. Nobody pushes a voltage up, so two speaking at once damages no chip, and from the property “zero wins” both ACK and arbitration simply fall out. In exchange the pull-up is done by a resistor, so rising is slow, and that sets I2C’s speed ceiling.

NameClockBytes per second (roughly)Note
standard100 kHzabout 11,000most sensors
fast400 kHzabout 44,000today’s default
fast plus1 MHzabout 110,000the wiring must be short
high speed3.4 MHzabout 380,000needs dedicated circuitry — uncommon

Table 105.13 — I2C speed modes

Q. I attached several I2C devices and none is found. What do I look at first?

A. Four things in order. ① Are the pull-up resistors there (without them the line never rises)? ② Do the addresses collide (two sensors of the same kind usually do — change one with its address pin)? ③ Is the ground shared? ④ Is the wiring so long, or the device count so high, that the rise time is too slow? Three of the four are electrical problems and only one is software.

Fast on four wires — SPI#

Where I2C gave up speed to save wires, SPI does the reverse: four wires, and fast.

NameOther namesWhatDirection
SCKSCLK, CLKthe clock — the controller drives itcontroller → device
MOSISDO, COPIdata the controller sendscontroller → device
MISOSDI, CIPOdata the device sendsdevice → controller
CSSS, NSS, CEthe marker meaning I am speaking to this device (usually active low)controller → device

Table 105.14 — The four wires of SPI

★ There is no address. Instead one CS line is run to each device. More devices means more wires — exactly the opposite trade from I2C.

examples-en/apx-links/spi_shift.c

/* SPI 는 「고리 모양으로 이어 붙인 시프트 레지스터 둘」이다.
   클록마다 한 비트씩 서로 밀어 넣는다 --- 그래서 주고받기가 *동시에* 일어난다. */
#include <stdio.h>
#include <stdint.h>

static void bits8(uint8_t v, char *out)
{ for (int i = 0; i < 8; i++) out[i] = (char)('0' + (v >> (7 - i) & 1)); out[8] = 0; }

int main(void)
{
    uint8_t m = 0xA5;      /* 주인이 보낼 값 */
    uint8_t s = 0x3C;      /* 장치가 보낼 값 (미리 제 레지스터에 넣어 둔다) */
    char mb[9], sb[9];

    bits8(m, mb); bits8(s, sb);
    printf("== at the start ==\n");
    printf("  controller register : 0x%02X (%s)\n", m, mb);
    printf("  device register     : 0x%02X (%s)\n\n", s, sb);

    printf("== one bit per clock ==\n");
    printf("  %-6s %-6s %-6s %-12s %-12s\n", "clock", "MOSI", "MISO", "controller", "device");
    for (int c = 1; c <= 8; c++) {
        unsigned mosi = m >> 7 & 1;          /* 주인이 내놓는 비트 (높은 자리부터) */
        unsigned miso = s >> 7 & 1;          /* 장치가 내놓는 비트 */
        m = (uint8_t)(m << 1 | miso);        /* 서로의 비트를 낮은 자리로 받아 넣는다 */
        s = (uint8_t)(s << 1 | mosi);
        bits8(m, mb); bits8(s, sb);
        printf("  %-6d %-6u %-6u %-12s %-12s\n", c, mosi, miso, mb, sb);
    }

    printf("\n== after eight clocks ==\n");
    printf("  controller register : 0x%02X  <- what the device sent\n", m);
    printf("  device register     : 0x%02X  <- what the controller sent\n", s);
    printf("  * sending and receiving finished in the same eight clocks. Full duplex is\n");
    printf("    free because the two registers form one ring. Even with nothing to receive\n");
    printf("    something must be sent (usually 0x00 or 0xFF): a dummy byte.\n");

    printf("\n== the four modes --- which edge presents, which samples ==\n");
    printf("  %-8s %-6s %-6s %-16s %s\n", "mode", "CPOL", "CPHA", "SCK at rest", "sampling edge");
    for (int mode = 0; mode < 4; mode++) {
        int cpol = mode >> 1, cpha = mode & 1;
        printf("  %-8d %-6d %-6d %-16s %s\n", mode, cpol, cpha,
               cpol ? "high" : "low",
               cpha == 0 ? (cpol ? "falling (first edge)" : "rising (first edge)")
                         : (cpol ? "rising (second edge)" : "falling (second edge)"));
    }

    printf("\n== when the mode is mismatched ==\n");
    uint8_t src = 0xA5, wrong = 0, prev = 0;
    for (int c = 0; c < 8; c++) {
        unsigned bit = src >> (7 - c) & 1;
        wrong = (uint8_t)(wrong << 1 | prev);   /* 한 모서리 늦게 읽으면 *직전* 비트를 본다 */
        prev = bit;
    }
    bits8(0xA5, mb); bits8(wrong, sb);
    printf("  sent 0xA5 (%s)\n", mb);
    printf("  read with one edge of error: 0x%02X (%s)  <- everything shifted one place\n", wrong, sb);
    printf("  * so when SPI gives half-right values, suspect the mode first.\n");

    printf("\n== speed ==\n");
    printf("  SPI has no start, stop, address or ACK. Exactly eight clocks per byte.\n");
    printf("  %-14s %-16s %s\n", "SCK", "bytes per second", "compared");
    struct { const char *name; double hz; } sck[] = {
        { "1 MHz", 1e6 }, { "10 MHz", 10e6 }, { "50 MHz", 50e6 },
    };
    const double i2c_std = 100e3 / 9.0;      /* I2C 표준 모드: 바이트마다 9비트 */
    for (unsigned i = 0; i < sizeof sck / sizeof *sck; i++)
        printf("  %-14s %-16.0f %.0f times I2C standard (100 kHz)\n", sck[i].name, sck[i].hz / 8,
               (sck[i].hz / 8) / i2c_std);
    printf("  (UART spends 10 bits per byte, I2C 9, SPI 8 --- no overhead)\n");
    return 0;
}

Output

== at the start ==
  controller register : 0xA5 (10100101)
  device register     : 0x3C (00111100)

== one bit per clock ==
  clock  MOSI   MISO   controller   device      
  1      1      0      01001010     01111001    
  2      0      0      10010100     11110010    
  3      1      1      00101001     11100101    
  4      0      1      01010011     11001010    
  5      0      1      10100111     10010100    
  6      1      1      01001111     00101001    
  7      0      0      10011110     01010010    
  8      1      0      00111100     10100101    

== after eight clocks ==
  controller register : 0x3C  <- what the device sent
  device register     : 0xA5  <- what the controller sent
  * sending and receiving finished in the same eight clocks. Full duplex is
    free because the two registers form one ring. Even with nothing to receive
    something must be sent (usually 0x00 or 0xFF): a dummy byte.

== the four modes --- which edge presents, which samples ==
  mode     CPOL   CPHA   SCK at rest      sampling edge
  0        0      0      low              rising (first edge)
  1        0      1      low              falling (second edge)
  2        1      0      high             falling (first edge)
  3        1      1      high             rising (second edge)

== when the mode is mismatched ==
  sent 0xA5 (10100101)
  read with one edge of error: 0x52 (01010010)  <- everything shifted one place
  * so when SPI gives half-right values, suspect the mode first.

== speed ==
  SPI has no start, stop, address or ACK. Exactly eight clocks per byte.
  SCK            bytes per second compared
  1 MHz          125000           11 times I2C standard (100 kHz)
  10 MHz         1250000          112 times I2C standard (100 kHz)
  50 MHz         6250000          562 times I2C standard (100 kHz)
  (UART spends 10 bits per byte, I2C 9, SPI 8 --- no overhead)

The demonstration shows what SPI really is. Two shift registers form a single ring. Push one bit per clock and after eight clocks the two values have swapped places. So “sending and receiving at the same time” is free — and even to receive only, you must send something (a dummy byte).

ModeCPOLCPHAClock at restSampling edge
000lowrising (the first edge)
101lowfalling (the second edge)
210highfalling (the first edge)
311highrising (the second edge)

Table 105.15 — SPI’s four modes

A common misconception. SPI is a specification, so any device can just be wired up

SPI has no official specification. There are four modes; whether data goes most-significant or least-significant bit first differs by device; whether CS is raised between bytes or held low for a whole transaction differs too. As at the end of the demonstration, one edge of mode mismatch shifts every value by one place — which is why a “half-right” value should make you suspect the mode first.
I2CSPISo
wires2 (unchanged as devices are added)3 + 1 per devicemany devices favour I2C
speed100 kHz to 1 MHztens of MHzneeding speed favours SPI
bits per byte9 (the ACK included)8SPI has no overhead either
directionhalf duplexfull duplexneeding both at once favours SPI
confirmation of receiptthere is an ACKnoneSPI cannot tell that the other end has died
specificationwritten downcustomwith SPI the datasheet must be read

Table 105.16 — I2C against SPI

On one wire, and with anyone speaking — 1-Wire and CAN#

NameWiresThe ideaWhere
1-Wire1 (+ ground)zero and one are told apart by pulse length, and power is stolen from the data linetemperature sensors, identity chips
CAN2 (differential)anyone speaks, and on a collision the lower message number wins (arbitration)cars, industrial machinery

Table 105.17 — Two unusual schemes

★ CAN’s arbitration is the same idea as I2C’s “zero wins”. When two speak at once the signal of the one sending zero remains on the wire, and the one sending a one sees that the wire differs from what it sent and withdraws. It is not collision detection followed by retransmission: the collision itself is the priority decision — which is why an urgent message is never delayed.

The motorway inside the machine — PCI and PCI Express#

So far these were “wires between chips”. Now the scale changes. Where graphics cards, network cards and NVMe storage attach — the PCI family.

The name says “bus”, but today’s PCIe is really point to point. Old PCI really did have several devices sharing the same lines (so one slow device slowed everything), while PCIe gives each device its own lines with a switch joining them in the middle.

old PCIPCIeSo
connectionseveral devices share one buspoint to point per device, plus switchesone device cannot hold up the others
wiresparallel 32/64 bits + clockserial differential lanes ×1 to ×16exactly the conclusion of Figure 105.1
going fasterraise the clock (33 → 66 MHz)add lanes or raise the generationthe direction of growth differs
interruptsshared INTA to INTD linesMSI/MSI-X — sent as a memory writeit becomes data rather than a wire

Table 105.18 — PCI and PCI Express

GenerationOn the wireCodingEffective bandwidth
1.02.5 GT/s8b/10b (20% lost)about 250 MB/s
2.05 GT/s8b/10babout 500 MB/s
3.08 GT/s128b/130b (1.5% lost)about 985 MB/s
4.016 GT/s128b/130babout 1.97 GB/s
5.032 GT/s128b/130babout 3.94 GB/s

Table 105.19 — PCIe generations and effective speed (per lane)

★ The “coding” column matters for a reason. There is no separate clock line, so the clock must be planted in the data (otherwise a long run of zeros loses the receiver’s timing), and what is planted is the loss. Dropping 8b/10b for 128b/130b at 3.0 is why “1.6 times the speed” became “twice the bandwidth”.

Configuration space — the 256 bytes in which a device introduces itself#

PCI’s best idea is this. Every device carries a business card of the same shape. So the operating system can find out what is plugged in without knowing in advance.

OffsetSizeNameWhatTypical value
0x002vendor IDa number handed out by PCI-SIG0x8086 (Intel)
0x022device IDa model number chosen by the vendor—
0x042commandthe switch that turns this device on — I/O, memory, bus master0x0007
0x062statuserrors, and a flag saying “there is a capability list”0x0010
0x081revisionthe revision of the same model—
0x09–0x0B3class codewhat kind of device this is — known without a driver02 00 00 = Ethernet
0x0C1cache line sizea trace of old PCI0x10
0x0E1header type0 device, 1 bridge, 0x80 means multiple functions0x00
0x10–0x2424BAR 0 to 5the address windows this device requests (Table 105.21)—
0x2C–0x2E4subsystem IDthe company and model that built the board—
0x304expansion ROMwhere firmware on the card lives—
0x341capability pointerthe offset of the first item of a linked list0x50
0x3C1interrupt linewhich IRQ it was wired to in the old scheme11
0x3D1interrupt pinwhich of INTA to INTD1 (A)

Table 105.20 — The main fields of PCI configuration space (header type 0)

BitsValueMeaningNote
00memory spaceusually this — a pointer reaches it
01I/O spacex86′s narrow 64 KiB world
2–10032-bit address
2–11064-bit addressthe next BAR holds the upper 32 bits — it eats two slots
31prefetchablereading has no side effect, so caching and burst transfers are allowed
restaddressthe base addressthe low bits are hardwired to zero for the size

Table 105.21 — The low bits of a BAR say what kind it is

examples-en/apx-links/pci_config.c

/* PCI 설정 공간(256바이트)을 필드대로 짓고 되읽는다. 그리고 BAR 의 *크기를 알아내는*
   고전적인 수법 --- 전부 1을 써 보고 되읽기 --- 을 장치 흉내로 재현한다. */
#include <stdio.h>
#include <stdint.h>
#include <string.h>

static unsigned char cfg[256];
/* 각 BAR 가 실제로 요구하는 크기. 장치가 하드웨어로 정해 두는 값이다. */
static uint32_t bar_size[6] = { 16u << 20, 0, 256u << 20, 0, 0, 256 };
static int bar_is_io[6]     = { 0, 0, 0, 0, 0, 1 };
static int bar_is_64[6]     = { 0, 0, 1, 0, 0, 0 };   /* BAR2+3 이 짝을 이룬다 */

static uint32_t rd32(unsigned off)
{ uint32_t v = 0; for (int i = 3; i >= 0; i--) v = v << 8 | cfg[off + i]; return v; }
static void wr32(unsigned off, uint32_t v)
{ for (int i = 0; i < 4; i++) cfg[off + i] = (unsigned char)(v >> (8 * i)); }
static uint16_t rd16(unsigned off) { return (uint16_t)(cfg[off] | cfg[off + 1] << 8); }

/* 장치 흉내: BAR 에 값을 쓰면, 크기보다 낮은 비트는 장치가 *0 으로 굳혀* 돌려준다. */
static void bar_write(int i, uint32_t v)
{
    unsigned off = 0x10 + 4u * (unsigned)i;
    uint32_t low = bar_is_io[i] ? 1u : (uint32_t)(bar_is_64[i] ? 0x4 : 0x0);
    uint32_t mask = ~(bar_size[i] - 1);
    wr32(off, (v & mask) | low);
}

int main(void)
{
    /* ── 머리말 필드 ── */
    wr32(0x00, 0x10FA8086u);           /* 0x00 제조사 0x8086, 0x02 장치 0x10FA */
    wr32(0x04, 0x00100007u);           /* 0x04 명령: 메모리·IO·버스마스터 켬 / 0x06 상태 */
    wr32(0x08, 0x02000003u);           /* 0x08 개정 03, 0x09~0x0B 분류: 02 00 00 = 이더넷 */
    wr32(0x0C, 0x00000010u);           /* 0x0C 캐시줄 16, 0x0E 머리말 종류 0 */
    for (int i = 0; i < 6; i++)
        if (bar_size[i])
            /* 입출력 공간은 64 KiB 짜리 좁은 세계라 주소도 작다(옛 x86 의 흔적) */
            bar_write(i, bar_is_io[i] ? 0xC000u : 0xF0000000u + (uint32_t)i * 0x1000000u);
    wr32(0x2C, 0x00108086u);           /* 하위 시스템 */
    cfg[0x34] = 0x50;                  /* 능력 목록의 첫 자리 */
    cfg[0x3C] = 11;                    /* 인터럽트 선 (옛 방식) */
    cfg[0x3D] = 1;                     /* 인터럽트 핀 A */
    /* 능력 목록: MSI(0x05) → PCIe(0x10) → 끝 */
    cfg[0x50] = 0x05; cfg[0x51] = 0x60;
    cfg[0x60] = 0x10; cfg[0x61] = 0x00;

    printf("== configuration space header (type 0) ==\n");
    printf("  %-8s %-6s %-18s %s\n", "offset", "size", "name", "value");
    printf("  0x00     2      vendor ID          0x%04X%s\n", rd16(0x00),
           rd16(0x00) == 0x8086 ? " (Intel)" : "");
    printf("  0x02     2      device ID          0x%04X\n", rd16(0x02));
    printf("  0x04     2      command            0x%04X  [IO %s · memory %s · bus master %s]\n",
           rd16(0x04), rd16(0x04) & 1 ? "on" : "off", rd16(0x04) & 2 ? "on" : "off",
           rd16(0x04) & 4 ? "on" : "off");
    printf("  0x06     2      status             0x%04X  [capability list %s]\n", rd16(0x06),
           rd16(0x06) & 0x10 ? "present" : "absent");
    printf("  0x08     1      revision           0x%02X\n", cfg[0x08]);
    printf("  0x09-0B  3      class              %02X %02X %02X = %s\n",
           cfg[0x0B], cfg[0x0A], cfg[0x09],
           cfg[0x0B] == 0x02 ? "network controller (Ethernet)" : "other");
    printf("  0x0E     1      header type        0x%02X  (0=device, 1=bridge)\n", cfg[0x0E]);
    printf("  0x34     1      capability pointer 0x%02X\n", cfg[0x34]);
    printf("  0x3C-3D  2      interrupt line/pin %u / %c\n\n", cfg[0x3C], 'A' + cfg[0x3D] - 1);

    printf("== decoding the BARs ==\n");
    printf("  %-6s %-14s %-12s %-10s %s\n", "BAR", "raw value", "kind", "note", "base address");
    for (int i = 0; i < 6; i++) {
        uint32_t v = rd32(0x10 + 4u * (unsigned)i);
        if (v == 0) { printf("  BAR%-3d (unused)\n", i); continue; }
        if (v & 1) {
            printf("  BAR%-3d 0x%08X     %-12s %-10s 0x%08X\n", i, v, "I/O space", "---",
                   v & ~0x3u);
        } else {
            const char *w = ((v >> 1) & 3) == 2 ? "64-bit" : "32-bit";
            printf("  BAR%-3d 0x%08X     %-12s %-10s 0x%08X\n", i, v, "memory space",
                   w, v & ~0xFu);
            if (((v >> 1) & 3) == 2) { printf("  BAR%-3d (upper 32 bits of the BAR above)\n", i + 1); i++; }
        }
    }

    printf("\n== how the size is found --- write all ones and read back ==\n");
    printf("  a device holds the bits below its requested size at zero.\n");
    printf("  so write all ones and the zeros read back tell you the size.\n\n");
    for (int i = 0; i < 6; i++) {
        if (!bar_size[i]) continue;
        unsigned off = 0x10 + 4u * (unsigned)i;
        uint32_t saved = rd32(off);
        bar_write(i, 0xFFFFFFFFu);                 /* ① 전부 1을 쓴다 */
        uint32_t probe = rd32(off);                /* ② 되읽는다 */
        uint32_t mask = probe & (bar_is_io[i] ? ~0x3u : ~0xFu);
        uint32_t size = ~mask + 1;                 /* ③ 뒤집고 1을 더하면 크기 */
        bar_write(i, saved);                       /* ④ 원래 값을 되돌려 놓는다 */
        printf("  BAR%d: read back 0x%08X -> size %u bytes (%s)  %s\n", i, probe, size,
               size >= (1u << 20) ? "in MiB" : size >= 1024 ? "in KiB" : "bytes",
               size == bar_size[i] ? "matches what the device asked for" : "does not match");
    }
    printf("\n  * these four steps are what real firmware and kernels do. Forget the last one\n");
    printf("    (restoring) and the device lands at the wrong address and the machine hangs.\n");

    printf("\n== walking the capability list ==\n");
    unsigned p = cfg[0x34];
    while (p) {
        unsigned id = cfg[p], next = cfg[p + 1];
        printf("  0x%02X: capability 0x%02X (%s) -> next 0x%02X\n", p, id,
               id == 0x05 ? "MSI --- an interrupt sent as a memory write" :
               id == 0x10 ? "PCI Express" : "other", next);
        p = next;
    }
    printf("  * it is a linked list. Zero means the end --- following the chain tells you\n");
    printf("    what this device knows how to do.\n");
    return 0;
}

Output

== configuration space header (type 0) ==
  offset   size   name               value
  0x00     2      vendor ID          0x8086 (Intel)
  0x02     2      device ID          0x10FA
  0x04     2      command            0x0007  [IO on · memory on · bus master on]
  0x06     2      status             0x0010  [capability list present]
  0x08     1      revision           0x03
  0x09-0B  3      class              02 00 00 = network controller (Ethernet)
  0x0E     1      header type        0x00  (0=device, 1=bridge)
  0x34     1      capability pointer 0x50
  0x3C-3D  2      interrupt line/pin 11 / A

== decoding the BARs ==
  BAR    raw value      kind         note       base address
  BAR0   0xF0000000     memory space 32-bit     0xF0000000
  BAR1   (unused)
  BAR2   0xF0000004     memory space 64-bit     0xF0000000
  BAR3   (upper 32 bits of the BAR above)
  BAR4   (unused)
  BAR5   0x0000C001     I/O space    ---        0x0000C000

== how the size is found --- write all ones and read back ==
  a device holds the bits below its requested size at zero.
  so write all ones and the zeros read back tell you the size.

  BAR0: read back 0xFF000000 -> size 16777216 bytes (in MiB)  matches what the device asked for
  BAR2: read back 0xF0000004 -> size 268435456 bytes (in MiB)  matches what the device asked for
  BAR5: read back 0xFFFFFF01 -> size 256 bytes (bytes)  matches what the device asked for

  * these four steps are what real firmware and kernels do. Forget the last one
    (restoring) and the device lands at the wrong address and the machine hangs.

== walking the capability list ==
  0x50: capability 0x05 (MSI --- an interrupt sent as a memory write) -> next 0x60
  0x60: capability 0x10 (PCI Express) -> next 0x00
  * it is a linked list. Zero means the end --- following the chain tells you
    what this device knows how to do.

The middle of the demonstration is the cleverest part of this specification. There is no field in which to ask a device its size. Instead you write all ones into a BAR and read it back, and the low bits the device holds at zero tell you the size. Four steps — write, read back, invert and add one, restore the original value — are the heart of enumeration, and leaving out the last step hangs the machine.

Q. What is lspci actually showing?

A. It reads this business card (the configuration space) and renders it in human words. The vendor and device IDs are only numbers; the names are looked up in a list file called pci.ids. Which is why a new device in an old system shows up as “Unknown device 8086:10fa” — the device is not broken, the list of names is stale.

The motorway going out — USB#

USB’s design principle is one sentence. The host commands everything. A device cannot speak on its own; it answers only when the host asks.

NameGenerationOn the wireUsed for
Low Speed1.x1.5 Mbit/smice and keyboards
Full Speed1.x12 Mbit/saudio and simple devices
High Speed2.0480 Mbit/sstorage and cameras
SuperSpeed3.x5 to 20 Gbit/sexternal SSDs and displays

Table 105.22 — USB speed grades

NameWhat it guaranteesUsed forNote
controldeliveryrecognising and configuring a deviceevery device must have it
bulkdelivery, but not whenstorage and printersit uses whatever bandwidth is left
interruptdelivery within a set periodmice and keyboardsdespite the name it is not an interrupt — the host asks periodically
isochronoustiming guaranteed, delivery notaudio and videolate data is useless, so it is not retransmitted

Table 105.23 — USB’s four transfer types

★ The third row is the most confusing word in this whole appendix. USB’s “interrupt transfer” is not a device interrupting. The host asks “anything to say?” every millisecond or so, and the device answers only then. The real interrupt is the IRQ of the next section.

A common misconception. plug in a USB device and the device speaks to the computer

The device only signals electrically that it is attached (a voltage on the lines changes). Everything after that is commanded by the host — assigning an address, asking for the business card (the descriptors), choosing a configuration. So a USB device can do nothing by itself. That simple rule is what made cheap devices possible, at the price of complexity on the host side.

Interruption — IRQs#

So far it was “how things are moved”. Now it is “who speaks when”. The line by which a device tells the CPU “something happened to me” is the IRQ (interrupt request).

Where chapter 81 and the appendix on C without an operating system covered what may be done inside a handler, here we look at the stage before it — the wire and its value.

edgelevelSo
what is the signalthe instant the line changeswhile the line is held low (or high)
can it be missedyes — miss the instant and it is goneno — it is still therelevel is safer
how it is clearedit clears itselfa device register must be read to clear itnot clearing it raises it endlessly
sharing between deviceshard — two at once look like oneeasy — ask who is still holding ita shared IRQ is properly level triggered

Table 105.24 — Edge triggering and level triggering

★ The third row is the trap seen in the appendix on C without an operating system. With level triggering, a handler that does not read the device’s status register leaves the request asserted, and the program spins inside the handler forever. And the word that stops the compiler removing that read is volatile.

SchemeHowDevice countProblem
a dedicated lineone IRQ line per deviceas many as there are linespins run out
a shared lineseveral devices share one lineseveraleveryone must be asked who raised it
MSIan interrupt becomes writing a value to an agreed addresseffectively unlimitedno wire, so no sharing problem
MSI-XMSI extended, a different number per queuethousandseach queue can be routed to a different core

Table 105.25 — How interrupt delivery changed

Q. How can “writing a value to memory” replace an interrupt?

A. PCIe already has a path that carries data. So agree that when a device writes a set value to “a special address”, the interrupt controller catches that write and wakes the CPU. The real gain is not saving a wire but being able to number each interrupt — so “which queue filled up” is known without asking.

examples-en/apx-links/signal_cost.c

/* 「끼어들기에는 값이 있다」를 이 기계에서 *실제로 재어* 본다.
   ★ 주의: 이것은 하드웨어 인터럽트가 아니라 운영체제의 신호다. 자릿수의 감각을 얻는
   용도이고, 하드웨어 IRQ 지연과 같은 수가 아니다(그쪽이 대개 더 짧다). */
#define _POSIX_C_SOURCE 200809L
#include <stdio.h>
#include <signal.h>
#include <time.h>
#include <stdint.h>

static volatile sig_atomic_t hits;
static void handler(int sig) { (void)sig; hits++; }

static double now_ns(void)
{
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (double)ts.tv_sec * 1e9 + (double)ts.tv_nsec;
}

static volatile int sink;
static void plain(void) { sink++; }
static void (*volatile plain_p)(void) = plain;

int main(void)
{
    struct sigaction sa = { 0 };
    sa.sa_handler = handler;
    sigaction(SIGUSR1, &sa, NULL);

    const long N = 200000;

    /* 몸풀기 --- 첫 몇 번은 캐시가 차가워 느리다 */
    for (long i = 0; i < 1000; i++) raise(SIGUSR1);

    double t0 = now_ns();
    for (long i = 0; i < N; i++) raise(SIGUSR1);
    double t1 = now_ns();
    double per_signal = (t1 - t0) / (double)N;

    t0 = now_ns();
    for (long i = 0; i < N; i++) plain_p();
    t1 = now_ns();
    double per_call = (t1 - t0) / (double)N;

    printf("== measured on this machine (mean of %ld) ==\n", N);
    printf("  one ordinary function call     : %8.1f ns\n", per_call);
    printf("  raising and handling one signal: %8.1f ns\n", per_signal);
    printf("  ratio                          : %8.1f times\n\n", per_signal / per_call);
    printf("  the handler ran %lld times (a check that none were missed)\n\n",
           (long long)hits);

    printf("== what this number tells you ==\n");
    printf("  that one interruption is not free, and that it costs far more than an\n");
    printf("  ordinary function call --- because each interruption must put the state of\n");
    printf("  the work somewhere, go to the handler, and take it back out again.\n\n");

    printf("  from it, how many per second can be borne:\n");
    printf("  %-16s %-16s %s\n", "events/s", "CPU used", "note");
    const long rates[] = { 1000, 10000, 100000, 1000000 };
    for (unsigned i = 0; i < sizeof rates / sizeof *rates; i++) {
        double busy = per_signal * (double)rates[i] / 1e9 * 100.0;
        char pct[16]; snprintf(pct, sizeof pct, "%.2f%%", busy);
        printf("  %-16ld %-16s %s\n", rates[i], pct,
               busy > 100 ? "impossible --- it would do nothing but handle them"
               : busy > 50 ? "dangerous --- no room for other work"
               : busy > 5  ? "workable, but the margin is thin" : "comfortable");
    }
    printf("\n  * again, this is the cost of an operating system signal. A hardware interrupt\n");
    printf("    is usually far cheaper (a few hundred nanoseconds). But the *shape* of the\n");
    printf("    arithmetic is the same --- learning that shape is the point of this demonstration.\n");
    return 0;
}

Output

== measured on this machine (mean of 200000) ==
  one ordinary function call     :      1.9 ns
  raising and handling one signal:   2122.5 ns
  ratio                          :   1139.0 times

  the handler ran 201000 times (a check that none were missed)

== what this number tells you ==
  that one interruption is not free, and that it costs far more than an
  ordinary function call --- because each interruption must put the state of
  the work somewhere, go to the handler, and take it back out again.

  from it, how many per second can be borne:
  events/s         CPU used         note
  1000             0.21%            comfortable
  10000            2.12%            comfortable
  100000           21.22%           workable, but the margin is thin
  1000000          212.25%          impossible --- it would do nothing but handle them

  * again, this is the cost of an operating system signal. A hardware interrupt
    is usually far cheaper (a few hundred nanoseconds). But the *shape* of the
    arithmetic is the same --- learning that shape is the point of this demonstration.

★ The demonstration above measures operating system signals, not hardware interrupts (and says so itself). A hardware IRQ is usually far cheaper. The number is still useful because the shape of the arithmetic is the same — “cost per event × events per second = CPU occupied”.

Moving it instead — DMA#

If you cannot be interrupted per byte, what then? The answer is “take the CPU out”.

dma-path

Figure 105.2 — The path where the CPU carries the data, and the path where DMA does.

ModeHowTrade
single transferone word at a time, returning the bus to the CPU in betweenthe CPU is blocked less, and it is slower
bursta set number in one gofast, and the CPU waits meanwhile
scatter-gathergiven a list of “4 KiB here, 8 KiB there” it carries them itselfscattered memory in one go — the list itself becomes a data structure

Table 105.26 — How DMA uses the bus

ProblemWhat it isHow it is solved
cache coherenceDMA writes to memory while the CPU looks at the cacheinvalidate before reading, flush before sending. Or an uncached region
kinds of addressthe address the device sees may differ from the one the CPU seesthe driver converts to a “device address” and passes that
security and isolationa device can read and write any memoryan IOMMU limits what each device can see

Table 105.27 — Three things that catch you with DMA

★ The first row joins exactly on to the volatile story in the appendix on C without an operating system. volatile restrains the compiler and does nothing to the cache. Different layers need different devices.

Counter-example. putting a DMA buffer on the stack

Hand a local array on the stack to DMA and two problems compound. If the function returns first, that space becomes space something else is using (chapter 45), and being cached, what the device wrote and what the CPU sees can diverge. A DMA buffer goes in long-lived space, correctly aligned, allocated the way the driver says.

The three compared — polling, interrupts and DMA#

The arithmetic that amounts to this appendix’s conclusion.

examples-en/apx-links/link_budget.c

/* 같은 일을 폴링·인터럽트·DMA 로 할 때 무엇이 얼마나 드는가 --- 예산 계산.
   하드웨어가 없으므로 이것은 *계산*이다. 값(비용)은 흔히 인용되는 자릿수를 가정으로
   두고, 그 가정을 화면에 함께 적는다 --- 가정을 감추면 계산이 아니라 주장이 된다. */
#include <stdio.h>

/* 가정 --- 요즘 흔한 마이크로컨트롤러/PC 급의 어림값 */
#define IRQ_COST_US   1.0     /* 인터럽트 한 번 처리에 드는 시간 */
#define POLL_COST_US  0.2     /* 폴링 한 번(레지스터 읽고 판정)에 드는 시간 */

struct link { const char *name; double bytes_per_sec; };

static void budget(struct link L, double dma_block)
{
    double irq_rate  = L.bytes_per_sec;                  /* 바이트마다 끼어들기 */
    double irq_busy  = irq_rate * IRQ_COST_US / 1e6 * 100.0;
    double dma_rate  = L.bytes_per_sec / dma_block;      /* 덩어리마다 한 번 */
    double dma_busy  = dma_rate * IRQ_COST_US / 1e6 * 100.0;

    char a[24], b[24];
    snprintf(a, sizeof a, "%.1f%%", irq_busy);
    snprintf(b, sizeof b, "%.3f%%", dma_busy);
    printf("  %-22s %12.0f %10s %12.0f %10s\n", L.name, irq_rate, a, dma_rate, b);
}

int main(void)
{
    printf("== assumptions (the values used) ==\n");
    printf("  handling one interrupt    : %.1f microseconds\n", IRQ_COST_US);
    printf("  one poll (read and judge) : %.1f microseconds\n", POLL_COST_US);
    printf("  DMA batch size            : 256 bytes\n");
    printf("  * all three differ by machine. Change them and the conclusion changes --- that is the point.\n\n");

    printf("== an interrupt per byte vs DMA in 256-byte batches ==\n");
    printf("  %-22s %12s %10s %12s %10s\n", "link", "IRQ/s", "CPU", "DMA IRQ/s", "CPU");
    struct link links[] = {
        { "UART 9600 8N1",       960 },
        { "UART 115200 8N1",   11520 },
        { "I2C 400 kHz",       44000 },
        { "SPI 10 MHz",      1250000 },
        { "USB 2.0 (effective)",  30000000 },
        { "1 Gbit Ethernet",   118000000 },
    };
    for (unsigned i = 0; i < sizeof links / sizeof *links; i++) budget(links[i], 256);

    printf("\n  * reading the table downwards shows when DMA becomes necessary.\n");
    printf("    on a slow link an interrupt per byte is no trouble; past some point\n");
    printf("    that scheme leaves the machine doing nothing else.\n");

    printf("\n== where the crossover is (assuming an interrupt per byte) ==\n");
    printf("  %-14s %-16s %s\n", "CPU used", "events/s", "the data rate that means");
    for (double busy = 5; busy <= 100; busy *= 2) {
        double rate = busy / 100.0 * 1e6 / IRQ_COST_US;   /* 사건/초 */
        char pct[16]; snprintf(pct, sizeof pct, "%.0f%%", busy);
        printf("  %-14s %-16.0f %.1f KiB/s\n", pct, rate, rate / 1024.0);
    }

    /* 한글은 한 글자가 두 칸이라 %-Ns 로는 안 맞는다 --- 줄마다 손으로 적는다 */
    printf("\n== the three schemes compared ==\n");
    printf("  polling   : the CPU keeps asking.      latency = about half the period.\n");
    printf("              suits very frequent events, or very simple places.\n");
    printf("  interrupt : it wakes only when there is work. latency = the cost of waking.\n");
    printf("              suits events that come now and then.\n");
    printf("  DMA       : it wakes only when a batch ends. latency = until the batch fills.\n");
    printf("              suits data that is large and steady.\n");

    printf("\n== there are places where polling is better ==\n");
    double poll_hz[] = { 1e3, 1e4, 1e5, 1e6 };
    printf("  %-16s %-14s %s\n", "poll rate", "CPU used", "mean latency");
    for (unsigned i = 0; i < 4; i++) {
        char pct[16]; snprintf(pct, sizeof pct, "%.2f%%", poll_hz[i] * POLL_COST_US / 1e6 * 100.0);
        printf("  %-11.0f /s %-14s %.1f microseconds\n", poll_hz[i], pct, 1e6 / poll_hz[i] / 2);
    }
    printf("  * when events are very frequent polling wins --- there is always work, so\n");
    printf("    there is no reason to pay the cost of waking each time. That is why fast\n");
    printf("    network drivers switch to polling under load (Linux NAPI).\n");
    return 0;
}

Output

== assumptions (the values used) ==
  handling one interrupt    : 1.0 microseconds
  one poll (read and judge) : 0.2 microseconds
  DMA batch size            : 256 bytes
  * all three differ by machine. Change them and the conclusion changes --- that is the point.

== an interrupt per byte vs DMA in 256-byte batches ==
  link                          IRQ/s        CPU    DMA IRQ/s        CPU
  UART 9600 8N1                   960       0.1%            4     0.000%
  UART 115200 8N1               11520       1.2%           45     0.005%
  I2C 400 kHz                   44000       4.4%          172     0.017%
  SPI 10 MHz                  1250000     125.0%         4883     0.488%
  USB 2.0 (effective)        30000000    3000.0%       117188    11.719%
  1 Gbit Ethernet           118000000   11800.0%       460938    46.094%

  * reading the table downwards shows when DMA becomes necessary.
    on a slow link an interrupt per byte is no trouble; past some point
    that scheme leaves the machine doing nothing else.

== where the crossover is (assuming an interrupt per byte) ==
  CPU used       events/s         the data rate that means
  5%             50000            48.8 KiB/s
  10%            100000           97.7 KiB/s
  20%            200000           195.3 KiB/s
  40%            400000           390.6 KiB/s
  80%            800000           781.2 KiB/s

== the three schemes compared ==
  polling   : the CPU keeps asking.      latency = about half the period.
              suits very frequent events, or very simple places.
  interrupt : it wakes only when there is work. latency = the cost of waking.
              suits events that come now and then.
  DMA       : it wakes only when a batch ends. latency = until the batch fills.
              suits data that is large and steady.

== there are places where polling is better ==
  poll rate        CPU used       mean latency
  1000        /s 0.02%          500.0 microseconds
  10000       /s 0.20%          50.0 microseconds
  100000      /s 2.00%          5.0 microseconds
  1000000     /s 20.00%         0.5 microseconds
  * when events are very frequent polling wins --- there is always work, so
    there is no reason to pay the cost of waking each time. That is why fast
    network drivers switch to polling under load (Linux NAPI).

Three things to note.

First, there is no “good scheme”, only “the scheme that fits this speed”. A 9600 baud UART is only 960 bytes per second, so even an interrupt per byte takes 0.1% of the CPU. Use the same scheme on 1 Gbit Ethernet and the CPU is 118 times short.

Second, DMA does not remove “the moving”; it reduces “how often you are interrupted”. Batch 256 bytes and the events drop to a 256th. In exchange you must wait for the batch to fill, so latency grows — which is why data with a short deadline gets small batches.

Third, when events are very frequent, polling wins. There is always work anyway, so there is no reason to pay the cost of waking each time. Linux network drivers turning interrupts off under load and switching to polling (NAPI) is exactly that. The three are not a ladder but places to move between as the load changes.

pollinginterruptsDMA
what the CPU doeskeeps askingwakes only when there is workonly starts and finishes
latencyabout half the polling periodthe cost of wakinguntil the batch fills
when events are rarewastefulrightexcessive
when events are frequentrightthe machine does nothing elseright
when the data is largeevery byte goes through the CPUlikewiseright — it does not go through the CPU

Table 105.28 — The three at a glance

What to take from this#

Recap

  • Four questions settle a connection — how many wires, a clock, addressing, who starts.
  • The faster it goes the more parallel loses. Today’s answer is one very fast serial line, bundled several times over.
  • A UART cuts characters out with an agreed shape and no clock. The price is a baud error budget, and the theoretical limit (about 5%) and the design rule (2 to 3%) are different numbers.
  • I2C carries many devices on two wires — a design that only ever pulls down gets ACK and arbitration for free.
  • SPI is a ring of two shift registers. So full duplex is free, and a mode mismatch shifts every value by one place.
  • PCI’s idea is that every device carries a business card of the same shape. Even the size is discovered by “writing ones and reading back”.
  • USB has the host command everything. Its “interrupt transfer” is not an interrupt but a periodic question.
  • Polling, interrupts and DMA are not a ladder but three places chosen by load. The choosing is arithmetic, not taste — cost per event × events per second.