问题背景#
- 对于一个输入向量 X,大小为
1 x K(FP16) - 压缩的量化权重矩阵 Wq,大小为
N x (K / 2)(uint8)
解压量化权重:对于 Wq,Row-Major 存储,每一个 Byte 存 2 个 INT4
- Wint4[n,2k]=Wq[n,k]&0xF
- Wint4[n,2k+1]=(Wq[n,k]>>4)&0xF
量化参数:
- G=128,S 大小
N x (K/G),Z 大小N x (K/G)
量化公式:
W^[n,k]Y=(Wint4[n,k]−Z[n,⌊k/G⌋])×S[n,⌊k/G⌋]=XW^T(Y∈R1×N)本机环境#
- RTX 5060 Laptop GPU
- CUDA 12.8
Baseline#
一个很直观的想法是:一个线程处理一行,总共需要 N 个线程,每一个线程负责输出一个 y[n]。需要注意的是数据类型的转换,我们用 int 来接解包后的权重,反量化与最后结果计算都是浮点运算,因此需要将输入向量与量化参数由 FP16 转换成 FP32,计算需要高精度
值得一提的是:我们选择用一个
int来存放一个解包后权重,而没有选择更小位数的uint8_t,本质原因是 RTX 5060 的通用寄存器以 32 bit 为单位,单独参与运算通常也是采用同样粒度的寄存器
1__global__ void w4a16_kernel(2 const __half* x,3 const uint8_t* w_q,4 const __half* scales,5 const __half* zeros,6 __half* y,7 int64_t N, int K, int G)8{9 const int tid = threadIdx.x + blockIdx.x * blockDim.x;10 const int wq_cols = K / 2;11 const int groups_per_row = K / G;12 const int bytes_per_group = G / 2;13 if (tid >= N) return;14
15 float acc = 0.0f;16 for (int b = 0; b < wq_cols; b++)17 {18 const int packed = w_q[tid * wq_cols + b];19 const int q0 = packed & 0xfu;20 const int q1 = (packed >> 4) & 0xfu;21
22 const int group_idx = b / bytes_per_group + tid * groups_per_row;23
24 const float scale = __half2float(scales[group_idx]);25 const float zero = __half2float(zeros[group_idx]);26
27 const float w0 = (q0 - zero) * scale;28 const float w1 = (q1 - zero) * scale;29
30 const float x0 = __half2float(x[2 * b]);31 const float x1 = __half2float(x[2 * b + 1]);32 acc += x0 * w0 + x1 * w1;33 }34 y[tid] = __float2half_rn(acc);35}上面的代码很直观,但存在巨大的访存问题,当两个相邻线程在同一个 b 的循环里时,读取 w_q 元素相差 wq_cols 地址,我们可以先看 ncu 分析:
以下内容来自本机的 Nsight Compute 终端输出,用例规模
N = 4096K =4096,gird x block = (512, 1, 1) x (256, 1, 1)
1 Section: GPU Speed Of Light Throughput2 ----------------------- ----------- ------------3 Metric Name Metric Unit Metric Value4 ----------------------- ----------- ------------5 DRAM Frequency Ghz 5.976 SM Frequency Mhz 773.327 Elapsed Cycles cycle 1,190,1568 Memory Throughput % 56.249 DRAM Throughput % 8.5410 Duration ms 1.5311 L1/TEX Cache Throughput % 98.3012 L2 Cache Throughput % 1.6413 SM Active Cycles cycle 677,132.9214 Compute (SM) Throughput % 8.8815 ----------------------- ----------- ------------16
17 Section: Memory Workload Analysis18 -------------------- ----------- ------------19 Metric Name Metric Unit Metric Value20 -------------------- ----------- ------------21 Memory Throughput Gbyte/s 16.3022 Mem Busy % 56.2423 Max Bandwidth % 8.5524 L1/TEX Hit Rate % 98.9025 L2 Compression Ratio 026 L2 Hit Rate % 1.7027 Mem Pipes Busy % 8.5528 -------------------- ----------- ------------29
30 Section: Occupancy31 ------------------------------- ----------- ------------32 Metric Name Metric Unit Metric Value33 ------------------------------- ----------- ------------34 Block Limit Barriers block 2435 Block Limit SM block 2436 Block Limit Registers block 637 Block Limit Shared Mem block 1638 Block Limit Warps block 639 Theoretical Active Warps per SM warp 4840 Theoretical Occupancy % 10041 Achieved Occupancy % 16.7242 Achieved Active Warps Per SM warp 8.0343 ------------------------------- ----------- ------------- 接回刚刚提到的固定循环
b,相邻线程读取w_q的地址相隔 2048 B,所以一条权重加载指令,32 个 lane 各读一个 B,却触及了 32 个不同的 sector,这点与 L1/TEX Throughput 98.30% 密切相关,L1/TEX 处理压力大。 - 同样注意到 L1/TEX Hit Rate 98.90%,说明请求到 L1 的大多数请求还是命中了,这是源于每一个线程在下一个
b会继续在当前 sector 读取元素。 - 还有两个很关键的指标 Achieved Occupancy 16.72%,Achieved Active Warps Per SM 8.03Warps。由于 RTX 5060 Laptop GPU 是 SM_12,硬件规定每个 SM 最多同时驻留 48 个 warp,这里 warp 活跃度低是因为一共 4096 行权重,一共只发射了 16 个 block,导致一个 SM 里的 warp 太少,warp 调度器很难通过频繁发射 warp 来掩盖访存延迟。Compute (SM) Throughput 8.88% 远低于访存相关吞吐 Memory Throughput 56.24%,显然是计算瓶颈。
- 对于 SM Occupancy,其理论最大值为 Theoretical Occupancy 100%,Theoretical Active Warps per SM 48Warps
Warp profiling#
接下来可以一个 warp 处理一行压缩权重,这能很好地解决相邻线程访存地址跨度太大的问题。每一个 warp 计算一行结束后,进行一次 warp sum reduction
1__global__ void w4a16_kernel(2 const __half* x,3 const uint8_t* w_q,4 const __half* scales,5 const __half* zeros,6 __half* y,7 int64_t N, int K, int G)8{9 const int tid = threadIdx.x + blockIdx.x * blockDim.x;10 const int wq_cols = K / 2;11 const int groups_per_row = K / G;12 const int bytes_per_group = G / 2;13 const int lane = threadIdx.x & 31u;14 const int warp_idx = threadIdx.x >> 5;15 const int global_warp_idx = tid >> 5;16 if (global_warp_idx >= N) return;17
18 float acc = 0.0f;19 for (int b = lane; b < wq_cols; b += 32)20 {21 const int packed = w_q[global_warp_idx * wq_cols + b];22 const int q0 = packed & 0xfu;23 const int q1 = (packed >> 4) & 0xfu;24
25 const int global_group_idx = b / bytes_per_group + groups_per_row * global_warp_idx;26 const float scale = __half2float(scales[global_group_idx]);27 const float zero = __half2float(zeros[global_group_idx]);28
29 const float w0 = (q0 - zero) * scale;30 const float w1 = (q1 - zero) * scale;31
32 const float x0 = __half2float(x[2 * b]);33 const float x1 = __half2float(x[2 * b + 1]);34
35 acc += x0 * w0 + x1 * w1;36 }37
38 for (int offset = 16; offset > 0; offset >>= 1)39 {40 acc += __shfl_down_sync(0xffffffffu, acc, offset);41 }42
43 if (lane == 0)44 y[global_warp_idx] = __float2half_rn(acc);45}下面还是先看 ncu 分析:
1 Section: GPU Speed Of Light Throughput2 ----------------------- ----------- ------------3 Metric Name Metric Unit Metric Value4 ----------------------- ----------- ------------5 DRAM Frequency Ghz 10.996 SM Frequency Ghz 1.257 Elapsed Cycles cycle 161,9238 Memory Throughput % 63.919 DRAM Throughput % 23.0010 Duration us 128.6111 L1/TEX Cache Throughput % 66.8312 L2 Cache Throughput % 9.9413 SM Active Cycles cycle 154,162.9214 Compute (SM) Throughput % 66.7915 ----------------------- ----------- ------------16
17 Section: Memory Workload Analysis18 -------------------- ----------- ------------19 Metric Name Metric Unit Metric Value20 -------------------- ----------- ------------21 Memory Throughput Gbyte/s 80.8722 Mem Busy % 32.0023 Max Bandwidth % 63.9124 L1/TEX Hit Rate % 90.0825 L2 Compression Ratio 026 L2 Hit Rate % 3.9727 Mem Pipes Busy % 63.9128 -------------------- ----------- ------------29
30 Section: Occupancy31 ------------------------------- ----------- ------------32 Metric Name Metric Unit Metric Value33 ------------------------------- ----------- ------------34 Block Limit Barriers block 2435 Block Limit SM block 2436 Block Limit Registers block 637 Block Limit Shared Mem block 1638 Block Limit Warps block 639 Theoretical Active Warps per SM warp 4840 Theoretical Occupancy % 10041 Achieved Occupancy % 87.7742 Achieved Active Warps Per SM warp 42.1343 ------------------------------- ----------- ------------- 首先值得关注的是 Achieved Occupancy 87.77%,Achieved Active Warps Per SM 42.13Warps,相较于上一个版本,这个版本发射的总 block 数量为 512 个,一个 SM 内可以驻留更多的 warp,相应的 Compute (SM) Throughput 也从 8.88% 上升至 66.79%
- 其次是之前探讨过的话题,用 warp 为单位处理一行
w_q能明显改善访存。L1/TEX Cache Throughput 66.83%,访存压力明显有所改善。 - 当前版本的 Compute (SM) Throughput 66.79% 和 Memory Throughput 63.91% 大致相当,计算和访存是平衡的
通过观察不难看出,每一次循环我们都进行了一次解包,但显然一个 group 横跨的范围是两个循环,因此下一个优化版本我们可以考虑按量化参数组循环,复用量化参数
group profiling#
为了能够复用同一组的量化参数,这里给出一个解法:在外层加上一个 group 循环
1__global__ void w4a16_kernel(2 const __half* x,3 const uint8_t* w_q,4 const __half* scales,5 const __half* zeros,6 __half* y,7 int64_t N, int K, int G)8{9 const int tid = threadIdx.x + blockIdx.x * blockDim.x;10 const int wq_cols = K / 2;11 const int groups_per_row = K / G;12 const int bytes_per_group = G / 2;13 const int lane = threadIdx.x & 31u;14 const int warp_idx = threadIdx.x >> 5;15 const int global_warp_idx = tid >> 5;16 if (global_warp_idx >= N) return;17
18 float acc = 0.0f;19 for (int g = 0; g < groups_per_row; g++)20 {21 const int global_group_idx = g + groups_per_row * global_warp_idx;22 const float scale = __half2float(scales[global_group_idx]);23 const float zero = __half2float(zeros[global_group_idx]);24
25 for (int b_in_group = lane; b_in_group < bytes_per_group; b_in_group += 32)26 {27 const int b = g * bytes_per_group + b_in_group;28 const int packed = w_q[global_warp_idx * wq_cols + b];29 const int q0 = packed & 0xfu;30 const int q1 = (packed >> 4) & 0xfu;31
32 const float w0 = (q0 - zero) * scale;33 const float w1 = (q1 - zero) * scale;34
35 const float x0 = __half2float(x[2 * b]);36 const float x1 = __half2float(x[2 * b + 1]);37
38 acc += x0 * w0 + x1 * w1;39 }40 }41
42 for (int offset = 16; offset > 0; offset >>= 1)43 {44 acc += __shfl_down_sync(0xffffffffu, acc, offset);45 }46
47 if (lane == 0)48 y[global_warp_idx] = __float2half_rn(acc);49}还可以进一步在外层循环中让第一个线程读取,其他线程通过 warp 内广播来获取相同参数
1float scale, zero;2if (lane == 0)3{4 scale = __half2float(scales[global_group_idx]);5 zero = __half2float(zeros[global_group_idx]);6}7zero = __shfl_down_sync(0xffffffffu, zero, 0);8scale = __shfl_down_sync(0xffffffffu, scale, 0);先具体的 ncu 性能指标:
本次性能测试新加了 Section: Compute Workload Analysis 与 Section: Instruction Statistics,方便与后续的版本分析对比
第一项为无广播版本的指标
1 Section: GPU Speed Of Light Throughput2 ----------------------- ----------- ------------3 Metric Name Metric Unit Metric Value4 ----------------------- ----------- ------------5 Elapsed Cycles cycle 141,4356 Memory Throughput % 58.947 DRAM Throughput % 27.958 Duration us 109.349 L1/TEX Cache Throughput % 62.6610 L2 Cache Throughput % 11.9711 SM Active Cycles cycle 132,239.8812 Compute (SM) Throughput % 65.6413 ----------------------- ----------- ------------14
15 Section: Compute Workload Analysis16 -------------------- ----------- ------------17 Metric Name Metric Unit Metric Value18 -------------------- ----------- ------------19 Executed Ipc Active inst/cycle 2.7920 Executed Ipc Elapsed inst/cycle 2.6321 Issue Slots Busy % 69.7822 Issued Ipc Active inst/cycle 2.7923 SM Busy % 69.7824 -------------------- ----------- ------------25
26 Section: Memory Workload Analysis27 -------------------- ----------- ------------28 Metric Name Metric Unit Metric Value29 -------------------- ----------- ------------30 Memory Throughput Gbyte/s 98.2931 Mem Busy % 29.5232 Max Bandwidth % 58.9433 L1/TEX Hit Rate % 88.9934 L2 Compression Ratio 035 L2 Hit Rate % 4.9136 Mem Pipes Busy % 58.9437 -------------------- ----------- ------------38
39 Section: Instruction Statistics40 ---------------------------------------- ----------- ------------41 Metric Name Metric Unit Metric Value42 ---------------------------------------- ----------- ------------43 Avg. Executed Instructions Per Scheduler inst 92,278.1544 Executed Instructions inst 9,596,92845 Avg. Issued Instructions Per Scheduler inst 92,278.1546 Issued Instructions inst 9,596,92847 ---------------------------------------- ----------- ------------48
49 Section: Occupancy50 ------------------------------- ----------- ------------51 Metric Name Metric Unit Metric Value52 ------------------------------- ----------- ------------53 Theoretical Active Warps per SM warp 4854 Theoretical Occupancy % 10055 Achieved Occupancy % 85.7056 Achieved Active Warps Per SM warp 41.1357 ------------------------------- ----------- ------------第二项为有广播版本的指标
1 Section: GPU Speed Of Light Throughput2 ----------------------- ----------- ------------3 Metric Name Metric Unit Metric Value4 ----------------------- ----------- ------------5 Elapsed Cycles cycle 150,6046 Memory Throughput % 68.907 DRAM Throughput % 21.698 Duration us 117.099 L1/TEX Cache Throughput % 71.2010 L2 Cache Throughput % 9.9911 SM Active Cycles cycle 144,713.1912 Compute (SM) Throughput % 68.9013 ----------------------- ----------- ------------14
15 Section: Compute Workload Analysis16 -------------------- ----------- ------------17 Metric Name Metric Unit Metric Value18 -------------------- ----------- ------------19 Executed Ipc Active inst/cycle 2.7920 Executed Ipc Elapsed inst/cycle 2.7021 Issue Slots Busy % 69.8622 Issued Ipc Active inst/cycle 2.7923 SM Busy % 69.8624 -------------------- ----------- ------------25
26 Section: Memory Workload Analysis27 -------------------- ----------- ------------28 Metric Name Metric Unit Metric Value29 -------------------- ----------- ------------30 Memory Throughput Gbyte/s 76.2531 Mem Busy % 34.5032 Max Bandwidth % 68.9033 L1/TEX Hit Rate % 88.9734 L2 Compression Ratio 035 L2 Hit Rate % 4.9636 Mem Pipes Busy % 68.9037 -------------------- ----------- ------------38
39 Section: Instruction Statistics40 ---------------------------------------- ----------- ------------41 Metric Name Metric Unit Metric Value42 ---------------------------------------- ----------- ------------43 Avg. Executed Instructions Per Scheduler inst 101,100.3144 Executed Instructions inst 10,514,43245 Avg. Issued Instructions Per Scheduler inst 101,100.3146 Issued Instructions inst 10,514,43247 ---------------------------------------- ----------- ------------48
49 Section: Occupancy50 ------------------------------- ----------- ------------51 Metric Name Metric Unit Metric Value52 ------------------------------- ----------- ------------53 Theoretical Active Warps per SM warp 4854 Theoretical Occupancy % 10055 Achieved Occupancy % 85.7756 Achieved Active Warps Per SM warp 41.1757 ------------------------------- ----------- ------------可以看到,有广播的版本反而比无广播版本的耗时更多。具体原因可以从指令看出:
- 前者 warp 调度器发射的 warp 级指令总数为 Issued Instructions 9,596,928;后者的该项值为 Issued Instructions 10,514,432
指令数量变多主要来自于几个方面,这里将从 SASS 分析具体每一条相关汇编指令执行了多少次:
可以用 ncu 测量每一条 SASS 实际执行了几次
- 广播版本新增了如下指令:(后者是动态 warp 执行次数)
SHFL.IDX广播指令:262,144ISETP.NE.AND生成谓词分支:131,072BRA条件跳转:131,072BSSY.RECONVERGENT:设定重汇合位置:131,072BSYNC.RECONVERGENT:执行重汇合:262,144
上述新增指令累计 917,504,正好是多出来的指令数。
多出来的指令数抵消掉了 warp 内广播所带来的收益(查资料可得,SHFL 约 29 个周期,L1 命中加载约 44 个周期,两者的延迟不同),并且相邻线程命中的是同一个 sector。因此后续的版本继续是以非广播为基线优化
向量加载#
为了进一步减少指令数以及访存压力,可以采取向量加载。本次版本选择 W-q 用 uint32_t 加载,x 用 uint4 加载。
在开始优化前,建议先查看上一个版本的 SASS 指令,因为很多情况下,编译器会自动发射宽加载指令。
1LDG.E.U16 R29, desc[UR8][R8.64]2LDG.E.U16 R27, desc[UR8][R8.64+0x2]以上是编译器生成的加载连续两个相邻 x 的指令,可以看到这里编译器并没有优化为宽加载。
值得注意的是,向量加载通常要求严格的地址对齐,在本次例子中,x 需要 16 字节对齐,w_q 需要 4 字节对齐。否则要单独处理跨扇区访问。
优化代码如下:
1__global__ void w4a16_kernel(2 const __half* x,3 const uint8_t* w_q,4 const __half* scales,5 const __half* zeros,6 __half* y,7 int64_t N, int K, int G)8{9 const int tid = threadIdx.x + blockIdx.x * blockDim.x;10 const int wq_cols = K / 2;11 const int groups_per_row = K / G;12 const int bytes_per_group = G / 2;13 const int lane = threadIdx.x & 31u;14 const int warp_idx = threadIdx.x >> 5;15 const int64_t global_warp_idx = tid >> 5;16
17 if (global_warp_idx >= N) return;18
19 const int64_t wq_row_begin = global_warp_idx * wq_cols;20 const int64_t group_row_begin = glabal_warp_idx + groups_per_row;21 const int lane_in_group = lane % 16;22 float acc = 0.0f;23
24 for (int g = lane >> 4; g < groups_per_rows; g += 20)25 {26 const int64_t group_id = group_row + g;27 const float scale = __half2float(scales[group_id]);28 const float zero = __half2float(zeros[group_id]);29
30 const int b = g * bytes_per_group + lane_in_group * 4; // b: 第一个打包权重字节的下标31 const uint32_t w_packed = *reinterpret_cast<const uint32_t*>(w_q + wq_row_begin + b);32 const uint4 x_packed = *reinterpret_cast<const uint4*>(x + 2 * b);33 const uint32_t w_word[4] = {x_packed.x, x_packed.y, x_packed.z, x_packed.w};34
35 #pragma unroll36 for (int i = 0; i < 4; i++)37 {38 const uint32_t packed = w_packed >> (i * 8);39 const int q0 = packed & 0xfu;40 const int q1 = (packed >> 4) & 0xFu;41
42 const float w0 = (q0 - zero) * scale;43 const float w1 = (q1 - zero) * scale;44
45 const uint32_t x_bits = x_words[i];46 const __half x0 = __ushort_as_half(static_cast<uint16_t>(x_bits));47 const __half x1 = __ushort_as_half(static_cast<uint16_t>(x_bits >> 16));48
49 acc += w0 * __half2float(x0);50 acc += w1 * __half2float(x1);51 }52 }53
54 for (int offset = 16; offset > 0; offset >>= 1)55 {56 acc += __shfl_down_sync(0xffffffffu, acc, offset);57 }58
59 if (lane == 0)60 y[global_warp_idx] = __float2half_rn(acc);61}以下是当前版本的 ncu 指标:
1 Section: GPU Speed Of Light Throughput2 ----------------------- ----------- ------------3 Metric Name Metric Unit Metric Value4 ----------------------- ----------- ------------5 Elapsed Cycles cycle 66,4496 Memory Throughput % 46.787 DRAM Throughput % 46.788 Duration us 54.469 L1/TEX Cache Throughput % 36.2510 L2 Cache Throughput % 11.8411 SM Active Cycles cycle 61,716.8112 Compute (SM) Throughput % 75.2613 ----------------------- ----------- ------------14
15 Section: Compute Workload Analysis16 -------------------- ----------- ------------17 Metric Name Metric Unit Metric Value18 -------------------- ----------- ------------19 Executed Ipc Active inst/cycle 3.2320 Executed Ipc Elapsed inst/cycle 3.0121 Issue Slots Busy % 80.7922 Issued Ipc Active inst/cycle 3.2323 SM Busy % 80.7924 -------------------- ----------- ------------25
26 Section: Memory Workload Analysis27 -------------------- ----------- ------------28 Metric Name Metric Unit Metric Value29 -------------------- ----------- ------------30 Memory Throughput Gbyte/s 164.3831 Mem Busy % 28.4032 Max Bandwidth % 46.7833 L1/TEX Hit Rate % 80.2434 L2 Compression Ratio 035 L2 Hit Rate % 3.7536 Mem Pipes Busy % 33.7737 -------------------- ----------- ------------38
39 Section: Instruction Statistics40 ---------------------------------------- ----------- ------------41 Metric Name Metric Unit Metric Value42 ---------------------------------------- ----------- ------------43 Avg. Executed Instructions Per Scheduler inst 49,860.9244 Executed Instructions inst 5,185,53645 Avg. Issued Instructions Per Scheduler inst 49,860.9246 Issued Instructions inst 5,185,53647 ---------------------------------------- ----------- ------------48
49 Section: Occupancy50 ------------------------------- ----------- ------------51 Metric Name Metric Unit Metric Value52 ------------------------------- ----------- ------------53 Theoretical Active Warps per SM warp 4854 Theoretical Occupancy % 10055 Achieved Occupancy % 85.9556 Achieved Active Warps Per SM warp 41.2657 ------------------------------- ----------- ------------- 首先关注 Memory Throughput 与 Compute (SM) Throughput,这次向量加载明显降低了访存的压力,导致该 kernel 的瓶颈转移到了计算上
- 向量加载最直接的影响是 Issued Instructions 5,185,536,比前版本少了将近一半的指令数,这部分指令数主要来自于访问 glabal memory
- 详细分析计算瓶颈:SM 吞吐量的总汇总值为 74.99%,展开相关子指标,整理如下:
1--------------- ----------- ---------------2Metric Name Metric Unit Metric Value3--------------- ----------- ---------------4Issue Active % 75.095Inst Executed % 75.096ALUHeavy % 57.487FMAHeavy % 50.698LSU % 33.691----------------------------------------------------------------------- ----------- ------------2Metric Name Metric Unit Metric Value3----------------------------------------------------------------------- ----------- ------------4smsp__warps_active.avg.peak_sustained warp 125smsp__warps_active.avg.per_cycle_active warp 10.146smsp__warps_eligible.avg.per_cycle_active warp 4.637smsp__issue_active.avg.per_cycle_active warp/cycle 0.808smsp__issue_active.avg.pct_of_peak_sustained_active % 79.569smsp__issue_inst0.avg.pct_of_peak_sustained_active % 20.4410
11smsp__warp_issue_stalled_not_selected_per_warp_active.pct % 38.0412smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct % 22.6313smsp__warp_issue_stalled_math_pipe_throttle_per_warp_active.pct % 15.8714smsp__warp_issue_stalled_wait_per_warp_active.pct % 7.1215smsp__warp_issue_stalled_short_scoreboard_per_warp_active.pct % 1.3416----------------------------------------------------------------------- ----------- ------------Issue Active / Inst Executed 75.09% 整个 SM 的指令发射与执行是比较繁忙的,平均每一个调度器有 10.14 个驻留 warp,4.63 个就绪 warp,每周期约发射 0.8 条指令。smsp__issue_inst0.avg.pct_of_peak_sustained_active 20.44% 表示约有 1/5 的周期没有就绪 warp,smsp__warp_issue_stalled_not_selected_per_warp_active.pct 38.04% 表示同时有多个 warp 就绪。这表明就绪的 warp 在时间上分布不均匀。
后续通过 ncu Source 可以定位指标对应的具体的 SASS 指令,warp 未准备就绪的指令出自于这里:压缩权重加载到首次解包之间
0xb40 LDG.E R16, … // 加载压缩权重 0xc90 LOP3.LUT R32, R16, 0xf, … // 等 R16 就绪,取低 4 位
所以总的来说,当前版本的主要问题在于:一方面要发射大量解包、转换和反量化指令,另一方面在部分周期等压缩权重就绪。优化策略是,可以提前进行加载指令的发射。
inline PTX#
进一步优化可以采取 PTX 虚拟指令:
- 将解包过程转换成 PTX
- 将小分支转换成 PTX
对于小分支,例如 if (lane == 0),SASS 已经在最后一条线程间 shuffle 后用 @p0 EXIT 让其余 lane 退出。这表明编译器其实是可以自动用谓词优化了小分支,这里就不再对其写 PTX
可以将按位与取位改写成 PTX:
1int q1;2asm("bfe.u32 %0, %1, 4, 4;"3 : "=r"(q1)4 : "r"(packed));上述 PTX 表示直接从第 4 位开始取 4 位放入 p1
实测 ncu 相关指标如下:
1无 PTX2 Avg. Executed Instructions Per Scheduler inst 45,489.233 Executed Instructions inst 4,730,8804 Avg. Issued Instructions Per Scheduler inst 45,489.235 Issued Instructions inst 4,730,8806
7有 PTX8 Avg. Executed Instructions Per Scheduler inst 46,119.389 Executed Instructions inst 4,796,41610 Avg. Issued Instructions Per Scheduler inst 46,119.3811 Issued Instructions inst 4,796,416可看到,更改为 PTX 后指令数量反而变多,这里核对了 SASS 发现多出的指令来源于取最高 4 位:
源代码里,内层循环只有 4 次,使用 #pragma unroll 展开循环。在取最后高位时,编译器知道最后 i = 3,所以原版的第 4 次循环可以简化为 w_packed >> 28(右移后只剩下最高 4 位,& 0xfu 可以省略),省略的证据来自于下面的 SASS 指令:
1C++ 版20x1110 SHF.R.U32.HI R26, RZ, 0x1c, R26 // 右移 28 位30x1150 I2FP.F32.U32 R26, R264
5bfe.u32 版60x10b0 SHF.R.U32.HI R26, RZ, 0x18, R26 // 右移 24 位,q0 也用这份结果70x10f0 SHF.R.U32.HI R9, RZ, 0x4, R26 // 再右移 4 位80x1140 SGXT.U32 R9, R9, 0x4 // 再取低 4 位并做零扩展90x1180 I2FP.F32.S32 R9, R9这里 0x10b0 的右移还供同一字节的 q0 使用,整个循环来看,两版的移位指令总数相同。净增来自掩码和位域提取指令从 14 条变成了 16 条。N = K = 4096、G = 128 时,一行一个 warp,所以有 4,096 个 warp。每个 warp 在源码里跑 32 / 2 = 16 轮 g 循环,编译器把两轮合进一个 SASS 循环体,所以实际跑 8 轮。每轮净增 2 条,刚好是 4096 × 8 × 2 = 65,536
当然还可以对其他代码写 PTX,这里实际测量个人感觉提升效果不大。所以暂且不采用 PTX
如果这篇文章对你有帮助,欢迎分享给更多人!
部分信息可能已经过时
