异步设备IO：OVERLAPPED和IOCompletionPort
本文内容为《windows核心编程》第10章内容的总结，仅记录一些本人感兴趣的内容。
1：OVERLAPPED
“overlapped”的意思是执行IO请求的事件与线程执行其他任务的时间是重叠的（overlapped）。
overlapped是执行设备异步IO的基础。overlapped结构定义如下：
Descriptio：Contains information used in asynchronous (or overlapped) input and output (I/O).保存异步IO信息的结构体。
1 typedef struct _OVERLAPPED { 2 ULONG_PTR Internal; 3 ULONG_PTR InternalHigh; 4 union { 5 struct { 6 DWORD Offset; 7 DWORD OffsetHigh; 8 }; 9 PVOID Pointer; 10 }; 11 HANDLE hEvent; 12 } OVERLAPPED, *LPOVERLAPPED;
Members
- Internal
-
Reserved for operating system use. This member, which specifies a system-dependent status, is valid when the GetOverlappedResult function returns without setting the extended error information to ERROR_IO_PENDING.
系统预留的关键字。
- InternalHigh
-
Reserved for operating system use. This member, which specifies the length of the data transferred, is valid when the GetOverlappedResult function returns TRUE.
系统预留的关键字。
- Offset
-
The file position at which to start the transfer. The file position is a byte offset from the start of the file. The calling process must set this member before calling the ReadFile or WriteFile function.
This member is used only when the device is a file. Otherwise, this member must be zero.
- OffsetHigh
-
The high-order word of the file position at which to start the transfer.
This member is used only when the device is a file. Otherwise, this member must be zero.
Offset和OffsetHigh参数仅文件设备有效，非文件设备参数应置为0，否则IO请求将会失败，调用GetLastError会返回ERROR_INVALID_PARAMETER。当设备是文件时，该参数指示访问文件时应该从哪里进行IO操作
- Pointer
-
Reserved for system use; do not use.
系统预留。
- hEvent
-
A handle to the event that will be set to the signaled state when the operation has been completed. The calling process must set this member either to zero or a valid event handle before calling any overlapped functions. To create an event object, use the CreateEvent function. This function returns a handle that can be used to synchronize simultaneous I/O requests for a device.
Functions such as ReadFile and WriteFile set this handle to the nonsignaled state before they begin an I/O operation. When the operation has completed, the handle is set to the signaled state.
Functions such as GetOverlappedResult and the wait functions reset auto-reset events to the nonsignaled state. Therefore, if you use an auto-reset event, your application can hang if you wait for the operation to complete then call GetOverlappedResult.
IO操作完成的通知事件。在使用IOCP时可用。
如果IO操作是同步执行的，ReadFile和WriteFile将会返回非0值。如果IO操作异步执行，ReadFile和WriteFile将返回False，yongGetLastError将会返回ERROR_IO_PENDING，表示IO请求已经加入队列，晚些时候完成。
要注意的是，使用ReadFile和WriteFile这样的异步操作要将ReadFile和WriteFile的参数定义为全局变量，不要定义成局部变量。局部变量被释放后，ReadFile和WriteFile使用局部变量的地址空间，将会产生无法预料的错误。
2：IOCompletionPort
2.1 IO完成端口定义
首先，要明确IO完成端口是一个内核对象，既然是内核对象，按照windows惯例，内部结构是未知的（HADNLDE是指针，外部不可见）。
IO完成端口背后的理论是并发运行的线程的数量必须有一个上限，因为过多的线程会导致Windows内核频繁在可运行线程里面上下文切换，浙这将占用大量的cpu。一旦可运行线程数量大于可用cpu数量，系统将花时间在执行线程上下文切换，这是并发模型的一个潜在缺点。IO完成端口的创建函数如下：
Description：
Associates an input/output (I/O) completion port with one or more file handles, or it can create an I/O completion port that is not associated with a file handle.
Associating an instance of an opened file with an I/O completion port lets an application receive notification of the completion of asynchronous I/O operations involving that file.
1 HANDLE WINAPI CreateIoCompletionPort( 2 __in HANDLE FileHandle, 3 __in HANDLE ExistingCompletionPort, 4 __in ULONG_PTR CompletionKey, 5 __in DWORD NumberOfConcurrentThreads 6 );
Parameters
- FileHandle
-
A handle to a file opened for overlapped I/O completion. You must specify the FILE_FLAG_OVERLAPPED flag when using the CreateFile function to obtain the handle.
If FileHandle specifies INVALID_HANDLE_VALUE, CreateIoCompletionPort creates an I/O completion port without associating it with a file. In this case, the ExistingCompletionPort parameter must be NULL and the CompletionKey parameter is ignored.
如果CreateIoCompletionPort 用来创建一个IO完成端口，则应配置FileHandle为INVALID_HANDLE_VALUE，ExistingCompletionPort 为NULL ，CompletionKey将被忽略调。
- ExistingCompletionPort
-
A handle to the I/O completion port.
If this parameter specifies an existing completion port, the function associates it with the file specified by the FileHandle parameter. The function returns the handle of the existing completion port; it does not create a new I/O completion port.
If this parameter is NULL, the function creates a new I/O completion port and associates it with the file specified by FileHandle. The function returns the handle to the new I/O completion port.
如果此值为NULL，函数将创建一个新的IO完成端口。如果FileHandle不是INVALID_HANDLE_VALUE，则新的IO完成端口将与FileHandle关联。
- CompletionKey
-
The per-file completion key that is included in every I/O completion packet for the specified file.
IO完成端口处理完任务后将把此值传回。
- NumberOfConcurrentThreads
-
The maximum number of threads that the operating system can allow to concurrently process I/O completion packets for the I/O completion port. This parameter is ignored if the ExistingCompletionPort parameter is not NULL.
If this parameter is zero, the system allows as many concurrently running threads as there are processors in the system.
允许IO完成端口关联的线程并发运行的数量。如果NumberOfConcurrentThreads赋值为0，将默认为系统的cpu数量。
Return Value
If the function succeeds, the return value is the handle to the I/O completion port that is associated with the specified file.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.
CreateIoCompletionPort函数有2个功能，1：创建一个IO完成端口；2：将一个设备与一个IO完成端口关联。
使用第一个功能时，前3个参数固定传入INVALID_HANDLE_VALUE、NULL、0，最后一个参数传入IOCP允许并发线程数量（为0则默认为cpu数量）。
使用第二个功能时，第一个参数传入设备句柄（包含文件、socket、邮件槽、管道等）、第二个参数传入IOCP的句柄、第三个参数传入完成键值、第四个参数传入0。
2.2 IO完成端口周边架构
IOCP内部维护了设备列表、IO完成队列、等待线程队列、已释放线程队列、已暂停线程队列5个表。
设备列表：使用CreateIoCompletionPort关联设备时，会加入该列表。当设备句柄关闭时，会从设备列表中移除设备信息。
IO完成队列：当设备的异步IO请求完成时，系统会检查设备是否与IOCP关联，如果设备与IOCP关联，那么系统将该项已完成的IO请求加入IO完成队列末尾。也可以使用PostQueuedCompletionStatus向IO完成队列末尾发出请求，进入入队操作。
等待线程队列：如果线程调用GetQueuedCompletionStatus去获取IO完成队列事件，操作系统将线程切换到睡眠状态（注意是睡眠状态，该状态下不占用CPU），并记录线程ID到等待队列。如果IO完成队列非空，且当前并发线程数量小于最大并发线程数量，则唤醒线程，将唤醒线程的ID从等待线程队列中移除。值得注意的是，等待线程队列是后入先出的类型。比如，ABCD4个线程在等待，一个事件到达，D被唤醒，处理完后D又加入线程队列。当下个事件到达时，依旧是D被唤醒。《win核心》解释，通过这种后入先出的算法，系统可以将那些未被调度线程（某些线程总不会调度）的内存资源换到磁盘，将其内容凑个高速缓存中清除。
已释放线程列表：线程调用GetQueuedCompletionStatus得到IO完成队列事件后，如果线程处于可运行状态，则信息记录在已释放线程队列。
已暂停线程列表：如果线程调用函数将自己挂起，信息将进入已暂停线程列表中。注意，非GetQueuedCompletionStatus阻塞的线程才进入已暂停线程列表，而GetQueuedCompletionStatus阻塞的线程进入等待线程列表。
2.3 GetQueuedCompletionStatus
Description：Attempts to dequeue an I/O completion packet from the specified I/O completion port. If there is no completion packet queued, the function waits for a pending I/O operation associated with the completion port to complete.
1 BOOL WINAPI GetQueuedCompletionStatus( 2 __in HANDLE CompletionPort, 3 __out LPDWORD lpNumberOfBytes, 4 __out PULONG_PTR lpCompletionKey, 5 __out LPOVERLAPPED* lpOverlapped, 6 __in DWORD dwMilliseconds 7 );
Parameters
- CompletionPort
-
A handle to the completion port. To create a completion port, use the CreateIoCompletionPort function.
- lpNumberOfBytes
-
A pointer to a variable that receives the number of bytes transferred during an I/O operation that has completed.
IO完成端口接收到数据的数据。
- lpCompletionKey
-
A pointer to a variable that receives the completion key value associated with the file handle whose I/O operation has completed. A completion key is a per-file key that is specified in a call to CreateIoCompletionPort.
- lpOverlapped
-
A pointer to a variable that receives the address of the OVERLAPPED structure that was specified when the completed I/O operation was started.
Return Value
If the function dequeues a completion packet for a successful I/O operation from the completion port, the return value is nonzero. The function stores information in the variables pointed to by the lpNumberOfBytes, lpCompletionKey, and lpOverlapped parameters.
如果函数从完成队列中取出IO事件，返回值非0.并在 lpNumberOfBytes, lpCompletionKey, and lpOverlapped 参数中存储信息。
If *lpOverlapped is NULL and the function does not dequeue a completion packet from the completion port, the return value is zero. The function does not store information in the variables pointed to by the lpNumberOfBytes and lpCompletionKey parameters. To get extended error information, call GetLastError. If the function did not dequeue a completion packet because the wait timed out, GetLastError returns WAIT_TIMEOUT.
如果*lpOverlapped非NULL，且函数未冲完成端口中取出事件信息，返回值为0。函数不会再lpNumberOfBytes, lpCompletionKey存储信息。如果函数因时间到达未取出事件，则GetLastError 返回WAIT_TIMEOUT。
If *lpOverlapped is not NULL and the function dequeues a completion packet for a failed I/O operation from the completion port, the return value is zero. The function stores information in the variables pointed to by lpNumberOfBytes, lpCompletionKey, and lpOverlapped. To get extended error information, call GetLastError.
如果*lpOverlapped非NULL，且IO操作失败，函数取出了IO失败的事件信息，则返回值是0.lpNumberOfBytes, lpCompletionKey, and lpOverlapped存储了失败事件的信息。
2.4 PostQueuedCompletionStatus
Description：Posts an I/O completion packet to an I/O completion port.
1 BOOL WINAPI PostQueuedCompletionStatus( 2 __in HANDLE CompletionPort, 3 __in DWORD dwNumberOfBytesTransferred, 4 __in ULONG_PTR dwCompletionKey, 5 __in LPOVERLAPPED lpOverlapped 6 );
Parameters
- CompletionPort
-
A handle to an I/O completion port to which the I/O completion packet is to be posted.
- dwNumberOfBytesTransferred
-
The value to be returned through the lpNumberOfBytesTransferred parameter of the GetQueuedCompletionStatus function.
- dwCompletionKey
-
The value to be returned through the lpCompletionKey parameter of the GetQueuedCompletionStatus function.
- lpOverlapped
-
The value to be returned through the lpOverlapped parameter of the GetQueuedCompletionStatus function.
Return Value
If the function succeeds, the return value is nonzero.
If the function fails, the return value is zero. To get extended error information, call GetLastError .
PostQueuedCompletionStatus参数和GetQueuedCompletionStatus参数的前4项一一对应。
3 IOCP的封装类
《win核心》示例程序提供了对IOCP的封装类，代码如下：
1 class CIOCP { 2 public: 3 CIOCP(int nMaxConcurrency = -1) { 4 m_hIOCP = NULL; 5 if (nMaxConcurrency != -1) 6 (void) Create(nMaxConcurrency); 7 } 8 9 ~CIOCP() { 10 if (m_hIOCP != NULL) 11 chVERIFY(CloseHandle(m_hIOCP)); 12 } 13 14 //关闭IOCP 15 BOOL Close() { 16 BOOL bResult = CloseHandle(m_hIOCP); 17 m_hIOCP = NULL; 18 return(bResult); 19 } 20 21 //创建IOCP，nMaxConcurrency指定最大线程并发数量，0默认为cpu数量 22 BOOL Create(int nMaxConcurrency = 0) { 23 m_hIOCP = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, nMaxConcurrency); 24 chASSERT(m_hIOCP != NULL); 25 return(m_hIOCP != NULL); 26 } 27 28 //为设备（文件、socket、邮件槽、管道等）关联一个IOCP 29 BOOL AssociateDevice(HANDLE hDevice, ULONG_PTR CompKey) { 30 BOOL fOk = (CreateIoCompletionPort(hDevice, m_hIOCP, CompKey, 0) == m_hIOCP); 31 chASSERT(fOk); 32 return(fOk); 33 } 34 35 //为设备（文件、socket、邮件槽、管道等）关联一个IOCP 36 BOOL AssociateSocket(SOCKET hSocket, ULONG_PTR CompKey) { 37 return(AssociateDevice((HANDLE) hSocket, CompKey)); 38 } 39 40 //为iocp传递事件通知 41 BOOL PostStatus(ULONG_PTR CompKey, DWORD dwNumBytes = 0, 42 OVERLAPPED* po = NULL) { 43 44 BOOL fOk = PostQueuedCompletionStatus(m_hIOCP, dwNumBytes, CompKey, po); 45 chASSERT(fOk); 46 return(fOk); 47 } 48 49 //从IO完成队列中获取事件通知。IO完成队列无事件时，该函数将阻塞 50 BOOL GetStatus(ULONG_PTR* pCompKey, PDWORD pdwNumBytes, 51 OVERLAPPED** ppo, DWORD dwMilliseconds = INFINITE) { 52 53 return(GetQueuedCompletionStatus(m_hIOCP, pdwNumBytes, 54 pCompKey, ppo, dwMilliseconds)); 55 } 56 57 private: 58 //IOCP句柄 59 HANDLE m_hIOCP; 60 };