接口中转stream传输 request/response

php实现

CURLOPT_WRITEFUNCTION: for response ，把接口返回的结果拿回来，会进行多次回调，直到接口中的内容全部读完
CURLOPT_READFUNCTION ：for request  ，把要请求接口的参数数据写出去

CURLOPT_READFUNCTION    
    回调函数名。该函数应接受三个参数。第一个是 cURL resource；
    第二个是通过选项 CURLOPT_INFILE 传给 cURL 的 stream resource；
    第三个参数是最大可以读取的数据的数量。回 调函数必须返回一个字符串，长度小于或等于请求的数据量（第三个参数）。一般从传入的 stream resource 读取。返回空字符串作为 EOF（文件结束） 信号。
CURLOPT_WRITEFUNCTION    
    回调函数名。该函数应接受两个参数。
    第一个是 cURL resource；
    第二个是要写入的数据字符串。数 据必须在函数中被保存。 
    函数必须准确返回写入数据的字节数，否则传输会被一个错误所中断。

A bit more documentation (without minimum version numbers):

- CURLOPT_WRITEFUNCTION
- CURLOPT_HEADERFUNCTION

  Pass a function which will be called to write data or headers respectively. The callback function prototype:

long write_callback (resource ch, string data)

The ch argument is CURL session handle. The data argument is data received. Note that its size is variable. When writing data, as much data as possible will be returned in all invokes. When writing headers, exactly one complete header line is returned for better parsing.
The function must return number of bytes actually taken care of. If that amount differs from the amount passed to this function, an error will occur.

- CURLOPT_READFUNCTION
  Pass a function which will be called to read data. The callback function prototype:

string read_callback (resource ch, resource fd, long length)

The ch argument is CURL session handle. The fd argument is file descriptor passed to CURL by CURLOPT_INFILE option. The length argument is maximum length which can be returned.
The function must return string containing the data which were read. If length of the data is more than maximum length, it will be truncated to maximum length. Returning anything else than a string means an EOF.

[Note: there is more callbacks implemented in current cURL library but they aren't unfortunately implemented in php curl interface yet.]

　　文件下载

client.php

<?php

function receiveResponse($curlHandle, $xmldata)
{
    var_dump($xmldata);
    flush();
    ob_flush();
    //需要准确返回读取的字节数
    return strlen($xmldata);
}

$ch = curl_init('http://jksong.cm/api.php?id=1&name=jksong');
curl_setopt($ch, CURLOPT_WRITEFUNCTION, "receiveResponse");
curl_exec($ch);
curl_close($ch);

api.php

<?php

for ($i=0;$i<10;$i++){

    var_dump(str_repeat("a", 20));
//    var_dump(time());

    sleep(1);

    ob_flush();
    flush();
}

JAVA httpclient 实现

CloseableHttpClient client = HttpClients.createDefault();
        HttpGet httpGet = new HttpGet("http://jksong.cm/api.php");
        try (CloseableHttpResponse response1 = client.execute(httpGet)) {
            final HttpEntity entity = response1.getEntity();
            if (entity != null) {
                try (InputStream inputStream = entity.getContent()) {
                    // do something useful with the stream
                    //System.out.println(IOUtils.toString(inputStream));

                    BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream);
                    
                    int b = 0;
                    while ((b = inputStream.read()) != -1) {
                        System.out.println("===>"+(char)b);
                    }

                }
            }
        }

 node

http

var http = require('http');
var body = '';

http.get("http://jksong.cm/api.php", function(res) {
    res.on('data', function(chunk) {
        body += chunk;
        console.log(chunk);
    });
    res.on('end', function() {
        // all data has been downloaded
    });
});

axios

var fs = require('fs');
var http = require('axios');
// 获取远端图片
http({
    method: 'get',
    url: 'http://jksong.cm/api.php',
    responseType: 'stream'
})
    .then(function (response) {
        // console.log(response.data.read);
        response.data.pipe(fs.createWriteStream('ada_lovelace.txt'))
    });

　　文件上传

PHP

php中form-data不支持 php://input 方式读取，可以采用 binary或urlencode 方式 直接上传文件

验证：

api.php

//可以实时读取最新上传的数据
$handle = fopen("php://input", "rb");
$contents = '';
while (!feof($handle)) {
    $body = fread($handle, 8192);
    var_dump($body);
    file_put_contents("/tmp/a.txt", $body, FILE_APPEND);
}
fclose($handle);

curl请求

#注意发送速率，由于服务端缓冲区的存在，太小时不容易测出实时接收的效果
curl --limit-rate 100k -L -X GET 'http://jksong.cm/api.php' -H 'Content-Type: application/x-rar-compressed' --data-binary '@/tmp/file.data'
curl --limit-rate 100k -L -X GET 'http://jksong.cm/api.php' -H 'Content-Type: application/x-rar-compressed' --data-urlencode 'song@/tmp/file.data'

JAVA

request.getInputStream() 中的流可以实时获取，这样就可以边上传，边进行转发

如果获取 inputstream 原因参看   接口中转stream传输

由于multipart方式上传不够优雅

　　1、要不就是不可以实时获取到流，必须要等到全上传完毕 【MultipartFile 参数方式】

　　2、要不就是关闭框架中 multipart【MultipartResolver】 解析的功能 【spring.servlet.multipart.enabled=false】，这样会导致 MultipartFile 参数方式全部失败

　　    并且获取到流中的body是按照 multipart 方式处理的，有boundary边界等新，非原始的文件信息

 

更好的处理文件上传，直接使用binary方式，可以直接流中获取文件的原始内容

curl --limit-rate 100k  -X POST 'http://127.0.0.1:8080/upload' -H 'Content-Type: application/x-sql' --data-binary '@/Users/siqi/Desktop/file.sql'>out.txt

控制器示例

    @RequestMapping(value = "/upload")
    public IdNamePair upload(HttpServletRequest request)
        throws IOException {

　　　　　//实时读取上传的文件信息
        ServletInputStream inputStream = request.getInputStream();

        StringBuffer sb = new StringBuffer();
        ServletInputStream fis = inputStream;
        int b = 0;
        while ((b = fis.read()) != -1) {
            sb.appendCodePoint(b);
        }
        inputStream.close();
        System.out.println(sb.toString());

        IdNamePair idNamePair = new IdNamePair();
        idNamePair.setId(123);
        idNamePair.setName("蓝银草");

        return idNamePair;
    }

php请求示例

<?php


$curl = curl_init();

curl_setopt_array($curl, array(
    CURLOPT_URL => "http://127.0.0.1:8080/upload",
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_ENCODING => "",
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_TIMEOUT => 0,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
    CURLOPT_CUSTOMREQUEST => "POST",
    CURLOPT_POSTFIELDS => file_get_contents("/Users/siqi/Desktop/file.sql"),
    CURLOPT_HTTPHEADER => array(
        "Content-Type: application/octet-stream",
    ),
));

$response = curl_exec($curl);

curl_close($curl);
echo $response;

  

参考：

https://stackoverflow.com/questions/16462088/php-curl-and-stream-forwarding

https://www.php.net/manual/zh/function.curl-setopt.php

https://www.ashleysheridan.co.uk/blog/Creating+a+Streaming+Proxy+with+PHP#stream-the-response