chiu
/
google-api-ruby-client
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Recover from non-Ranged responses without rewinding 

Not all IO objects know how to #rewind themselves. For example, Ruby
pipes (returned by `IO.pipe`) do implement #rewind, but they will throw
an error if you try to call it.

  rd, wr = IO.pipe
  wr.rewind # Errno::ESPIPE: Illegal seek

But we don't need to rewind and overwrite the IO object if we didn't get
the Ranged response we expected, we could instead wait out the content
that has already been downloaded, and start appending again once we
reached where we left off. This is what this commit does.
This commit is contained in:
Janko Marohnić 2017-06-23 11:22:37 +02:00
parent e7f64be149
commit 0eeb3ee9f9
1 changed files with 18 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
lib/google/apis/core/download.rb
View File

@ -67,12 +67,11 @@ module Google
def execute_once(client, &block) def execute_once(client, &block)
request_header = header.dup request_header = header.dup
apply_request_options(request_header) apply_request_options(request_header)
download_offset = nil
check_if_rewind_needed = false
if @offset > 0 if @offset > 0
logger.debug { sprintf('Resuming download from offset %d', @offset) } logger.debug { sprintf('Resuming download from offset %d', @offset) }
request_header[RANGE_HEADER] = sprintf('bytes=%d-', @offset) request_header[RANGE_HEADER] = sprintf('bytes=%d-', @offset)
check_if_rewind_needed = true
end end
http_res = client.get(url.to_s, http_res = client.get(url.to_s,
@ -80,17 +79,24 @@ module Google
header: request_header, header: request_header,
follow_redirect: true) do |res, chunk| follow_redirect: true) do |res, chunk|
status = res.http_header.status_code.to_i status = res.http_header.status_code.to_i
if OK_STATUS.include?(status) next unless OK_STATUS.include?(status)
if check_if_rewind_needed && status != 206
# Oh no! Requested a chunk, but received the entire content download_offset ||= (status == 206 ? @offset : 0)
# Attempt to rewind the stream download_offset += chunk.bytesize
@download_io.rewind
check_if_rewind_needed = false if download_offset - chunk.bytesize == @offset
end next_chunk = chunk
# logger.debug { sprintf('Writing chunk (%d bytes, %d total)', chunk.length, bytes_read) } else
@download_io.write(chunk) # Oh no! Requested a chunk, but received the entire content
@offset += chunk.length chunk_index = @offset - (download_offset - chunk.bytesize)
next_chunk = chunk.byteslice(chunk_index..-1)
next if next_chunk.nil?
end end
# logger.debug { sprintf('Writing chunk (%d bytes, %d total)', chunk.length, bytes_read) }
@download_io.write(next_chunk)
@offset += next_chunk.bytesize
end end
@download_io.flush @download_io.flush