Skip to content

funzip and large files #10

Open
Open
funzip and large files#10

Description

@lazureykis
lazureykis
opened on Aug 26, 2022

Hi there!

I've faced an issue with funzip. It crashes when uncompressed data reaches about 2Gb.

Here is an example:

> curl -s https://data.binance.vision/data/spot/monthly/trades/BTCUSDT/BTCUSDT-trades-2020-07.zip | funzip > ok.csv


> curl -s https://data.binance.vision/data/spot/monthly/trades/BTCUSDT/BTCUSDT-trades-2020-08.zip | funzip > fail.csv                                                                 

funzip error: invalid compressed data--length error

Is there something I can do to make it work? I'm not an expert, but maybe there is a simple solution to my problem.

Activity

  1. madler commented on Oct 10, 2022

    @madler
    madler
    on Oct 10, 2022
    Owner

    It looks like funzip hasn't been touched in 20 years. It does not support the Zip64 format being used by your larger file. I will look into updating funzip.

  2. madler commented on Oct 10, 2022

    @madler
    madler
    on Oct 10, 2022
    Owner

    I have added commit 0b82c20 to workaround this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions