Automate usage of fsspec hdfs URLs via new pyarrow HDFS API
-
fsspec_use_new_pyarrow_api - call this to make fsspec always use new pyarrow API with all hdfs:// URLs. This is only needed until https://github.com/fsspec/filesystem_spec/issues/874 is resolved.
-
set_hadoop_env_vars - sets needed env vars to work with new pyarrow HDFS API. This is also called by fsspec_use_new_pyarrow_api() by default.