Automate usage of fsspec hdfs URLs via new pyarrow HDFS API (!5) · Merge requests · repos / data-engineering / WMF Data Workflow Utils · GitLab

How to register an account on GitLab. To prevent spam, new accounts are locked until approved by an admin or the approver bot. You can also file an unlock request to expedite access.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Ottomata requested to merge fsspec_pyarrow into main Mar 01, 2022

fsspec_use_new_pyarrow_api - call this to make fsspec always use new pyarrow API with all hdfs:// URLs. This is only needed until https://github.com/fsspec/filesystem_spec/issues/874 is resolved.
set_hadoop_env_vars - sets needed env vars to work with new pyarrow HDFS API. This is also called by fsspec_use_new_pyarrow_api() by default.

https://phabricator.wikimedia.org/T300876