Remotely upload files to a Hugging Face repo using access token

Uploading Files to Hugging Face with Python

Hugging Face provides easy access to its repos for storing and sharing machine learning models through simple Python APIs. You can upload files without having them stored locally by leveraging the huggingface_hub Python library.

Prerequisites

• Python 3.x
• huggingface_hub library installed (pip install huggingface_hub)
• Hugging Face account and access token

Getting an Access Token

To upload files, you need an access token for your Hugging Face account:

1. Go to your Hugging Face profile settings
2. Click "New token" and give it a description like "Python uploads"
3. Copy the generated token - this will be used later

Uploading a File

With the access token, uploading a file just takes a few lines of code:

from huggingface_hub import login, upload_file

# Login using your access token 
login(token="hf_abc123") 

# File data to upload
file_data = read_binary_file(...) 

# Upload file data
upload_file(
    path_or_fileobj=file_data,
    path_in_repo="files/image.jpg",
    repo_id="username/repo", 
    repo_type="space"
)

The key steps:

1. Login with login() using your access token
2. Get the binary file data to upload
3. Call upload_file(), passing the data and destination file path

The file data can come from anywhere - downloading a remote file, reading a database BLOB, etc. As long as you have the binary data in memory, it can be uploaded.

Uploading from a URL

To upload a file from a remote URL, you can use the Requests library:

import requests
from huggingface_hub import login, upload_file

url = "https://example.com/image.jpg"

# Download file data
data = requests.get(url).content

# Upload data
upload_file(data, "images/remote_image.jpg", "username/repo") 

This provides a simple way to upload public files without needing local storage.

Getting File Data

There are a few options for getting the binary file data to upload:

User Input

You can ask the user to provide a file path and read the data:

file_path = input("Enter file path to upload: ")
file_data = open(file_path, "rb").read()

Web Form

If building a web app with Flask, you can get the data from the file upload:

from flask import request

@app.route("/upload", methods=["POST"])
def upload_file():
    file_data = request.files["file"].read() 
    # Upload file...

Downloading

You can download data from a remote URL:

import requests

url = "https://example.com/file.jpg" 
file_data = requests.get(url).content

This provides some options forsourcing the binary file data before uploading to Hugging Face. The huggingface_hub library handles the upload seamlessly once you have the data in memory.

Conclusion

The huggingface_hub Python library makes it easy to upload files to your Hugging Face repositories with just a few lines of code using your access token. By passing binary file data directly, you can upload files from anywhere without requiring them to be stored locally.