Tencent Cloud Cos Get File Operation Practice--Go SDK

question

Tencent Cloud's object storage can be regarded as an online KV, so I tried to use it because of the recent demand. I encountered some problems when using it, specifically cos.BucketGetOptions the use of Delimiter in .

• The Delimiter used in the list object is "/"
• The Delimiter used in the download object is ""

According to the official statement, delimiter means to list the objects in the current directory. When it is set to empty, it lists all objects. I don't know exactly what this all means, so I put it into practice.

Practice and Code

The code is similar to Tencent's cos Go SDK to use and learn , and there should be no difficulty in itself.
The code contains the following:

1. Environment Construction: Build a complex file environment by uploading files in batches, providing a basis for subsequent acquisition and download of files
2. File download: Test whether the content of the file is as expected
3. File batch download: test the effect of the delimiter option on the download
4. file listing: test the effect of the delimiter option in file listing

environmental construction

Use the following functions to construct the environment. Note: Tencent Cloud Object Storage is not a free service. Pay attention to the cost when using it. The file operations of the example in this article will theoretically not exceed the free limit, but please pay attention when modifying the code.

func main(){
    u, _ := url.Parse("https://wtytest-1252789333.cos.ap-guangzhou.myqcloud.com")
b := &cos.BaseURL{BucketURL: u}
c := cos.NewClient(b, &http.Client{
Transport: &cos.AuthorizationTransport{
https://console.cloud.tencent.com/cam/capi
SecretID: "secretID",
SecretKey: "secret",

Transport: &debug.DebugRequestTransport{
RequestHeader:  true,
RequestBody:    true,
ResponseHeader: true,
ResponseBody:   false,
},
},
})

uploadFileToCos(0, "", c)
}

func uploadFileToCos(depth int, prefix string, c *cos.Client) {
if depth > 5 {
return
}
for i := 0; i < 3; i++ {
s := rand.Intn(100)
filePath := prefix + "file" + strconv.Itoa(s)
err := uploadFileByName(filePath, "file"+strconv.Itoa(s), c)
if err != nil {
log_status(err)
}
}

for i := 0; i < 2; i++ {
s := rand.Intn(100)
filePrefix := prefix + "path" + strconv.Itoa(s) + "/"
uploadFileToCos(depth+1, filePrefix, c)
}
}

copy

file download

func getFileByName(name string, c *cos.Client) {
resp, err := c.Object.Get(context.Background(), name, nil)
log_status(err)
bs, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
fmt.Println("content:", string(bs))
}

copy

File batch download

func main(){
const NoDelimiter = ""
const FolderDelimiter = "/"
const path1 = "path59/path33/path43/" 
    const path2 = "path59/path33/path43" 
getAllFileInPrefix(path1, NoDelimiter, c)
getAllFileInPrefix(path1, FolderDelimiter, c)
}

func getAllFileInPrefix(prefix string, delimiter string, c *cos.Client) {
totalFileCount := 0
isTruncated := true
marker := ""
count := 0
for isTruncated {
fmt.Printf("count:%d marker:%s isTruncated:%v", count, marker, isTruncated)
count++

opt := &cos.BucketGetOptions{
Prefix:    prefix,
Marker:    marker,
Delimiter: delimiter,
}
v, _, err := c.Bucket.Get(context.Background(), opt)
if err != nil {
log_status(err)
return
}
for _, content := range v.Contents {
resp, err := c.Object.Get(context.Background(), content.Key, nil)
if err != nil {
log_status(err)
return
}
bs, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
fmt.Printf("key:%s content:%s\n", content.Key, string(bs))
totalFileCount++
}
marker = v.NextMarker
isTruncated = v.IsTruncated
}
fmt.Println("total file count is ", totalFileCount, " with count:", count)
}

copy
When the prefix of the cos key has "/" at the end, 21 files can be obtained with NoDelimiter, and 3 files can be obtained with FolderDelimiter.
When there is no "/" at the end of the prefix of the cos key, 21 files can be obtained with NoDelimiter, and 0 files can be obtained with FolderDelimiter.
This shows that when the delimiter is "/", it can be understood as a folder mode, and the folder mode only works on the simulated folder (that is, there is a "/" at the end of the prefix).
If delimiter is an empty string, it will try to match objects with the same prefix.

file list

The file listing is somewhat similar to the logic at the beginning of the file download, but with the commonPrefix added. The test code is the same as before and will not be repeated here.

func listAllFileInPrefix(prefix string, delimiter string, c *cos.Client) {
var marker string
opt := &cos.BucketGetOptions{
Prefix:    prefix,
Delimiter: delimiter,
MaxKeys:   1000,
}
totalFileCount := 0
commonPrefixCount := 0
isTruncated := true
for isTruncated {
opt.Marker = marker
v, _, err := c.Bucket.Get(context.Background(), opt)
if err != nil {
fmt.Println(err)
break
}
for _, content := range v.Contents {
fmt.Printf("Object: %v\n", content.Key)
}
totalFileCount += len(v.Contents)

for _, commonPrefix := range v.CommonPrefixes {
fmt.Printf("CommonPrefixes: %v\n", commonPrefix)
}
commonPrefixCount += len(v.CommonPrefixes)
isTruncated = v.IsTruncated 
marker = v.NextMarker       
}
fmt.Println("total file count", totalFileCount, " commonPrefix:", commonPrefixCount)
}

copy
When the prefix of the cos key has "/" at the end, using NoDelimiter can get 21 files without truncating the path, and using FolderDelimiter can get 3 files and truncating 2 paths (two subdirectories under the current directory).
When there is no "/" at the end of the prefix of the cos key, using NoDelimiter can get 21 files without truncating the path, and using FolderDelimiter can get 0 files, but truncating a path ( path59/path33/path43/ ).

Summarize

The delimiter of object storage can enable folder mode to simulate folders in general operating systems, which is generally useful.