Skip to content

gs.md

Latest commit

 

History

History
1011 lines (716 loc) · 32.7 KB

gs.md

File metadata and controls

1011 lines (716 loc) · 32.7 KB
Raw

gs

import "github.com/c2fo/vfs/v7/backend/gs"

Package gs - Google Cloud Storage VFS implementation.

Usage

Rely on github.com/c2fo/vfs/v7/backend

import(
    "github.com/c2fo/vfs/v7/backend"
    "github.com/c2fo/vfs/v7/backend/gs"
)

func UseFs() error {
    fs := backend.Backend(gs.Scheme)
    ...
}

Or call directly:

import "github.com/c2fo/vfs/v7/backend/gs"

func DoSomething() {
    fs := gs.NewFileSystem()
    ...
}

gs can be configured with functional options passed to NewFileSystem:

func DoSomething() {
    ...

    // to use your own "context"
    ctx := context.Background()
    fs := gs.NewFileSystem(gs.WithContext(ctx))

    // to pass in client options
    fs = gs.NewFileSystem(
        gs.WithOptions(
            gs.Options{
                CredentialFile: "/root/.gcloud/account.json",
                Scopes:         []string{storage.ScopeReadOnly},
                // when Scopes is empty, the storage client's default scopes are used
            },
        ),
    )

    // to pass specific client, for instance no-auth client
    client, _ := storage.NewClient(ctx, option.WithoutAuthentication())
    fs = gs.NewFileSystem(gs.WithClient(client))
}

Authentication

Authentication, by default, occurs automatically when Client() is called. It looks for credentials in the following places, preferring the first location found:

  1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable
  2. A JSON file in a location known to the gcloud command-line tool. On Windows, this is %APPDATA%/gcloud/application_default_credentials.json. On other systems, $HOME/.config/gcloud/application_default_credentials.json.
  3. On Google App Engine it uses the appengine.AccessToken function.
  4. On Google Compute Engine and Google App Engine Managed VMs, it fetches credentials from the metadata server.

See https://cloud.google.com/docs/authentication/production for more auth info

Service Account Impersonation

Set ImpersonateServiceAccount to have Client() build the storage client with short-lived tokens for that service account, requested by a base identity, instead of using the base identity's own access:

fs := gs.NewFileSystem(
    gs.WithOptions(
        gs.Options{
            ImpersonateServiceAccount: "target-sa@my-project.iam.gserviceaccount.com",
        },
    ),
)

The same option decoded from JSON:

{"impersonateServiceAccount": "target-sa@my-project.iam.gserviceaccount.com"}

The base identity is CredentialFile if set, otherwise Application Default Credentials resolved as described above. ADC can be any supported credential type, including an external_account (Workload Identity Federation) config referenced by GOOGLE_APPLICATION_CREDENTIALS, so impersonation also works outside GKE and GCE. The base identity needs the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) on the target service account.

Impersonated tokens use the storage client's default scopes (devstorage.full_control and cloud-platform). Endpoint and Scopes are ignored when ImpersonateServiceAccount is set. Combining it with APIKey makes Client() return an error, because an API key can't impersonate a service account.

Copies between gs filesystems with the same ImpersonateServiceAccount and the same CredentialFile (or neither has one) are done server-side.

Anyone who can set ImpersonateServiceAccount can act as any service account the base identity is allowed to impersonate. If your application builds Options from user-supplied input, restrict which service accounts may be named.

See Also

See: https://github.com/googleapis/google-cloud-go/tree/master/storage

Index

Constants

Scheme defines the file system type.

const Scheme = "gs"

func GetClient

func GetClient(ctx context.Context, opts Options) (*storage.Client, error)

GetClient returns a new Google Cloud Storage client with the provided context and Options.

func WithClient

func WithClient(c *storage.Client) options.NewFileSystemOption[FileSystem]

WithClient returns clientOpt implementation of NewFileOption

WithClient is used to explicitly specify a Client to use for the filesystem. The client is used to interact with the S3 service.

func WithContext

func WithContext(ctx context.Context) options.NewFileSystemOption[FileSystem]

WithContext returns contextOpt implementation of NewFileOption

WithContext is used to specify a context for the filesystem. The context is used to control the lifecycle of the filesystem.

func WithOptions

func WithOptions(opts Options) options.NewFileSystemOption[FileSystem]

WithOptions returns optionsOpt implementation of NewFileOption

WithOptions is used to specify options for the filesystem. The options are used to configure the filesystem.

func WithRetryer

func WithRetryer(retryer Retryer) options.NewFileSystemOption[FileSystem]

WithRetryer returns retryerOpt implementation of NewFileOption

WithRetryer is used to specify a Retryer for the filesystem. The retryer is used to retry operations on the filesystem.

type BucketHandle

BucketHandle is an interface which contains a subset of the functions provided by storage.BucketHandler. Any function normally called directly by storage.BucketHandler should be added to this interface to allow for proper retry wrapping of the functions which call the GCS API.

type BucketHandle interface {
    Attrs(ctx context.Context) (*storage.BucketAttrs, error)
}

type BucketHandleWrapper

BucketHandleWrapper is a unique, wrapped type which should mimic the behavior of BucketHandler, but with modified return types. Each function that returns a sub type that also should be wrapped should be added to this interface with the 'Wrapped' prefix.

type BucketHandleWrapper interface {
    BucketHandle
    WrappedObjects(ctx context.Context, q *storage.Query) ObjectIteratorWrapper
}

type ClientCreator

ClientCreator defines an interface for creating a new Google Cloud Storage client.

type ClientCreator interface {
    NewClient(ctx context.Context, opts ...option.ClientOption) (*storage.Client, error)
}

type Copier

Copier implements the CopierWrapper interface.

type Copier struct {
    Retry Retryer
    // contains filtered or unexported fields
}

func (*Copier) ContentType

func (c *Copier) ContentType(val string)

ContentType is the MIME type of the object's content.

func (*Copier) Run

func (c *Copier) Run(ctx context.Context) (*storage.ObjectAttrs, error)

Run performs the copy, wrapped in a retry

type CopierWrapper

CopierWrapper is an interface which contains a subset of the functions provided by storage.Copier.

type CopierWrapper interface {
    Run(ctx context.Context) (*storage.ObjectAttrs, error)
    ContentType(string)
}

type File

File implements vfs.File interface for GS fs.

type File struct {
    // contains filtered or unexported fields
}

func (*File) Close

func (f *File) Close() error

Close commits any writes, either from the GCS writer stream or from a tempfile (in the case where Seek or Read are called after Write). It then cleans up any open resources and resets the file's state.

func (*File) CopyToFile

func (f *File) CopyToFile(file vfs.File) (err error)

CopyToFile puts the contents of File into the target vfs.File passed in. Uses the GCS CopierFrom method if the target file is also on GCS, otherwise uses io.CopyBuffer. This method should be called on a closed file or a file with 0 cursor position to avoid errors.

func (*File) CopyToLocation

func (f *File) CopyToLocation(location vfs.Location) (vfs.File, error)

CopyToLocation creates a copy of *File, using the file's current name as the new file's name at the given location. If the given location is also GCS, the GCS API for copying files will be utilized, otherwise, standard io.Copy will be done to the new file.

func (*File) Delete

func (f *File) Delete(opts ...options.DeleteOption) error

Delete clears any local temp file, or write buffer from read/writes to the file, then makes a DeleteObject call to GCS for the file. If delete.AllVersions option is provided, DeleteObject call is made to GCS for each version of the file. Returns any error returned by the API.

func (*File) Exists

func (f *File) Exists() (bool, error)

Exists returns a boolean of whether the object exists in GCS.

func (*File) LastModified

func (f *File) LastModified() (*time.Time, error)

LastModified returns the 'Updated' property from the GCS attributes.

func (*File) Location

func (f *File) Location() vfs.Location

Location returns a Location instance for the file's current location.

func (*File) MoveToFile

func (f *File) MoveToFile(file vfs.File) error

MoveToFile puts the contents of File into the target vfs.File passed in using File.CopyToFile. If the copy succeeds, the source file is deleted. Any errors from the copy or delete are returned.

func (*File) MoveToLocation

func (f *File) MoveToLocation(location vfs.Location) (vfs.File, error)

MoveToLocation works by first calling File.CopyToLocation(vfs.Location) then, if that succeeds, it deletes the original file, returning the new file. If the copy process fails the error is returned, and the Delete isn't called. If the call to Delete fails, the error and the file generated by the copy are both returned.

func (*File) Name

func (f *File) Name() string

Name returns the file name.

func (*File) Path

func (f *File) Path() string

Path returns full path with leading slash of the GCS file key.

func (*File) Read

func (f *File) Read(p []byte) (n int, err error)

Read implements the standard for io.Reader.

func (*File) Seek

func (f *File) Seek(offset int64, whence int) (int64, error)

Seek implements the standard for io.Seeker.

func (*File) Size

func (f *File) Size() (uint64, error)

Size returns the 'Size' property from the GCS attributes.

func (*File) String

func (f *File) String() string

String returns the file URI string.

func (*File) Touch

func (f *File) Touch() error

Touch creates a zero-length file on the vfs.File if no File exists. Update File's last modified timestamp. Returns error if unable to touch File.

func (*File) URI

func (f *File) URI() string

URI returns a full GCS URI string of the file.

func (*File) Write

func (f *File) Write(data []byte) (int, error)

Write implements the standard for io.Writer. Note that writes are not committed to GCS until Close() is called.

type FileSystem

FileSystem implements vfs.FileSystem for the GCS file system.

type FileSystem struct {
    // contains filtered or unexported fields
}

func NewFileSystem

func NewFileSystem(opts ...options.NewFileSystemOption[FileSystem]) *FileSystem

NewFileSystem initializer for FileSystem struct accepts google cloud storage client and returns FileSystem or error.

func (*FileSystem) Client

func (fs *FileSystem) Client() (*storage.Client, error)

Client returns the underlying google storage client, creating it, if necessary See Overview for authentication resolution

func (*FileSystem) Name

func (fs *FileSystem) Name() string

Name returns "Google Cloud Storage"

func (*FileSystem) NewFile

func (fs *FileSystem) NewFile(authorityStr, filePath string, opts ...options.NewFileOption) (vfs.File, error)

NewFile function returns the gcs implementation of vfs.File.

func (*FileSystem) NewLocation

func (fs *FileSystem) NewLocation(authorityStr, locPath string) (loc vfs.Location, err error)

NewLocation function returns the GCS implementation of vfs.Location.

func (*FileSystem) Retry

func (fs *FileSystem) Retry() vfs.Retry

Retry will return a retryer provided via options, or a no-op if none is provided.

Deprecated: This method is deprecated and will be removed in a future release.

func (*FileSystem) Scheme

func (fs *FileSystem) Scheme() string

Scheme return "gs" as the initial part of a file URI ie: gs://

func (*FileSystem) WithClient

func (fs *FileSystem) WithClient(client *storage.Client) *FileSystem

WithClient passes in a google storage client and returns the file system (chainable)

Deprecated: This method is deprecated and will be removed in a future release. Use WithOptions option:

fs := gs.NewFileSystem(gs.WithOptions(opts))

instead of:

fs := gs.NewFileSystem().WithOptions(opts)

func (*FileSystem) WithContext

func (fs *FileSystem) WithContext(ctx context.Context) *FileSystem

WithContext passes in user context and returns the file system (chainable)

Deprecated: This method is deprecated and will be removed in a future release. Use WithContext option:

fs := gs.NewFileSystem(WithContext(ctx))

instead of:

fs := s3.NewFileSystem().WithContext(ctx)

func (*FileSystem) WithOptions

func (fs *FileSystem) WithOptions(opts vfs.Options) *FileSystem

WithOptions sets options for client and returns the file system (chainable)

Deprecated: This method is deprecated and will be removed in a future release. Use WithClient option:

fs := gs.NewFileSystem(gs.WithClient(client))

instead of:

fs := gs.NewFileSystem().WithClient(client)

type Location

Location implements vfs.Location for gs fs.

type Location struct {
    // contains filtered or unexported fields
}

func (*Location) Authority

func (l *Location) Authority() authority.Authority

Authority returns the Authority for the Location.

func (*Location) ChangeDir

func (l *Location) ChangeDir(relativePath string) error

ChangeDir changes the current location's path to the new, relative path.

Deprecated: Use NewLocation instead:

loc, err := loc.NewLocation("../../")

func (*Location) DeleteFile

func (l *Location) DeleteFile(fileName string, opts ...options.DeleteOption) error

DeleteFile deletes the file at the given path, relative to the current location.

func (*Location) Exists

func (l *Location) Exists() (bool, error)

Exists returns whether the location exists or not. In the case of an error, false is returned.

func (*Location) FileSystem

func (l *Location) FileSystem() vfs.FileSystem

FileSystem returns the GCS file system instance.

func (*Location) List

func (l *Location) List() ([]string, error)

List returns a list of file name strings for the current location.

func (*Location) ListByPrefix

func (l *Location) ListByPrefix(filenamePrefix string) ([]string, error)

ListByPrefix returns a slice of file base names and any error, if any List functions return only file basenames

func (*Location) ListByRegex

func (l *Location) ListByRegex(regex *regexp.Regexp) ([]string, error)

ListByRegex returns a list of file names at the location which match the provided regular expression.

func (*Location) ListEntries

func (l *Location) ListEntries(ctx context.Context, opts ...options.ListOption) iter.Seq2[vfs.Entry, error]

ListEntries lists the location's contents as a stream of entries in byte order, implementing vfs.EntryLister. Use list.Entries to list a location with options and have every option applied.

Objects are read from one GCS object iterator, which fetches a page of up to 1,000 objects only when the listing needs more, so a listing that stops early stops its requests. The size and modification time of every file come from the listing itself, so reading them makes no further request, and Entry.Sys returns the listed *storage.ObjectAttrs. A non-recursive listing asks GCS to roll objects up at "/", as does a recursive one whose MaxDepth stops one level inside the directory of its prefix; other recursive listings do not. A recursive listing that skips a directory, because of MaxDepth, DirFilter or a Glob, starts a new iterator after it instead of reading the objects inside it. GCS has no directories in flat buckets, so a directory entry exists wherever an object name has a "/" below the listed location, whether or not a folder placeholder object exists.

A 403 response is reported as an error matching fs.ErrPermission. A bucket that does not exist is an error, not an empty listing, so a misspelled bucket is not mistaken for an empty one.

func (*Location) NewFile

func (l *Location) NewFile(relFilePath string, opts ...options.NewFileOption) (vfs.File, error)

NewFile returns a new file instance at the given path, relative to the current location.

func (*Location) NewLocation

func (l *Location) NewLocation(relativePath string) (vfs.Location, error)

NewLocation creates a new location instance relative to the current location's path.

func (*Location) Path

func (l *Location) Path() string

Path returns the path of the file at the current location, starting with a leading '/'

func (*Location) String

func (l *Location) String() string

String returns the full URI of the location.

func (*Location) URI

func (l *Location) URI() string

URI returns a URI string for the GCS location.

func (*Location) Volume

func (l *Location) Volume() string

Volume returns the GCS bucket name.

Deprecated: Use Authority instead.

authStr := loc.Authority().String()

type ObjectHandleCopier

ObjectHandleCopier is a unique, wrapped type which should mimic the behavior of ObjectHandler, but with modified return types. Each function that returns a sub type that also should be wrapped should be added to this interface with the 'Wrapped' prefix.

type ObjectHandleCopier interface {
    ObjectHandleWrapper
    WrappedCopierFrom(src *storage.ObjectHandle) CopierWrapper
    ObjectHandle() *storage.ObjectHandle
}

type ObjectHandleWrapper

ObjectHandleWrapper is an interface which contains a subset of the functions provided by storage.ObjectHandler. Any function normally called directly by storage.ObjectHandler should be added to this interface to allow for proper retry wrapping of the functions which call the GCS API.

type ObjectHandleWrapper interface {
    NewWriter(ctx context.Context) *storage.Writer
    NewReader(ctx context.Context) (*storage.Reader, error)
    NewRangeReader(ctx context.Context, offset, length int64) (*storage.Reader, error)
    Attrs(ctx context.Context) (*storage.ObjectAttrs, error)
    Delete(ctx context.Context) error
    Update(ctx context.Context, attrs storage.ObjectAttrsToUpdate) (*storage.ObjectAttrs, error)
}

type ObjectIteratorWrapper

ObjectIteratorWrapper is an interface which contains a subset of the functions provided by storage.ObjectIterator.

type ObjectIteratorWrapper interface {
    Next() (*storage.ObjectAttrs, error)
}

type Options

Options holds Google Cloud Storage -specific options. Currently only client options are used.

type Options struct {
    APIKey         string   `json:"apiKey,omitempty"`
    CredentialFile string   `json:"credentialFilePath,omitempty"`
    Endpoint       string   `json:"endpoint,omitempty"`
    Scopes         []string `json:"withoutAuthentication,omitempty"`
    FileBufferSize int      `json:"fileBufferSize,omitempty"` // Buffer Size In Bytes Used with utils.TouchCopyBuffered
    // ImpersonateServiceAccount is the email of a service account to impersonate. The base identity is CredentialFile
    // if set, otherwise Application Default Credentials. It can't be combined with APIKey, and Endpoint and Scopes are
    // ignored when it's set.
    ImpersonateServiceAccount string `json:"impersonateServiceAccount,omitempty"`
}

type RetryBucketHandler

RetryBucketHandler implements the BucketHandle interface

type RetryBucketHandler struct {
    Retry Retryer
    // contains filtered or unexported fields
}

func (*RetryBucketHandler) Attrs

func (r *RetryBucketHandler) Attrs(ctx context.Context) (*storage.BucketAttrs, error)

Attrs accepts a context and returns bucket attrs wrapped in a retry

func (*RetryBucketHandler) WrappedObjects

func (r *RetryBucketHandler) WrappedObjects(ctx context.Context, q *storage.Query) ObjectIteratorWrapper

WrappedObjects returns an iterator over the objects in the bucket that match the Query q, all wrapped in a retry. If q is nil, no filtering is done.

type RetryObjectHandler

RetryObjectHandler implements the ObjectHandleCopier interface (which also is composed with ObjectHandleWrapper)

type RetryObjectHandler struct {
    Retry Retryer
    // contains filtered or unexported fields
}

func (*RetryObjectHandler) Attrs

func (r *RetryObjectHandler) Attrs(ctx context.Context) (*storage.ObjectAttrs, error)

Attrs represents the metadata for a Google Cloud Storage (GCS) object, wrapped in a retry.

func (*RetryObjectHandler) Delete

func (r *RetryObjectHandler) Delete(ctx context.Context) error

Delete deletes the single specified object, wrapped in a retry.

func (*RetryObjectHandler) NewRangeReader

func (r *RetryObjectHandler) NewRangeReader(ctx context.Context, offset, length int64) (*storage.Reader, error)

NewRangeReader creates a new Reader to read the contents of the object starting at an offset, wrapped in a retry. ErrObjectNotExist will be returned if the object is not found.

func (*RetryObjectHandler) NewReader

func (r *RetryObjectHandler) NewReader(ctx context.Context) (*storage.Reader, error)

NewReader creates a new Reader to read the contents of the object, wrapped in a retry. ErrObjectNotExist will be returned if the object is not found.

func (*RetryObjectHandler) NewWriter

func (r *RetryObjectHandler) NewWriter(ctx context.Context) *storage.Writer

NewWriter returns a storage Writer that writes to the GCS object associated with this ObjectHandle, wrapped in a retry.

func (*RetryObjectHandler) ObjectHandle

func (r *RetryObjectHandler) ObjectHandle() *storage.ObjectHandle

ObjectHandle returns the underlying GCS object handle.

func (*RetryObjectHandler) Update

func (r *RetryObjectHandler) Update(ctx context.Context, attrs storage.ObjectAttrsToUpdate) (*storage.ObjectAttrs, error)

Update will update the metadata for a Google Cloud Storage (GCS) object, wrapped in a retry. Here we can re-use the objectAttributeRetry() func because it has the same signature.

func (*RetryObjectHandler) WrappedCopierFrom

func (r *RetryObjectHandler) WrappedCopierFrom(src *storage.ObjectHandle) CopierWrapper

WrappedCopierFrom creates a Copier that can copy src to dst, wrapped in a retry. You can immediately call Run on the returned Copier, or you can configure it first.

type RetryObjectIterator

RetryObjectIterator implements the ObjectIteratorWrapper interface

type RetryObjectIterator struct {
    Retry Retryer
    // contains filtered or unexported fields
}

func (*RetryObjectIterator) Next

func (r *RetryObjectIterator) Next() (*storage.ObjectAttrs, error)

Next returns the next result, wrapped in retry. Its second return value is iterator.Done if there are no more results. Once Next returns iterator.Done, all subsequent calls will return iterator.Done.

If Query.Delimiter is non-empty, some of the ObjectAttrs returned by Next will have a non-empty Prefix field, and a zero value for all other fields. These represent prefixes.

type Retryer

Retryer is a function that retries a function

type Retryer func(wrapped func() error) error

Generated by gomarkdoc