6 GCS Introduction and Stay Image Upload

6 GCS Introduction and Stay Image Upload

Goal

• A brief introduction to Google Cloud Storage.
• Create and configure your GCS buckets.
• Update the Staybooking project save stay images to GCS.
• Protect stay management API with token authentication.

Google Cloud Storage Introduction

Google Cloud Storage (https://cloud.google.com/storage/docs/) is a Powerful, Simple, and Cost-effective Object Storage Service. Same as the S3 service on AWS.

Features

• Durable

• Google Cloud Storage is designed for 99.999999999% durability. It stores data redundantly, with automatic checksums to ensure data integrity. With Multi-Regional storage, your data is maintained in geographically distinct locations.

• Available

• All storage classes offer very high availability around the world.

• Scalable

• Distributed storage

• Inexpensive

• Good to store unstructured data like images, files, videos, etc.

Question: Why not store media files in the database directly?

First, it’s much slower to store media files in a database than to store them in a file system directly. Second, media files will increase the size of the database a lot which will make it hard to maintain. Third, there’s no way to do database-related optimization(e.g. indexing) based on a binary file.

GCS is good for media files because it behaves like a file system, has good availability and durability, is less expensive, and also provides CDN service to serve files in edge servers to reduce loading latency.

GCS Bucket

• Buckets are the basic containers that hold your data. You can image buckets like the root directory in your file system. You can upload any file as an object into your bucket.
• The bucket name must be globally unique, there’s only one single namespace in GCS.
• You can apply labels to buckets to group buckets together.

GCS Object

• Objects are the individual pieces of data that you store in GCS. You can image an object like a file in your file system. You can put multiple objects in a single bucket.

• Objects are immutable, so you can’t make incremental changes, but only overwrite the whole object.

Create Your GCS Bucket

1. Back to the navigation menu and click Storage.

2. Click CREATE BUCKET to create a new bucket.

3. Under Name your bucket, choose a name for your bucket. You need to remember this bucket name since we will use it later in the code.

4. Under Choose where to store your data, first, choose Region instead of Multi-region, then choose a location that is near to your GCE instance. It’s not necessary to have your GCE instance and GCS bucket in the same region.

5. Under Choose a default storage class for your data, choose Standard.

6. Finally, click Create to create your bucket.

7. After creation, you should be redirected to the Bucket details page.

HTTP Multipart Request

An HTTP multipart request is an HTTP request that HTTP clients construct to send files and data over to an HTTP Server. It is commonly used by browsers and HTTP clients to upload files to the server. The content-type “multipart/form-data” should be used for submitting forms that contain files, non-ASCII data, and binary data.

More about multipart: https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2

How to send a multipart request in Postman?

Implement Stay Image Upload

Maven Update

1. Open your project in Intellij, and find the pom.xml file. Add the following dependencies.

     <dependencies>

  <!-- ... existing dependencies -->

  <!-- Only insert the following, do not change or touch other lines -->
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-storage</artifactId>
        <version>2.40.1</version>
    </dependency>
  <!-- Only insert the above, do not change or touch other lines -->
</dependencies>

2. Save your changes in pom.xml and run Maven Install to download the java libraries. Make sure you see the BUILD SUCCESS information from the console output.

Export GCP Credential

1. Go to the GCP console home page at https://console.cloud.google.com, under the navigation menu, click Credentials under APIs & Services.

2. Click Create credentials button, then select Service Account.

3. Pick any name you want, add a description and click CREATE AND CONTINUE.

4. Select the Project. Owner as the role of the service account, then click CONTINUE.

5. Click Done to finish the service account creation.

6. Back to the service account dashboard, click the account you’ve just created.

7. Under the KEY tab, click ADD KEY and select Create new key in the dropdown list. Make sure the browse download a JSON file to your machine.

8. Find the JSON file downloaded to your machine, rename it to credentials.json and drag it to the resources folder of your staybooking project.

Create StayImage Model

1. Go to the com.eve.staybooking.model package, create the StayImage class.

package com.eve.staybooking.model;

public class StayImage {
}

2. The content for StayImage class is simple, we need the URL as the link of the image after you upload it to GCS, and the stay field as a foreign key.

package com.eve.staybooking.model;

import com.fasterxml.jackson.annotation.JsonIgnore;
import jakarta.persistence.*;

import java.io.Serializable;

@Entity
@Table(name = "stay_image")
public class StayImage implements Serializable {
    private static final long serialVersionUID = 1L;

    @Id
    private String url;

    @ManyToOne
    @JoinColumn(name = "stay_id")
    @JsonIgnore
    private Stay stay;

    public StayImage() {}

    public StayImage(String url, Stay stay) {
        this.url = url;
        this.stay = stay;
    }

    public String getUrl() {
        return url;
    }

    public StayImage setUrl(String url) {
        this.url = url;
        return this;
    }

    public Stay getStay() {
        return stay;
    }

    public StayImage setStay(Stay stay) {
        this.stay = stay;
        return this;
    }
}

3. Go back to the Stay class to add a list of StayImage as a private field.

package com.eve.staybooking.model;

import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import jakarta.persistence.*;

import java.io.Serializable;
import java.util.List;
@Entity
@Table(name = "stay")
@JsonDeserialize(builder = Stay.Builder.class)
public class Stay implements Serializable {
    private static final long serialVersionUID = 1L;

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;
    private String name;
    private String description;
    private String address;

    @JsonProperty("guest_number")
    private int guestNumber;

    @ManyToOne
    @JoinColumn(name = "user_id")
    private User host;

    @JsonIgnore
    @OneToMany(mappedBy = "stay", cascade = CascadeType.ALL, fetch=FetchType.LAZY)
    private List<StayReservedDate> reservedDates;

    @OneToMany(mappedBy = "stay", cascade = CascadeType.ALL, fetch=FetchType.EAGER)
    private List<StayImage> images;

    public Stay() {}

    private Stay(Builder builder) {
        this.id = builder.id;
        this.name = builder.name;
        this.description = builder.description;
        this.address = builder.address;
        this.guestNumber = builder.guestNumber;
        this.host = builder.host;
        this.reservedDates = builder.reservedDates;
        this.images = builder.images;
    }

    public Long getId() {
        return id;
    }

    public String getName() {
        return name;
    }

    public String getDescription() {
        return description;
    }

    public String getAddress() {
        return address;
    }

    public int getGuestNumber() {
        return guestNumber;
    }

    public User getHost() {
        return host;
    }

    public List<StayReservedDate> getReservedDates() {
        return reservedDates;
    }

    public List<StayImage> getImages(){
        return images;
    }

    public Stay setImages(List<StayImage> images) {
        this.images = images;
        return this;
    }

    public static class Builder {

        @JsonProperty("id")
        private Long id;

        @JsonProperty("name")
        private String name;

        @JsonProperty("description")
        private String description;

        @JsonProperty("address")
        private String address;

        @JsonProperty("guest_number")
        private int guestNumber;

        @JsonProperty("host")
        private User host;

        @JsonProperty("dates")
        private List<StayReservedDate> reservedDates;

        @JsonProperty("images")
        private List<StayImage> images;

        public Builder setId(Long id) {
            this.id = id;
            return this;
        }

        public Builder setName(String name) {
            this.name = name;
            return this;
        }

        public Builder setDescription(String description) {
            this.description = description;
            return this;
        }

        public Builder setAddress(String address) {
            this.address = address;
            return this;
        }

        public Builder setGuestNumber(int guestNumber) {
            this.guestNumber = guestNumber;
            return this;
        }

        public Builder setHost(User host) {
            this.host = host;
            return this;
        }

        public Builder setReservedDates(List<StayReservedDate> reservedDates) {
            this.reservedDates = reservedDates;
            return this;
        }

        public Builder setImages(List<StayImage> images){
            this.images = images;
            return this;
        }

        public Stay build() {
            return new Stay(this);
        }









    }

}

Create Image Upload Service

1. Open the application.properties file and add a new variable named gcs.bucket. Remember to use your bucket name as the value.

spring.datasource.driver-class-name = com.mysql.cj.jdbc.Driver
spring.jpa.hibernate.ddl-auto = update
spring.jpa.show-sql = true
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
spring.datasource.url = jdbc:mysql://localhost:3306/staybooking?createDatabaseIfNotExist=true&serverTimezone=UTC
spring.datasource.username = root
spring.datasource.password = Eve123456
jwt.secret=secret
gcs.bucket=staybookingeve-bucket

gcs.bucket=YOUR_GCS_BUCKET_NAME

write in the same style

I got bug here at the first time.

2. Go to the com.eve.staybooking.exception package, create a new custom exception called GCSUploadException.

package com.eve.staybooking.exception;

public class GCSUploadException extends RuntimeException {
    public GCSUploadException(String message) {
        super(message);
    }
}

3. Go to the com.eve.staybooking.config package and create a class to provide GCS connections.

package com.eve.staybooking.config;

import com.google.auth.Credentials;
import com.google.auth.oauth2.ServiceAccountCredentials;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.io.IOException;

@Configuration
public class GoogleCloudStorageConfig {

    @Bean
    public Storage storage() throws IOException {
        Credentials credentials = ServiceAccountCredentials.fromStream(getClass().getClassLoader().getResourceAsStream("credentials.json"));
        return StorageOptions.newBuilder().setCredentials(credentials).build().getService();
    }
}

4. Go to the com.eve.staybooking.service package and create a new service class called ImageStorageService.

package com.eve.staybooking.service;

import org.springframework.stereotype.Service;

@Service
public class ImageStorageService {

}

5. Add the bucket name as the private field and inject the value from the application.properties.

package com.eve.staybooking.service;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class ImageStorageService {
    @Value("${gcs.bucket}")
    private String bucketName;
}

6. Add the storage private field and implement the save() method to save the given image to GCS.

package com.eve.staybooking.service;

import com.eve.staybooking.exception.GCSUploadException;

import com.google.cloud.storage.Acl;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.UUID;

@Service
public class ImageStorageService {
    @Value("${gcs.bucket}")
    private String bucketName;

    private Storage storage;

    @Autowired
    public ImageStorageService(Storage storage) {
        this.storage = storage;
    }

    public String save(MultipartFile file) throws GCSUploadException {
        String filename = UUID.randomUUID().toString();
        BlobInfo blobInfo = null;
        try {
            blobInfo = storage.createFrom(
                    BlobInfo
                            .newBuilder(bucketName, filename)
                            .setContentType("image/jpeg")
                            .setAcl(new ArrayList<>(Arrays.asList(Acl.of(Acl.User.ofAllUsers(), Acl.Role.READER))))
                            .build(),
                    file.getInputStream());
        } catch (IOException exception) {
            throw new GCSUploadException("Failed to upload file to GCS");
        }

        return blobInfo.getMediaLink();
    }
}

Integrate ImageUploadService with StayService

1. Open StayService class and add ImageStorageService as a private field.

package com.eve.staybooking.service;

import com.eve.staybooking.exception.StayNotExistException;
import com.eve.staybooking.model.Stay;
import com.eve.staybooking.model.User;
import com.eve.staybooking.repository.StayRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Isolation;
import org.springframework.transaction.annotation.Transactional;

import java.util.List;

@Service
public class StayService {
    private StayRepository stayRepository;

    private ImageStorageService imageStorageService;

    @Autowired
    public StayService(StayRepository stayRepository, ImageStorageService imageStorageService) {
        this.stayRepository = stayRepository;
        this.imageStorageService = imageStorageService;
    }
    public List<Stay> listByUser(String username) {
        return stayRepository.findByHost(new User.Builder().setUsername(username).build());
    }

    public Stay findByIdAndHost(Long stayId, String username) throws StayNotExistException {
        Stay stay = stayRepository.findByIdAndHost(stayId, new User.Builder().setUsername(username).build());
        if (stay == null) {
            throw new StayNotExistException("Stay doesn't exist");
        }
        return stay;
    }

    public void add(Stay stay) {
        stayRepository.save(stay);
    }

    @Transactional(isolation = Isolation.SERIALIZABLE)
    public void delete(Long stayId, String username) throws StayNotExistException {
        Stay stay = stayRepository.findByIdAndHost(stayId, new User.Builder().setUsername(username).build());
        if (stay == null) {
            throw new StayNotExistException("Stay doesn't exist");
        }
        stayRepository.deleteById(stayId);
    }


}

2. Update the add() method to support image saving.

package com.eve.staybooking.controller;

import com.eve.staybooking.model.Stay;
import com.eve.staybooking.service.StayService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
public class StayController {
    private StayService stayService;

    @Autowired
    public StayController(StayService stayService) {
        this.stayService = stayService;
    }

    @GetMapping(value = "/stays")
    public List<Stay> listStays(@RequestParam(name = "host") String hostName) {
        return stayService.listByUser(hostName);
    }

    @GetMapping(value = "/stays/id")
    public Stay getStay(
            @RequestParam(name = "stay_id") Long stayId,
            @RequestParam(name = "host") String hostName) {
        return stayService.findByIdAndHost(stayId, hostName);
    }

    @PostMapping("/stays")
    public void addStay(@RequestBody Stay stay) {
        stayService.add(stay);
    }

    @DeleteMapping("/stays")
    public void deleteStay(
            @RequestParam(name = "stay_id") Long stayId,
            @RequestParam(name = "host") String hostName) {
        stayService.delete(stayId, hostName);
    }

}

3. Upload stay upload API in StayController to read images from requests and pass them to StayService.

package com.eve.staybooking.controller;

import com.eve.staybooking.model.Stay;
import com.eve.staybooking.model.User;
import com.eve.staybooking.service.StayService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;

import java.util.List;

@RestController
public class StayController {
    private StayService stayService;

    @Autowired
    public StayController(StayService stayService) {
        this.stayService = stayService;
    }

    @GetMapping(value = "/stays")
    public List<Stay> listStays(@RequestParam(name = "host") String hostName) {
        return stayService.listByUser(hostName);
    }

    @GetMapping(value = "/stays/id")
    public Stay getStay(
            @RequestParam(name = "stay_id") Long stayId,
            @RequestParam(name = "host") String hostName) {
        return stayService.findByIdAndHost(stayId, hostName);
    }

    @PostMapping("/stays")
    public void addStay(
            @RequestParam("name") String name,
            @RequestParam("address") String address,
            @RequestParam("description") String description,
            @RequestParam("host") String host,
            @RequestParam("guest_number") int guestNumber,
            @RequestParam("images") MultipartFile[] images) {

        Stay stay = new Stay.Builder().setName(name)
                .setAddress(address)
                .setDescription(description)
                .setGuestNumber(guestNumber)
                .setHost(new User.Builder().setUsername(host).build())
                .build();
        stayService.add(stay, images);
    }

    @DeleteMapping("/stays")
    public void deleteStay(
            @RequestParam(name = "stay_id") Long stayId,
            @RequestParam(name = "host") String hostName) {
        stayService.delete(stayId, hostName);
    }

}

4. Go to CustomExceptionHandler class to include GCSUploadException.

package com.eve.staybooking.controller;

import com.eve.staybooking.exception.GCSUploadException;
import com.eve.staybooking.exception.StayNotExistException;
import com.eve.staybooking.exception.UserAlreadyExistException;
import com.eve.staybooking.exception.UserNotExistException;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.context.request.WebRequest;

@ControllerAdvice
public class CustomExceptionHandler {
    @ExceptionHandler(UserAlreadyExistException.class)
    public final ResponseEntity<String> handleUserAlreadyExistExceptions(Exception ex, WebRequest request){
        return new ResponseEntity<>(ex.getMessage(), HttpStatus.CONFLICT);
    }

    @ExceptionHandler(UserNotExistException.class)
    public final ResponseEntity<String> handleUserNotExistExceptions(Exception ex, WebRequest request){
        return new ResponseEntity<>(ex.getMessage(), HttpStatus.UNAUTHORIZED);
    }

    @ExceptionHandler(StayNotExistException.class)
    public final ResponseEntity<String> handleStayNotExistExceptions(Exception ex, WebRequest request) {
        return new ResponseEntity<>(ex.getMessage(), HttpStatus.NOT_FOUND);
    }

    @ExceptionHandler(GCSUploadException.class)
    public final ResponseEntity<String> handleGCSUploadExceptions(Exception ex, WebRequest request) {
        return new ResponseEntity<>(ex.getMessage(), HttpStatus.INTERNAL_SERVER_ERROR);
    }

}

Test Your Result

1. Save all your changes and start your project. Make sure there’s no error in the log.

2. Open Postman and select the Stay Upload request. Under the request body, select several images from your local laptop. You can also change other parameters if you want.

If here alway wrong, please check.

com.google.api.client.http.HttpResponseException: 400 Bad Request
PUT https://storage.googleapis.com/upload/storage/v1/b/staybookingeve-bucket/o?name=8f2387da-3c67-4c12-aafb-07701b4f6599&uploadType=resumable&upload_id=ACJd0No2sth_JrHl3ibjSjbES-OnAPsaQT9wfDRiugMatCq1x8hYVzes6xCC8t_fukp2FOvXsYwk0LjSrJW0tBuIixfky3sHxHrkO7iyKU7DwE0aBQ
{
  "error": {
    "code": 400,
    "message": "Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access",
    "errors": [
      {
        "message": "Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
}

3. Send the request and make sure there are no errors in the response.

4. Use the List Stay by Host request and make sure you can see the image links in the response.