Skip to main content

Cost Optimization Guideline

If you think that delivering an good app would make sure of its success, then you are completely wrong. Our app is a Saas, aka Software as a Service. It is not just software, it is a service. It is not just a product, it is a business.

When it is business, it has operation fee and revenue. Besides the human resource, office free and tax, the most important cost that reflect the app's success is the server cost. If you open a coffee shop, and the premises cost is already 30%-40% of the revenue, then you cannot survive or grow. The same with the app, if the cost of the server is absurdly high to serve a small number of users, then you cannot grow. However, to optimize the cost, you just need to care about how important it is, and know how to do math.

Let's dive in.

Shopify CDN​

As you may notice in every production CI/CD job, we will have a step in which we push and deploy to Shopify CDN. On staging, we will deploy the JS, CSS directly to the Firebase Hosting. This is different from then React artifacts approach , which is used to prevent removed version of our React admin app. This will use Shopify CDN to serve the assets built from our scripttag folder, which is served via Shopify Theme App Extension.

The main purpose of this approach is to save cost on Firebase Hosting, as we are charged based on the bandwidth usage and the number of requests. By serving the assets from Shopify CDN, we can save a significant amount of cost, which later creates room for the app's growth. Imagine if we have 1000 stores, and each store has 1000 visitors per day, then we will have 1,000,000 requests per day. Firebase charges $0.15 for 1GB of data transfer. So, each file is 50kb, then we will have 50kb * 1,000,000 = 50GB of data transfer per day, and 1500GB per month. This will cost us $225 a month just for hosting. By serving the assets from Shopify CDN, we can save this cost.

Imagine your script is not optimized for size, the number may increase, and also the more app you have.

Some Shopify CDN limits​

However, it will have some limits, which are:

  • You cannot have nested folders structure uploading to Shopify CDN; it should be fine because we put all the files in the static/scripttag folder to the CDN
  • The file size limit is 10MB, which is fine for our case. We can register many apps for CDN, which we are doing now.
  • Cannot control your caching policy. However, serving from Shopify CDN save the browser from making preconnect to another domain. This will save some time for the browser to load the assets. Also, the default Shopify caching policy is quite proficient; it purges the cache when the file is updated. Make sure to call the file with ? v=timestamp to prevent the browser from caching the old file.
  • Cannot upload font files like ttf, woff, just JS, CSS, jpg, png. gif and webp may not work.

How to estimate the cost​

Every developer should be aware of the cost of the app. This is important because it will affect the app's growth. You can ask your leader whether a feature will cost or not, but it is not that hard to estimate the cost by yourself. Google provides a pricing calculator, you can check it yourself here or Firebase Pricing page here.

Which often costs the most?​

I will list the most common cost that you may encounter when developing an app which might leads to pricing spikes and then I will dive into each of them:

  • Function CPU times and memory usage
  • Firestore read and write
  • Firestore Data Storage cost
  • Firebase Hosting bandwidth and request
  • Firebase Storage bandwidth and request
  • Logging cost
  • Redundant configuration

Before we start, just want to remind you that math is not hard, just simple multiplication and addition. Don't be lazy.

Function CPU times and memory usage​

View Firebase Functions examples first here

This is the most common cost and also most hard to estimate cost, people seem to underestimate the cost of running. This might be caused by these reasons:

  • A lot of traffic from store: You might implement a feature that makes API request to our Functions instead of saving cost with Shopify Metafield. Imagine that the same previous example, we have every function run with 256MB and last only 300ms 1000 * 1000 * 300 * 0. 000000463 * 30 = $4167 per month. That is a lot. So do your own math. Go to GCP logging console to see what amount of traffic your are serving for each function.
  • Registering redundant webhooks: Besides having traffic from storefront, the traffic from webhook would be incredibly high. You might register a webhook for every order creation, which is fine. But you might also register products/update webhook, cart/update webhook for example, the traffic if you have thousands of stores can be very high. So make sure to register only the necessary webhooks. The math is quite the same with the above.
  • Unnecessary delay: If you need to perform backoff time with function, use Cloud Task approach . The math is like this, if you have like delay(1000) inside a webhook or clientAPI handler, how much money you will waste? It is: 1000 * 1000 * 1000 * 0.000000463 * 30 = 13890 per month. Remember the lesson about Promise. all whenever necessary, remember it as well.
  • Infinite loop: This is the most common mistake, you might have a function that calls another function, and that function call the first function. This will create an infinite loop, which will cost you a lot. So make sure to have a proper error handling and also a proper retry mechanism. Beware while(true) loop, it is not a good practice. When a function has infinite loop, it cost Firestore as well as Firebase Functions.
  • Asynchronous: The classic Promise.all never gets old. Always remember.

Firestore read and write​

View Firestore pricing first here to understand the cost. Remember the price unit per 100k read and write. This is the most common cost that you will encounter. The cost is quite easy to do quick math.

The pricing per read is less than the write, so the read is not that expensive, right? No, it is the traffic actually. We tend to have more read than write, and the read is more expensive than the write, just we browse more than we post on Facebook. We will break it down now:

  • Document Write: The write is more expensive than the read. So beware placing it at a place in which there would be a lot of traffic. If you think of having a report feature allowing merchant to see how many times the widget is viewed, hover, click. This will cost you a lot since write is expensive, never do it without careful consideration. The math is like this, just for the view part, with a banner that show on every page, and every page has 1000 views per day, and you have 1000 stores, then you will have 1000* 1000 / 100000 * 0.094 * 30 = $28.2 per month . Not so much right? But just more one feature, besides you will have the function invoke, function CPU times as well.

  • Document Read: The read is less expensive than the write, but the traffic is more. So don't put it where there is huge traffic. If you have the sales pop feature, and you read 30 sales pop every API request + 1 for the setting, then you have: (31* 1000 * 1000 / 100000) * 0.031 * 30 = $288.3. Again, not so much right, but that is when you have 31 reads per request, what if you have more? What if you forget to limit the query?

  • Save the read: If inside an HTTP request, or Pub/Sub request, you read the setting or shop document out, remember to pass it to next functions instead of reading it again from the Firestore. This will save you a cost for both CPU times and reads.

Firestore Data Storage cost​

It cost will not appear in the first place, but over the time, it would stab you in the back. The cost is quite high, and gradually increase over time. It can be up to 50$ a day if you are not careful. However, the Firebase team have not yet provide us with a tool to see which collection has the most data, so you have to do it manually by estimating it with document count

Normally, it would investigate a collection that I think would have a lot of data, for example, email logs, API logs, webhook logs, and notifications. I would query the number of records with a day, and then multiply it with the time of its existence. If it has 1 million records a day, then gotcha, you will need to clean up this collection.

However, cleaning a collection is not that easy, you have to read and then delete it. This will cost you, but in the long run, it will save you. However, it is best to implement these:

  • Cron job to clean up the unused data from uninstalled shops, or old data.
  • Implement the TTL for the data that need temporary storage.

Firebase Hosting bandwidth and request​

Well, I mentioned this earlier in the Shopify CDN. If you have problems with Firebase Hosting fee, you might be not implementing the Shopify CDN approach.

Firebase Storage bandwidth and request​

Well, it is also the same as the Firebase Hosting, it is mostly the downloads bandwidth that cost you. Imagine you do a popup app, and the images of the popup are served from Firebase Storage, the image is 100kb, for 1000 store and 1000 views per day, then you will have 1000 * 1000 * 100 * 0.12 * 30 = 3TB per month. It cost $0.1 for 1GB, then it will cost you $300 per month. So if you do not have image, you have video sized in MB, then you know.

I will list out some common strategies to handle the cost:

  • Prefer use Shopify files API: Shopify allows you to upload merchant's content to Shopify files. Just like the SEOon Blog, we upload user-uploaded content to the Shopify CDN instead of Firebase storage. If we serve this content, we will take the cost of their Shopify when blog traffic increases over the time. However, remember that Shopify has limit on file uploads. So only upload the content that is necessary. You cannot upload the customers photo reviews, or video reviews to Shopify CDN, it will exceed the limit soon. We need to save this kind of content in our Storage.
  • Optimize the file: On file upload, you can trigger with Firebase Functions to optimize the file, compress the image, or video. This will save you a lot of cost. You can use sharp for image optimization, and ffmpeg for video optimization. See Firebase documentation here
  • Cache policy: If content should be uploaded to Firebase storage and served to customers, make sure to include cache-control header to the file metadata, see the below code:
 const campaignId = getCampaignId();
const file = renameFile(uploadedFile, uploadedFile.name.replace(/\(|\)/g, ''));
const timestamp = new Date().getTime();
const storageRef = ref(storage, getStoragePath(null, file, timestamp, campaignId));
const metadata = {contentType: file.type, cacheControl: 'max-age=3600'};
const uploadTask = uploadBytesResumable(storageRef, file, metadata);

uploadTask.on(
  'state_changed',
  function(snapshot) {
    // const progress = (snapshot.bytesTransferred / snapshot.totalBytes) * 100;
    // setUploadProgress(progress);
  },
  function(error) {
    console.log(error);
  },
  function() {
    getDownloadURL(uploadTask.snapshot.ref).then(function(url) {
      resolve({default: url});
    });
  }
);
  • Lazy load the content: Implement the lazy load for the content that is not necessary to load at the first time, or the content that is not visible in the viewport. This technique is not so hard, it would not talk about it.

Logging cost​

Another cost that you might not be aware of is the logging cost. The logging cost is not so high, but it is not worth it. If you want to log data to Google Cloud Logging by console.log, console.error, etc. Provide contextual description for your logging, also if it is an object, it takes multiple line in logging, JSON.stringify it if it is object or array:

console.log("current user", shopId, JSON.string(user || {}))

Firebase logging charges you by lines, so if you log the object or array, it takes multiple lines, and it will cost you more in an unnecessary way. So make sure to log the necessary information only.

Redundant configuration​

Actually, this would not be a problem of your does not have high traffic or huge Function CPU times. This redundant configuration of RAM, timeout will make things worse. See this related post on the Function timeout configuration

There are two most common redundant configurations that you might encounter: Functions timeout and Function RAM.

  • Function Timeout: With function timeout, the default value of timeoutSeconds is 60. With high traffic function that potentially has spiking issue or lagging issue, you should keep it 30s max for timeoutSeconds as contingency plan. Just in case the function is stuck, it will not cost you a lot. Never set it to 540s without consideration. For example, normally, Shopify Webhook requires you to response within 5s, so you should keep it at 10s for timeoutSeconds.
  • Function RAM: If you function just do the HTTP request, or just read the Firestore, you should keep it at 256MB. The more RAM you have, you will be able to make for concurrent requests. If you do, consider rising to 512MB to 2GB of RAM. If you send email, or do image optimization, you should consider 2GB-4GB of RAM.